The Swiss Army knife Programmer

MacGyver was unlike secret agents in other television series and films because, instead of relying on high-tech weapons and tools, he carried only a Swiss Army knife and duct tape.
[Wikipedia, MacGyver page]

After the infinite series of Duct Tape Programmer articles that has been populating my Google Reader last week, I've decided to write about a similar class of programmer who you surely know some instance of.
Just like MacGyver, the Swiss Army knife programmer carries his preferred tool along with a stock of duct tape, which as Joel says represents the using large amounts of duct tape to keep together the pieces of an application mindset.
Armed with his tool, which contains screwdrivers, a can opener, a magnifying glass, an altimeter and an mp3 player, the SAK programmer goes on and on and uses it for every task he has to do. When he has a new business idea, it involves the use of his tool to conquer the world. When he has to carry out a boring, obvious task, he tries to squeeze in his tool in some way.

Practically speaking the equivalent of Swiss Army knife is a programming language or particular technology which is overused, in applications where it is out of scope. A big toolbox is better than a Swiss Army knife if you know how to use all the tools contained instead of carrying a small one which promises to do everything you need.
These are examples of Swiss Army Knife cases:

• using php for gtk applications. The php gtk extension, which contains the bindings for the gtk graphic library, is an edge project which provides the capability for php to build desktop applications with windows and buttons on the machine where it is run. Certainly the developers of this extension are having a lot of fun taking php to its edge, but php is a language where the interpreter comes with a default max_execution_time directive set to 30 seconds. I don't think the core developers were thinking of a php script which runs for hours when designing the engine.
• usage of IDEs in every situation, when a bit of command line fu can solve problems efficiently and quickly. NetBeans and Eclipse try to provide every feature a developer needs in a single application, also via plugins: this resembles a overloaded knife. NetBeans here is a bit better then Eclipse in the sense that it takes avdantage of command line applications like svn.
• the opposite overusage of command line fu when opening a text editor and doing a find&replace is enough.
• Google Chrome operating system, which relies on web based applications only. Cloud computing is great and I strongly believe web applications are the next big thing: Gmail is proving it every day. But these kind of apps are probably not capable, at the moment, to replace every binary on your machine. Try to edit a video via web.
• Design patterns are great standard solutions for object-oriented languages pitfalls, but their overusage can plague a code base. Factories (for entities) and above all Singletons can and should be limited to the cases where they provide real value.
• I once fell in love with the Dojo grid and I tried to use it for every management application I could think of. It can be a serious issue if you are unaware of being a Swiss Army knife programmer on certain technologies, when you find yourself as an advocate in a religion war.
  

There are very few general-purpose technologies that can be used in nearly any project, and they are usually built from scratch to solve analogue and generic project management issues:

• Subversion and other source control systems work well in many situations. As the name says, source code files were the original subject of version control, but Git and Subversion have expanded their dominion on wiki formatted documentation, xml configuration files and plain text. A common suggestion is  to put everything you cannot build under source control (and this leaves out binaries, fortunately).
  
• Trac and Bugzilla can be used for every software project. Though, their scope is limited to software project management and for instance a novel developed with the help of Trac is a strange thing (unless it is written from a group of authors who need a communication tool). The power of such software is in facilitating communication between developers and usage from a single user is likely to be an overkill.
  

The power of open source applications resides in the reuse of code and libraries and even of entire projects in incredibly different fields, like in Subversion's case. But before stretching a technology over its limits, think if you are really using the right tool for the job.

Not so strangely, searching "the right tool for the job" on Google shows the first result is "Lisp is the best tool for large (and small!) projects."

The power of tracking and logging

Tracking resources - time, money, code changes and whatever else - is a powerful way to improve your understanding of them. Logging is even more powerful as it's automatic tracking done by your tools for you.

The practice of tracking time for the various activities one intraprends is one of the pillar of time management. Analyzing what is sucking up precious man-months gives you a clue to eliminate the tasks that really aren't worth the effort, instead of finding them by chance or by some preconcepts you may have. Tracking your spending is the first advice a financial counselor will tell you if you experience money problems.
The concept of time tracking is also present in programming: the only sure-fire way to optimize an algorithm or an application is to profile it as the first step, and then making changes to the code which constitutes the bottleneck. While an algorithm theorical analysis produces a result expressed in O(f(n)), profiling it on real data allows the programmer even to confront different O(n log n) sorting algorithms.

At a different perspective level, tracking is present in the modern methodologies of development with tools as source control systems. Every code check-in is tracked and remains forever in the history of the codebase: no line of code is ever lost in shared directories or email folders. This type of tracking information is better named as logging.
It is very cheap for a software system, if built correctly, to conserve every chunk of data that passes on its bridge. Subversion and other vcs do exactly the process of logging any single commit, and the logs reveal useful when viewed as changesets or while generating a changelog for a new release. Project management tools like trac, built upon subversion, log every change to the tickets which report bugs and feature requests, along with the edits of the wiki pages. It is a small job paragonated to the extent of tracking Wikipedia does.
If you're talking on irc or other istant messenger, your client is probably writing logs of the conversations to disk. Every enterprise java application logs exceptions to a file or to database, too.
Having a large amount of organized data is a source of valuable information, as this logging capabilities allow to:

• posting a conversation between developers on the wiki for further reference;
• giving commit access to new developers knowing their commits can be rolled back;
• listing the commits which affects a particular bug, which trac does;
• generating a changelog file by looking at the list of commits in a particular period on a branch;
  
• updating a working copy or a deployment of a php application transmitting only the modified files;
• generating a list of the locally modified files in a few seconds to see what is being committed (svn status).
  

There are infinite possibilities for the usage of logged data. It is often said that every item of a project which cannot be automatically generated (like builds) should be put under version control.
Logging was once expensive, when it was done by hand on dusted registers: the data was patiently annotated during the day and it wasn't going to be useful anywhere else. Now that the information era is arrived, take advantage of the logging capabilities of your tools and never write the same thing twice when a machine can do it for you.

The image is a photo of a rinascimental ledger, used for accounting. Money transaction have a long logging tradition for fiscal purposes.

Php deployment with Subversion

Subversion, svn for friends, is a source control system, which stores all your source files and their history in a centralized place. How to use Subversion for deployment of websites?

Have you ever had the pain of deploying some megabytes of php source files with ftp? It takes some time, and using library and/or frameworks in you application the total size of files to be transferred is high. Moreover, also the number of single files is huge and this fact will slow down the process while your ftp client walks in and out of thousands of directories.
What if some configuration files are modified? You have to manually search the folders to replace and upload only the "engine" files, while avoiding to overwrite some config.inc.php which database credential are stored in. This is a daunting task to do with ftp.
Take away the pain with Subversion!

Prerequisites
Using a subversion client to deploy your web application requires, obviously, that the php files are stored in an svn repository. On the other hand, using a svn repository gives you only benefits over plain storing of php files in the filesystem, that goes over the scope of this article. Since Ossigeno is open source, I use SourceForge svn repository, that comes for free. If you develop a commercial application, you can eaily set up a svn repository on your machine. I use an old laptop with Ubuntu Server installed to store private modules sources (modules built for contractors), with port forwarding from the router to the laptop and a dyndns account to provide.
To run svn in the document root of the webserver, you'll need to have ssh access to the machine, or, alternatively, have physical access. For instance, this blog runs on SourceForge webserver, and SourceForge gives shell access to registered users. Tipically 8€/year shared hosts does not offer this type of service, however.
Finally, the svn client has to be installed on the webserver. If you have shell access, it's very likely that also svn is present. To find out, simply run:
$ svn --version
on the shell and see what happens.
Now that we know we have the tools for the job, let's install an Ossigeno copy from svn repository.

Initial checkout
Supposing, we are in the folder where we want to install (a document root subfolder /var/www/blog or the document root itself /var/www), we simply run:
$ svn checkout https://ossigeno.svn.sourceforge.net/svnroot/ossigeno/tags/3.0_beta6/core/ .
The SourceForge path has to be substituted with your server path. An hostname should be used, while an ip will not work correctly if it is dinamic (that is probably the case if you don't have an hostname). Subversion stores in its .svn hidden folders the hostname used for the initial checkout, so you don't have to retype a long url everytime you update.
Now that you have a working copy, you can proceed to install the application as it was extracted from a tarball or from a zip package.
Some school of thought uses the svn export command to obtain a copy of the code, but a checkout is more useful for what we're doing next.

Common operations
You had found a typo in a source file, and you corrected it on the webserver. Since this is a working copy and not an exported one, you can simply do:
$ svn commit -m "fixed a typo" folder/script.php
and the diff with the original file will be sent to the repository, sending upstream the simple patch you have created. Depending on your repository setup, a password or a rsa key will be requested by the svn client to proceed with commit.
Now you want to update you application to a new version: this version (of Ossigeno in the example) is tagged 3.0_beta7. This is work for subversion.
$ svn switch https://ossigeno.svn.sourceforge.net/svnroot/ossigeno/tags/3.0_beta7/core/ .
The repository will calculate the diff between 3.0_beta6 and 3.0_beta7 and will send to the client a temporary patch that will be applied to the working copy on the server where you execute the command. Please note that this operations are possible only because we choose to use checkout instead of export. The checkout command duplicates the original files in .svn folders, and thus double the space occupied; but it is very fast on updates and commits because only the file deltas are sent over the network.
So you can switch over the main branch of Ossigeno to have the bleeding edge version:
$ svn switch https://ossigeno.svn.sourceforge.net/svnroot/ossigeno/trunk/core/ .
and every two days run
$ svn update
to obtain the last features.
The last two steps are not recommended as trunk randomly breaks the working copy, being a in-development version.

Issues
You have edited script.php, but not commit this to the repository since it is a local hack that must not spread to other copies of the application. What will happen when you run switch or update?
The answer is multiple. If the file is not modified in the repository, your local copy will be mantained as-is. If the remote file is changed, svn will try to merge the files in a new version and only if the changes overlap, you will be prompted for a manual edit that will resolve the conflict. In Ossigeno example, configuration files are stored in a folder with svn:ignore property set with a filename corresponding to the host, so a file is added for the local configuration in application/config directory and ignored from subversion in commits and updates.
I encountered another issues while installing Ossigeno on this server. In the static/ folder some public directories are kept; Ossigeno puts cached html in static/cache/ and image previews in static/preview/, thus the webserver/php process needs writing permissions on the folders. However, on SourceForge the document root htdocs/ is readonly (from the webserver), and writable directories has to be created in persistent/ and pointed by a simbolic lynk in the htdocs.
So what's the problem? Those dirs were in repository, so I cannot remove them and place some simbolyc links. I had to remove the public dirs in repository, saving static/ as an empty folder with svn:ignore property set to '*'. Then I set up a phing task to recreate them and to setting the writing permissions, leaving the option to the user to create them manually, with phing task or with the installation process, or eventually to put some symlinks in static/.

Conclusion
This points out a general rule to use with subversion deployment: do not import in repository what will change on the production box. Do not import configuration files, because if you do at the next svn commit (that defaults to the current folder) you will sent to your repository the database credentials of the website; do not import dirs that can be substituted by symlinks, because if you removes those folders you will break the working copy. Do not import temporary files. You can setup svn:ignore property on folders to have this files not listed by svn status (see the best Subversion documentation, its book).
Have a nice day using the power of Subversion not only for development but also for deployment!