Mistakes of a freelancer

I have been inspired by a post by Soon Hui to write about my mistakes. I have evolved much on the programming side during these years, but my biggest mistakes have been social and economic: dealing with other people. Thus, I have collect a list of my errors committed as a php freelance developer.

• Giving out your personal phone number: no matter what, use separate phone numbers for clients and friends. People have the tendency to call in awkward hours, and having a single number you can shut off after the workday has finished helps your work-life balance .
  
• Lack of tests: every application you write will be maintained in the future, often by you. Even small tweaks (that you can't honestly charge for) can break an application and the safety net of a test suite will free you from the burden of manual testing.
• Providing fixed estimates: estimates should be given in the form of a range, and the whole process of estimation and planning in software development is more complex than the average person thinks. Counting billable hours just does not work and an application's size should be assessed during the requirements gathering.
  
• Tasks instead of features: one pillar of Agile processes is that features are the metric of success and accomplishment, and not tasks. Even if you're working on fixed-price waterfall projects, focus on giving out the features requested because no customer has the time to comprehend technical and infrastructure tasks such as "database modelling".
• Thinking that a client knows what he needs: even in porting legacy applications, no customer really understands the kind of software he wants. It is our job to interact with him to distinguish between mandatory and exciting features and providing the highest value in an application, since we have great  programming skills but little domain knowledge. Often emphasis is put on gold plated features which are really not worth their cost and can cause disasters in the long run (and maybe you are even forced to prioritize features with high risk and little reward). Dialogue, dialogue, dialogue.
  
• Not defining economic terms early: when you write the first line of code you should have an agreement on your reward. This rule of thumb can seem obvious to us, but remember that customers usually come from a whole different world.
  

It's a long list, but I feel that I have grown for more than five years and since I started my journey in computer science even before, I'm approaching the 10000 hours as a developer but still gaining basic experience in business. These errors are something I really had to try by myself.
Have you some freelance experience to share? What do you feel you could have done differently during your career?

Firefox without a mouse

As a developer I have made an habit of using the keyboard for the majority of tasks. Vim for example is my favorite text editor, which does not require point-and-click. This is a productivity requirement: the less my fingers move between the keyboard and the mouse, the faster I am in consulting documentation and other developers' blogs; vim even goes further and lets you scan a document without leaving the home row.

Firefox is also an application where I try to avoid mouse (or touchpad if I am using the EeePC). Unfortunately most sites are not really accessible and I have to resort to mouse for links and forms: it's not satisfying when you [Tab] trough a form and end up in some other place in the page.
Though, Firefox's user interface is really usable without resorting to the mouse. Here are some shortcuts I wanted to share with you:

• <Ctrl>T: create a new, empty tab, and give it focus.
  
• <Ctrl>W: close the currently selected tab.
  
• <Ctrl><Shift>T: reopen the last closed tab.
  
• <Ctrl>PagUp, <Ctrl>PagDown: move between the opened tab.
  
• <Ctrl>L: give focus to the location bar.
  
• <Ctrl>K: give focus to the quick search bar. If you set the browser.search.openintab directive in about:config to true, search queries will be opened in new tabs. Remember that often search engines and websites like php.net and Wikipedia implement the OpenSearch specification, allowing you to add them to the quick search list of engines.
  
• <Alt>Down to select the search engine when you are typing in the quick search bar.

For instance, to search the strpos() function on php.net, assuming that you have stored it in the available engines:
<Ctrl>T, <Ctrl>K, <Alt>Down to select php.net, strpos<Enter>
Or, if browser.search.openintab is set:
<Ctrl>K, <Alt>Down to select php.net, strpos<Enter>
If php.net is already selected since you have already looked for other functions:
<Ctrl>K, strpos<Enter>
Or, given that php.net implements nice urls:
<Ctrl>T, <Ctrl>L, strpos<Enter>

Happy browsing with Firefox and the keyboard! :)

Mocking static methods: the road to Hell...

...is paved with good intentions.
On Thursday I came across the video presentation of a tool to mock static methods in Java:

PowerMock can be used to test code normally regarded as untestable! Have you ever heard anyone say that you should never use static or final methods in your code because it makes them impossible to test? Have you ever changed a method from private to protected for the sake of testability? What about avoiding “new”?

Horror. Not because of the technological revolution - maybe being able to subclass final classes only in the test suite would be fine, and this is just monkey patching - but because static methods have nothing to do with testability. Or, they make code untestable, but testing is not the reason why we want to avoid them.

Dependency Injection is not about testing: it is about good, decoupled design and reusable units. Static calls represent hidden connections between your classes that are not listed anywhere, and they give access to global and usually mutable state. They help creating a misleading Api and constitute the procedural part of object-oriented programming.
This is an example of bad Api:

<?php
class UserRepository
{
    public function getAllUsers()
    {
          return Db::findMany('User');
          // this would be the same: 
          // return Db::getInstance()->findMany('User'); 
    }
}

If in different tests we use this code:

$repository = new UserRepository();
$users = $repository->getAllUsers();

we probably get different results. But why? We had instantiated the same object and called the same method without any parameters. This Api is far from being referential transparent, and that's because it has a static reference that jumps to global state instead of having it injected.
So the question is not How can I mock a static method?, but How can I avoid static methods?

<?php
class UserRepository
{
    private $_db;

    public function __construct(Db $db)
    {
        $this->_db = $db; 
    }
    public function getAllUsers()
    {
          $this->_db->findMany('User');
    }
}

<sarcasm>It was very difficult, isn't it?</sarcasm> Dependency Injection is in fact very simple: ask for what you need, so that people that read your code know where collaborators are coming from and you are not secretly smuggling them in your classes.

I once heard a talk by Misko Hevery where he was making a point about dangerous global state, and he described the following situation. Suppose you have your beautiful test suite, and some classes in it accesses global state, so the tests are not real unit tests but have a bit of integration in their hearts. Keep in mind that in this environment tests are not isolated, since they depend on some global variables, maybe masked as singletons or static classes. The order of execution matters.
So you have MyClassTest, that is the last test case to run in the hour-long test suite, and it is failing. So you try to run it alone, and it pass. Then you run the suite again to figure out what is happening, and it fails again. The test is referencing some global state in which particular data is placed by the other tests before MyClassTest is run. The only way to hava a reliable failing is to run all the suite to set up the global state necessary.
The conclusion is: good luck in finding what is making MyClassTest failing.

Static methods are not object-oriented: they are a recipe for problems if they carry hidden state. They are global, because you cannot keep them on an instance you can throw away after tests. They make the Api difficult to grasp by hiding dependencies under the carpet. The solution is not "Let's open up the hood and wire these things differently so that we can mock static methods", but "Let's not use static methods."
You should even write the tests before the production code if you can. We do not write them only because they catch bugs and regression, but also because real unit tests force to code in a clean and decoupled design. If you are able to unit test your application, its architecture and flow is composed of focused, reliable and reusable parts. If you use static methods and spread new operators everywhere, there are hidden connections between components and mocking these things it's only Action at a distance.

The best backup

Yesterday I was reading some posts in my aggregator about backup techniques, and they reminded me of a (famous?) quote. Thus, I wanted to share my preferred technique for sofware projects backup.

Only wimps use tape backup: _real_ men just upload their important stuff on ftp, and let the rest of the world mirror it ; -- Linus Torvalds, (1996-07-20). Post to linux.dev.kernel newsgroup.

I know this is not applicable to every project, but usually open source is full of benefits:

• if you follow the doctrine (everything not reproducible is kept under version control), there are free services like Google Code and SourceForge which will host the Subversion (or Git) server and care for automatic backup and disaster recovery. This means that if your old hard disk breaks your work is saved up to the last commit.
• Obviously you can get collaboration and feedback from other developers.
• It takes minutes to set up a working copy of your project for development and testing as long as you have an Internet connection available: it is globally accessible all over the world.

There are cloud services for private projects, but they have a cost in money. Open source support is free and competitive today. Every management application you will need is provided and updated by SourceForge: trac, wikis, file release system...
In general, sharing with the rest of the world your code and your writings is the way to save them from oblivion:

Some years ago, two programmers at Cisco (the networking-equipment manufacturer) got assigned the job of writing a distributed print-spooling system for use on Cisco's corporate network. [...]
The duo came up with a clever set of modifications to the standard Unix print-spooler software, plus some wrapper scripts, that did the job. Then they realized that they, and Cisco, had a problem.
The problem was that neither of them was likely to be at Cisco forever. Eventually, both programmers would be gone, and the software would be unmaintained and begin to rot (that is, to gradually fall out of sync with real-world conditions). No developer likes to see this happen to his or her work, and the intrepid duo felt Cisco had paid for a solution under the not unreasonable expectation that it would outlast their own jobs there.
Accordingly, they went to their manager and urged him to authorize the release of the print spooler software as open source. Their argument was that Cisco would have no sale value to lose, and much else to gain. By encouraging the growth of a community of users and co-developers spread across many corporations, Cisco could effectively hedge against the loss of the software's original developers. -- Eric S. Raymond, The Magic Cauldron

What is your opinion on sharing your work as open source?

More questions on controllers testing

Sune wrote to me yesterday with some questions about testing Zend Framework controllers and proper dependency injection, which to me is a fundamental practice in object-oriented programming. I have already responded to similar mails in the past and this seems to be an hot topic nowadays, so as always I think other readers can benefit from this discussion and I'm sharing it here.

Because of you I am trying to move my software to use factories and dependency injection, also removing
singletons and Zend_Registry usage in controllers. But I am a bit confused, what is the right way to do this.
My plan is to bootstrap the main factory in the bootstrapper, and then use
$this->getInvokeArg('bootstrap')->getResource('factory') in controllers. Is this good practice?

Summarizing, the best thing would be creating the controller by yourself (or having its creation configured someway), but it can be an overkill to set up a similar approach on a Zend Framework application, since it requires a third-party DI container and controllers should always be thin.

Thin controllers means we probably want only to perform integration testing on them with Zend_Test, and not real unit testing as there is not much logic to exercise.

Since we cannot create in the bootstrap all the collaborators that could be possibly needed (we want to lazy-load collaborators that may not be referenced), your approach is similar to a Guice provider and I think it is very valid. In integration testing you should then use a different factory or configure this one to provide some fake components when the real ones are not applicable. For instance, a mailing service object ca be replaced with a fake implementation that records all the sent mail and lets you assert on them.

And in conjunction with that. Would it be good practice to inject a Zend_Config object into the factory?
Some models requires options from the config file, smtp username etc. and the factory would need those information to create the models.

Of course. It's up to you how to organize the config object, and you can and should change part of it in the testing environment. The power of DI containers is that during configuration you can specify not only scalars like database connection strings, but also different classes and implementation for the collaborators you inject.

To set or not to set

You probably know I am a test-infected developer and big proponent of Dependency Injection. You also have seen from the examples in this blog that I favor constructor injection, where a component asks for his collaborators in its own constructor:

class FacebookService
{
    private $_httpClient;
    public function __construct(HttpClient $client)
    {
        $this->_httpClient = $client;
    }
}

A factory or a configurable container can then recursively resolve dependencies and provide the class with what it asks for. The constructor's wiring code is dumb to write, but it is very concise and expresses intent: assigning the collaborator to a private property which will not be subsequently touched (as there are no setters).
The Api is also very clear as the constructor specifies everything is needed to compile and instantiate this class, while it does not provide the means to change the collaborators.

When the collaborators number is high, however, we may find difficult to use a constructor with a long signature. The obvious solution is trying to reduce coupling and analyzing the collaborators to see if everyone of them is really mandatory. It may be the case of a class that has too much responsibilities in accessing different parts of the object graph, or that collaborators leak into the class while they should be encapsulated in some other component.
Sometimes, there is nothing we can do to reduce the collaborators number:

class CommentsRepository
{
    private $_dbAdapter;
    private $_mailer;
    private $_logger;

    public function __construct(Zend_Db_Adapter $dbAdapter = null,
                                Zend_Mail_Transport_Abstract $mailer = null,
                                Logger $logger = null)
    {
        $this->_dbAdapter = $dbAdapter;
        $this->_mailer = $mailer;
        $this->_logger = $logger;
    }
}

This happens in some common cases:

• the collaborators are options (value objects or scalars) which change the behavior of the component;
  
• the class is a mediator between many objects and it is its own responsibility to deal with many collaborators.
  

Again, we can try to minimize the options or the objects involved, but the essential complexity will never vanish. Another solution might be passing the services via a method parameter only when they are used, but it usually violates encapsulation (a Controller having to keep a reference to a Logger even if it does not use it). Moreover, they may be needed in every method.
A long constructor is not clear as in almost any languages there are no named parameters (thanks Python) and we can forgot the parameters order in manual injection, or we may find difficult to extract automatically metadata on the collaborators if we are in a dynamic language like php.
The type of dependency injection we should adopt is slightly different: setter injection. This approach transforms the CommentsRepository class in:

class CommentsRepository
{
    private $_dbAdapter;
    private $_mailer;
    private $_logger;

    public function setDbAdapter(Zend_Db_Adapter $dbAdapter)
    {
        $this->_dbAdapter = $dbAdapter;
    }

    public function setMailer(Zend_Mail_Transport_Abstract $mailer)
    {
        $this->_mailer = $mailer;
    }

    public function setLogger(Logger $logger)
    {
        $this->_logger = $logger;
    }
}

Though, there are some problems with setter injection that we should solve:

• setters allow changing collaborators after the construction: often it is a conterproductive operation and so it should be avoided. The setters can check if the corresponding private property is null before accepting the parameter.
• the Api is not clear: why there are setters if I cannot set anything? I suggest to extract an interface where the setters are not present. This solves also the previous problem as the client class will depend only on an interface where setters are not defined and in static languages it is not even allowed to call them. In dynamic languages, the developer should refer to the Api documentation of CommentsRepositoryInterface and not of the CommentsRepository concrete class.
• we may forgot a collaborator: both in manual and automated dependency injection you can forgot to call a setter or to add a collaborator to the configuration, and the result is a broken object hanging around. So you should maintain some form of test for the factory or the container (typically in integration tests). A missing collaborator is a wiring bug and it is simple to solve since it is going to manifest nearly always: the application will explode saying you called a method on null. Note that since I use null defaults for constructor parameters this problem is also present in constructor injection.

I hope you consider setter injection, as I avoided it without real reasons in the past and the design of your application can benefit from it.

Doctrine 2 and Zend Framework first date

This morning I have tried for the first time to use Doctrine 2 in a Zend Framework application. I used the latest release, 2.0.0 alpha3, for this experiment.
The chosen application is my recently born project NakedPhp, a port of the Naked Objects Java framework which generates the user interface and let the end user manipulate domain objects directly.
During this first run, I have not set up an application resource yet and I have just hardcoded a few configurations to bootstrap correctly Doctrine. I will publish a resource class (conforming to the Zend_Application_Resource_Resource interface) soon when I have it ready.

Doctrine\ORM\EntityManager is the Facade class which act as a portal towards the functionality of Doctrine 2, and it is the homologue of Hibernate EntityManager. Our code should interact mainly with this class.
Though, I have isolated the EntityManager behind an interface since I do not want infrastructure code to slip in NakedPhp for now. The code will obviously depend on Doctrine but it is good practice to have an interface I can mock out easily, as I don't need all the methods of the EntityManager and this way I just hide everything is not mandatory instead of introducing coupling to it.

Doctrine 2 is released in three packages: Common, Database Abstraction Layer and ORM. Instead of downloading three different packages I just grab them from the subversion repository:

svn export http://svn.doctrine-project.org/tags/2.0.0-ALPHA3/lib/

and move the Doctrine/ and vendor/ folders in my library/ directory along with Zend/. The vendor folder contains a small annotation parser.
It can be useful also to export other resources:

svn export http://svn.doctrine-project.org/tags/2.0.0-ALPHA3/bin/
svn export http://svn.doctrine-project.org/tags/2.0.0-ALPHA3/sandbox/

The bin/ folder contains the doctrine.php and doctrine command-line scripts (same thing), while the sandbox provides a working example of Doctrine 2.

Doctrine 2 prescribes that model classes (entities) and proxies should be autoloaded, so after moving doctrine.php in application/ I deleted the reference to Doctrine autoloader and added:

require_once __DIR__ . '/../application/bootstrap.php';

which is my bootstrap file, where:

• the library/ folder is added to the include_path
• the autoloader is set up to load Zend/ classes
• a Zend_Loader_Autoloader_Resource sets up autoloading for my model classes.
• my autoloader is set up to take care of \Doctrine and \NakedPhp namespaces.

In the future, I will add proxies autoloading setup to this file. Since you probably don't have your own autoloader for namespaced classes, you can simply use IsolatedClassLoader from Doctrine\Common.

It's time to code a cli-config.php file to use with doctrine.php; this file should define two variables (it is well-documented in the sandbox example). My final result is:

$classLoader = new \Doctrine\Common\IsolatedClassLoader('Proxies');
$classLoader->setBasePath(__DIR__ . '/../application/');
$classLoader->register();

$config = new \Doctrine\ORM\Configuration();
$config->setMetadataCacheImpl(new \Doctrine\Common\Cache\ArrayCache);
$config->setProxyDir(__DIR__ . '/Proxies');
$config->setProxyNamespace('Proxies');

$connectionOptions = array(
    'driver' => 'pdo_sqlite',
    'path' => '/var/www/nakedphp/sqlite/database.sqlite'
);

// These are required named variables (names can't change!)
$em = \Doctrine\ORM\EntityManager::create($connectionOptions, $config);

$globalArguments = array(
    'class-dir' => __DIR__ . '/../application/models'
);

Which is practically the cli-config.php file grabbed from the sandbox, but slightly edited:

• there were two instances of Doctrine\Common\IsolatedClassLoader, one for the entities and one for the proxies. I deleted the first one since entities autoloading is already taken care for in the bootstrap.
• I haven't used proxies for now, but the configuration is mandatory. The default namespace and folder are enough.
• I changed the path to the sqlite database. Sqlite was the fastest choice to get the application up and running, but remember that both the sqlite database file and its directory must be writable by apache and php.
• I changed also the argument class-dir to specify my entities folder.

Before starting to use the Doctrine 2 command-line interface, you will have to define annotations on your model classes. For instance, @Column and @OneToOne annotations. Since I have developed without even caring about the database until now, I had to add also an $_id private property. :)
I also submitted a patch to improve the errors generated by the schema tool in case of incorrect field names referenced on relations, which is what happened to me today.

Now, it's time to generate your schema:

php bin/doctrine schema-tool --re-create --config=bin/cli-config.php

If cli-config.php is in the directory where you issue this command, you can leave out the --config option.
Maybe after playing a bit with Doctrine 2, you will want to see what was inserted in the database:

php bin/doctrine run-sql --sql="SELECT * FROM Example_Model_Place" --config=bin/cli-config.php 

To obtain an EntityManager reference in a controller, you can set up a dumb resource that includes cli-config.php and return $em. I used a factory which was already available and add a method for retrieving the instance.

So I now have an hacked working instance of Doctrine 2 in my project. The next step will be writing an application resource to allow configuration to be specified following the standard, in application.ini. I will publish this resource in the next days.

The image at the top is the NakedPhp example application screen that says saving was successful. I implemented the storage of my persistence-agnostic in-memory object graph in less than an hour.