Team learning session: surviving legacy code by J.B. Rainsberger

This is the description, and experience report, of an exercise I picked up many years ago and then used in a team I lead. I'll describe this in the present tense as I imagine applying the exercise in similar contexts. Credit to J.B. Rainsberger for coming up with this diabolical codebase and make us work on it!

Context

A team has been formed and has been working on a single product, for example a TypeScript monolith comprising a frontend and APIs.

The team is now tasked with taking over a set of services and frontends written in PHP (or some other programming language and ecosystem). It turns out this set of services is the flagship product of the organization, and it needs to support innovative business change. Yet, normal operations and incremental improvements had been mostly outsourced as part of the technical strategy for the last few years. 

The focus of innovation was elsewhere. Testing coverage is redundant, confusing or missing, depending on the area you are working on. Very few people in house know the traps of these codebases: we are now in the realm of legacy code.

Learning goal

The original exercise by J.B. Rainsberger is oriented to learning patterns to take over legacy code: inherited codebases that exhibit a lot of business value. If they don't have business value, then why working on it?

Specifically, the patterns regard:

• characterizing and understanding code
• testing it, at various level of scope from whole application to small units
• refactoring, with a safety net in place to support those changes without introducing regressions.

There isn't an emphasis on shipping new features, and there is no product owner.

To the original goals, we add a separate one here with its own new difficulty level: learn a new programming language that most of the team has not worked professionally in before, or has not picked up for a long time. With a new programming language, there's also knowledge of a new set of tools for running, testing, or linting the code. For PHP alone, in this example, the toolset ranges from Composer to PHPUnit, PHPStan, or PHP-CS-Fixer.

The learning at all these levels translate in being ready to use these patterns and tools on real projects, making it much easier and safer to deliver business value on those.

Activity

The original problem statement gives this guidance:

• Refactor this code to understand it
• [Decide] when to refactor and when to rewrite, and how to do that safely.
• lResolve the central conflict of legacy code: I need tests to refactor safely, but I need to refactor to write tests effectively.

I've been running these workshops back in the days of the Legacy Code Retreats, which is where the name came from. The setting diverges from a whole-day session, where iterations are handle by various pairs. In this case, the repository is forked at the beginning of the activity and several sessions can take place, for example with one or more hour per week reserved. The participants work together as an ensemble, but they could split into multiple rooms working independently if there are too many.

11? 12? What?!

A loose plan to follow can be:

1. run the application. Does everyone understand the problem domain and what problem the code solves?
2. introduce golden master testing for characterization. This step involves figuring out seeding and reproducibility, and how to achieve the isolation of automated tests from the outside world and its changing conditions.
3. introduce further testing at lower levels. This step involves introducing tooling for running code easily, testing it, or performing static analysis; manipulating the folders and file structure safely; and refactoring the code to isolate the units under test.

I act as a language expert here, but not with an agenda in mind. What to learn is decided by what moves the participants want to take, and what they are missing to be able to do so. It also pays off to prompt the team to research rather than trusting anyone's memory or word: tests and experiments are the source of truth.

The fact that this is a completely new, fictional, and ugly codebase should help with the feeling of safety when raising lack of understanding. The code is supposed to be hard to understand and fragile. Once that is established we can then work on our own learning, directed to improve our situation. It's a very different framing that delivering features on real legacy code with a deadline in mind.

Resources

The original code repository used in legacy code retreats.

The canonical explanation of a golden master approach to characterization testing.

Specifically for PHP, it helps to understand how seeding random number generation works to achieve reproducibility.

Retrospective

Add to a board two prompts to allow reflections to appear concurrently. For example:

• what did we learn today? Anything from a language construct that did not map easily to something I already knew, to a tool's use cases.
• any ideas for next time? There are lots of potential directions of exploration, and we want to crowdsource the gaps the participants are starting to see so we can fill them. This helps getting into context quickly when we start a new hourly session at another date.

Eris 0.11.0 is out

Eris 0.11.0 has been freshly released, and I'll be listing here various contributions that the project has received that are included in this new version and in the previous one, 0.10.0, which didn't have an associated blog post.

For a full list and links to the relevant pull requests and commits, see the ChangeLog.

0.10

• The Eris\Facade class was introduced to allow usage outside of a PHPUnit context.
• Official PHPUnit 7 support was introduced.
• Fixed a corner case in suchThat()

There are some small backward compatibility breaks with respect to 0.9; they regard unused features (or at least I thought) including Generator::contains().

0.11

• Official PHP 7.2 support
• Annotations support for configuring behavior that is usually configured through methods: @eris-method, @eris-shrink, @eris-ratio, @eris-repeat, @eris-duration

Some acknowledgements

Most of this work comes from contributions, not from me. I'd like to say a word of thanks to the people that have taken the time to use Eris in some of their projects but also to feed back a fix, an extension, or a substantial improvement.

Eris 0.9.0 is out

In 2016 I moved to another country and as a result of this change I didn't had much time to develop Eris further. Thankfully Eris 0.8 was already pretty much stable, and in this last period I could pick up development again.

What's new?

The ChangeLog for 0.9 contains one big new feature, multiple shrinking. While minimization of failing test cases is usually performed with a single linear search, multiple shrinking features a series of different options for shrinking a value.
For example, the integer 1234 was usually shrunk to 1233, 1232, 1231 and so on. With multiple shrinking, there are a series of options to explore that make the search logarithmic, such as 617, 925, 1080, 1157, 1195, 1214, 1224, 1129, and 1231. If the simplest failing values is below 617 for example, at least (1234-617) runs of the test will be skipped by this optimization, just in the first step.
This feature is the equivalent of QuickCheck's (and other property-based testing libraries') Rose Trees, but implemented here with an object-oriented approach that makes use of `GeneratedValueSingle` and `GeneratedValueOptions` as part of a Composite pattern.

This release also features support for the latest versions of basic dependencies:

• PHPUnit 6.x is now supported
• PHP 7.1 is officially supported (I expect there were mostly no issues in previous releases, but not the test suite fully passes.)

Several small bugs were fixed as part of feedback from projects using Eris:

• the pos() and neg() generators should not shrink to 0.
• float generation should never divide by 0.
• shrinking of dates fell into a case of wrong operator precedence.
• reproducible PHPUnit commands were not escaped correctly in presence of namespaced classes.

A few backward compatibility fixes were necessary to make room for new features:

• minimumEvaluationRatio is now a method to be called, not a private field.
• GeneratedValue is now an interface and not a class. This is supposed to be an internal value: project code should never depend on it and it should build custom generators with map() and other composite generators rather than implementing the Generator interface, which is much more complex.
• the Listener::endPropertyVerification() method now takes the additional parameters $iterations and the optional $exception. When creating listeners should always subclass EmptyListener in order not to have to modify the not implemented methods, which will be inherited.

What's next?

My Trello board says:

• still decoupling from PHPUnit, for usage in scripts, mainly as a programmable source of randomness.
• more advanced Generators for finite state machines and in general a more stateful approach, for testing stateful systems.
• faster feedback for developers, like having the option to run fewer test cases in a development environment but the full set in Continuous Integration.

I'm considering opening up the Trello board for public read-only visibility, as there's nothing sensible in there, but potential value in transparency and feedback from the random people encountering the project for the first time.

As always, if you feel there is a glaring feature missing in Eris, feel free to request it on the Github's project issues.

Eris 0.8.0 is out

In the period before my move to Cambridge I got some time to work on Eris, and to use it to test the guts of Onebip's infrastructure. Lots of new features are now incorporated in the 0.8.0 version, along with a modernization of PHP standards compliance carried out by @localheinz.

What's new?

Here's the most important news, a selection from the ChangeLog:

• The bind Generator allows to use the random output of a Generator to build another Generator.
• Optionally logging generations with `hook(Listener\log($filename))`.
• disableShrinking() option.
• limitTo() accepts a DateInterval to stop tests at a predefined maximum time.
• Configurability of randomness: choice between rand, mt_rand, and a pure PHP Mersenne Twister.
• The suchThat Generator accepts PHPUnit constraints like `when()`.

Some bugs and annoyances were fixed:

• No warnings on PHP 7 anymore.
• Fixed bug of size not being fully explored due to slow growth.
• Switched to PSR-2 coding standards and PSR-4 autoloading.

And there were some backward compatibility breaks (we are in 0.x after all):

• The frequency generator only accepts variadics args, not an array anymore.
• Removed strictlyPos and strictlyNeg Generators as duplicated of pos and neg ones.                                     
• Removed andAlso, theCondition, andTheCondition, implies, imply aliases which expand the surface area of the API for no good reason. Added and for multiple preconditions.

Eris is now quite extensible with custom Generators for new types of data; custom Listeners to know what's going on; and even different sources of randomness to tune repeatability and performance.
I believe what's very important about this release is the release of technical documentation. This is not a list of APIs generated by parsing the code, but is a full manual of Eris features, which will be kept up-to-date religiously in the repository itself and rebuilt automatically at each commit.

What's next?

My Trello board says:

• decoupling from PHPUnit: it should be possible to run Eris also with PHPSpec (already possible but not as robustly as it can be) or in scripts.
• Multiple possibilities for shrinking, borrowing from test.check rose trees. This feature may speed up the shrinking process and make it totally deterministic.
• A few more advanced Generators: for example testing Finite State Machines.

If you are using Eris and wanna give feedback, feel free to open a Github issue to discuss. 

When to write setters

I have set out, almost unconsciously, to use constructor injection by default in the last few years while writing object-oriented applications. With Dependency Injection as a given, constructor injection satisfy most of my requirements for building an object graph and dynamically configuring collaborators.

The spectrum

I see the statefulness of an object not as an absolute but over a spectrum.
At one end of the spectrum we have immutable objects: these objects acquire a configuration in their constructor and are effectively final (to employ a Java-specific term) for the rest of their lifecycles. Their fields are private and there's no way of modifying them outside of the constructor; their collaborators are only scalars or other immutable objects. In Java fields can even be final so that accidental reassignment is regarded as a compile-time error.
The physical state of an object may still change without its external behavior being affected, like in the case of caching. I still consider this kind of white-box-immutable objects as immutable.
Stateful objects instead may have a behavior that changes as a result of its own state (or of its collaborators). You call some Command methods on it, and the response of further calls to Query methods change. Hopefully the Commands encapsulate some domain logic to constrain the state transition to a valid and modelled one.
At the extreme end of the spectrum we find setters: methods which only mutate the value of one or more fields, possibly skipping any validation, domain modeling or consistency check. The setters considered here are public methods, because their limited-scope versions do not provide the same violations.
If you want to write procedural code, setters proliferate (still it's probably easier to just use public fields at that point). There are only a few valid use cases where I have found setters useful in object-oriented programming, and here is the (short) list.

Configuration which has default values

Classes may have a few configuration values that you are able to tune; especially when there is more than a few of these parameters, I find useful to separate hard dependencies in the constructor and setters that are able to override the default parameters when the object has already been constructed. If you forgot to call these methods, the object still has to work correctly.
Alternative solutions for this use case are of course constructors with default parameters, which I definitely prefer if there are not so many options to tune (1 or 2). You can also look into with Value Objects which produce a new instance upon reconfiguration, and model all of the configuration parameters as a single entity; or into a Builder if you want to invest in an additional class and its API.

Adding Observers

Observers (or listeners if you prefer) are collaborators which are notified of internal events happening inside an object that may interest them.
I treat the observers of an object as an append-only list data structure, with an empty list or array as the natural Null Object. The object is initialized without any observer, and a setter like addListener(...) has the more limited capability of adding an observer but not of removing or modifying an existing one.
The nature of the Observer pattern is investing on a common bus that many observers can be attached to, even if they come from different packages and libraries. Therefore I find it natural to support the dynamic wiring of other objects, even if they make the object more mutable can before. The needs of integration become more important than guaranteeing safety of construction in these scenarios.

Reconstitution

When objects are unserialized from a cold storage such as a stream of bytes or a JSON object, encapsulation is very likely going to be violated. Object-relational mappers have been doing this for ages by working directly on annotated private fields, with sometimes powerful results but also lots of dangers from storage and code being out of sync.
In the scenarios where you control the reconstitution of objects, such as rebuilding an object from a MongoDB document, it's often easier to provide an explicit API like a setState() method than to rely on the magic of a library which is going to bypass your public methods. To constrast the possible misuse, you can tag this method as @private (or package protected in Java), or make it very awkward to use outside of the persistence context by requiring a particular data structure to be passed in.

Conclusion

There are very few use cases for setters in real object-oriented programming; default to constructor injection and to immutable objects to avoid overcomplicating your design. Employ setters for non-mandatory, cross-cutting initializations so that your code does not have to bend over backwards to support these use cases while at the same time it can be robust to cowboy modification of internal state.

On property-based testing a highly concurrent job queue

Exploring Eris

Recruiter is a job queue written in PHP, open sourced by Onebip in its 2.x version. It has been used on some inward facing production services, but following the necessity to roll it out to more and more project, I have started a thorough testing campaigns to flush out possible concurrency bugs.
The job system is composed of a single Recruiter process and multiple Worker processes (moreover any other PHP process can enqueue a job). These processes may run on any machine inside a local network and share a MongoDB database where they collaborate to empty the collection of jobs to do. The design of these multiple collections is carefully tuned for scalability, as in its first version Recruiter used heavier findAndModify operations which is now free of.

Property-based testing

Testing some happy paths such as adding a job and executing it is fine for test-driving the code, but it's nowhere near enough for quality assurance. On system of any appreciable scale and/or quality, testing is a separate additional activity (that can hopefully be performed by developers wearing a different hat, or in any case inside a single cross-functional team.)
To test highly concurrent processes such as a recruiter and its dozens of workers insisting on the same database, we adopted Eris, the open source PHP QuickCheck implementation developed by me and some colleagues. Eris is able to generate random inputs for the System Under Test, according to a specification provided by the tester; it supports property-based testing which drives the system with this input while checking important properties are respected.
In this scenario, we generated a random sequence of actions to perform over these processes, checking invariants and post-conditions of operations. For example, an invariant is there is never more than one recruiter process alive. There are surprisingly few invariants when you work with distributed systems; as another example consider the number of workers registered in the related MongoDB collection. This number is not fixed, as crashed processes may still be present even if dead, as long as the rest of the system didn't detect the crash yet.
One postcondition of the job system is very important: any job enqueued is eventually executed, preferably as soon as possible. In these tests, we focused on testing the correctness of this property and not the performance. We monitor the collection of archived jobs (which have been executed correctly) and check that it fills up with all the jobs we expect. The timeout after which we declare the test failed is tuned to the total number of actions performed, which is random.
There are more advanced approaches such as generating a sequential prefix plus a few parallel sequences of actions. This could give more control over the process and may enable some form of shrinking with better determinism; however we retain a notion of parallelism by creating multiple processes. Unfortunately each run is non-deterministic as processes and the underlying MongoDB instance can be scheduled differently by the operating system, changing the interleave of their operation; therefore shrinking is not possible, or is possible only at the cost of running shrunk sequences multiple times to reliably mark them as passing.

Iteration: random number of jobs, graceful restarts

In the first version of the test, we generated a random number of identical jobs (executing a "echo 42" command), along with a series of restarts of the recruiter and a single worker process using SIGTERM. The jobs were enqueued serially by the test process, along with the restart actions. In theory, the processes intercept the signals and exit after having finished their current cycle of polling or execution.
Here are the bugs that we found:

• Jobs locked but not assigned. Since two different collections are at play for scalability reasons, one of jobs and one of workers, a worker may terminate after it is selected for a job but before it is written into its collection. https://github.com/onebip/recruiter/commit/ef490f4acde1b7d00d9b40aee5b2f02257f67d70
• Worker timeout. Its polling should stay at the maximum frequency when there are jobs being executed and only back off when no jobs are found. https://github.com/onebip/recruiter/commit/be13df0d818536bce45050495be1183d109a504d
• Recruiter high latency (>45s) in edge cases caused by the maximum back off increasing too much. This is not strictly a correctness problem but solving it improved the reliability of tests, which are timeout-based. https://github.com/onebip/recruiter/commit/8bb11f5c243ebce834eaeadbc1c5c98815581b1d
• Active Record overwriting the worker document inside its collection when polling, possibly losing the update of the last assigned job. https://github.com/onebip/recruiter/commit/0c950595a475f36778641f49c8b369135a329f8a
• Signals should be handled before interacting with the database, so that graceful shutdowns can be performed even in the startup phases of the recruiter and worker processes (same commit as before). https://github.com/onebip/recruiter/commit/0c950595a475f36778641f49c8b369135a329f8a
• Worker retiring before executing an assigned job. https://github.com/onebip/recruiter/commit/8b23399b567db23d368c4622063eb80db8e8666a
• Error handling code that was never executed, referencing a not existent class name. https://github.com/onebip/recruiter/commit/e8f85ddea7c574991a597b8a386e1ebfe72666d6

Iteration: multiple workers

Once the test suite was consistently green, we extended the testing environment by allowing multiple workers to be created and correctly restarted.
We found an additional problem with this extension:

• Assigned jobs must be a numeric array. When passing an array of job assignments it can get the form [0 => ..., 1 => ..., 3 => ...] and the missing index (coming from a deletion) makes it invalid for a MongoDB query, which fails. https://github.com/onebip/recruiter/commit/2b9af50985542caddb3395a80ad0131c0f666bec

Iteration: crashing workers

We added the possibility of killing a worker with SIGKILL, immediately interrupting it even in the middle of database updates.
The possibility of a worker crashing was already covered by the code. However, we tuned the timeout period after which workers are considered dead while inside the test suite; we set it to dozens of seconds instead of half an hour to allow for sane waiting periods in the test process.

Iteration: crashing the recruiter

Killing the single recruiter process was interesting because it usually takes a lock (in the form of a document inside a MongoDB collection with a unique index) to avoid accidental multiple executions. The process correctly waited on the previous lock to expire before restarting, but...

• An off-by-one error bug in the related MongoLock class. A lock could be waited upon but then failing on the acquisition attempt without any other process try to steal it. https://github.com/onebip/onebip-concurrency/commit/d7af1a37a5ae2c17e77a9545006c6abeb8cf95e3

Iteration: length of jobs

We introduced also a random length for enqueued jobs (sleeping from 0ms to 1000ms instead of executing a fixed command). At this point we did not find additional bugs at the time of this post, with the test suite running for several hours, exploring new random possible sequences of actions.

Final version

The final version of the test composes an Eris Generator that:

• generates a number of workers to start between 1 and 4.
• using this number, creates a new Generator that produces a tuple (in this case a pair, which means an array of two elements of disparate types). The tuple contains the number of workers itself and the list of actions.

The list of actions is a sequence of a random number of elements, where each of the elements can in turn be an action representing:

• a job to enqueue with an expected duration of a positive number of milliseconds
• a graceful restart of one of the workers
• a graceful restart of the recruiter
• a kill -9 on one of the worker processes
• a kill -9 on the recruiter process
• a sleep of a number of milliseconds between 0 and 1000

Here is an example of action sequence:

[ACTIONS][PHPUNIT][2016-03-14T12:11:14+01:00] ["enqueueJob",8]
[ACTIONS][PHPUNIT][2016-03-14T12:11:14+01:00] "restartRecruiterGracefully"

While here is a moderately complex example:

[ACTIONS][PHPUNIT][2016-03-14T12:11:59+01:00] ["restartWorkerByKilling",0]
[ACTIONS][PHPUNIT][2016-03-14T12:11:59+01:00] ["restartWorkerByKilling",0]
[ACTIONS][PHPUNIT][2016-03-14T12:11:59+01:00] "restartRecruiterGracefully"
[ACTIONS][PHPUNIT][2016-03-14T12:12:00+01:00] ["enqueueJob",7]
[ACTIONS][PHPUNIT][2016-03-14T12:12:00+01:00] ["restartWorkerByKilling",0]
[ACTIONS][PHPUNIT][2016-03-14T12:12:00+01:00] ["enqueueJob",13]
[ACTIONS][PHPUNIT][2016-03-14T12:12:00+01:00] "restartRecruiterByKilling"
[ACTIONS][PHPUNIT][2016-03-14T12:12:00+01:00] ["restartWorkerGracefully",0]
[ACTIONS][PHPUNIT][2016-03-14T12:12:00+01:00] ["restartWorkerByKilling",0]
[ACTIONS][PHPUNIT][2016-03-14T12:12:00+01:00] "restartRecruiterGracefully"
[ACTIONS][PHPUNIT][2016-03-14T12:12:10+01:00] ["sleep",860]
[ACTIONS][PHPUNIT][2016-03-14T12:12:11+01:00] "restartRecruiterByKilling"
[ACTIONS][PHPUNIT][2016-03-14T12:12:12+01:00] ["restartWorkerByKilling",0]
[ACTIONS][PHPUNIT][2016-03-14T12:12:12+01:00] ["enqueueJob",0]
[ACTIONS][PHPUNIT][2016-03-14T12:12:12+01:00] ["restartWorkerByKilling",0]
[ACTIONS][PHPUNIT][2016-03-14T12:12:12+01:00] ["enqueueJob",5]
[ACTIONS][PHPUNIT][2016-03-14T12:12:12+01:00] "restartRecruiterGracefully"
The parameter in the steps modelled as arrays is the duration of a job, or the number of the worker in case of restarting actions.

The test generates 100 of these sequences (this number is tunable, or can target a time limit). For each of them it creates an empty database, starts the workers and the recruiter, performs the actions and waits for all jobs to be performed. If the timeout for full execution expires, the test is marked as failed and lists the log files to look at to understand what happened. On my machine, the test now terminates in about one hour, with a green bar.

Conclusions

Testing is an important activity and can increase the quality of your software by removing bugs before they can get to one of your customers. Testing is becoming more and more incorporated in the lifes of developers (see Test-Driven Development and Behavior-Driven Development), but for core domains and infrastructure additional activities are required for stress and performance tests comparable to production traffic.
It is however impossible to write by hand tests for all the possible situations; however you can easily build a reasonable model of the input to your system. So let me quote John Hughes in saying "Don't write tests. Generate them"; with property-based testing you can write one test containing one property, and catch dozens of bugs like in this post's case study.

PHPUnit_Selenium 2.0.0 is out

Here is the text of the change I have just merged to make a new major version of PHPUnit_Selenium a reality:

As signaled in #351, there are incompatibilities between the current version of PHPUnit_Selenium and PHPUnit 5.
It is a losing proposition to still support Selenium 1 API (SeleniumTestCase), as it redefines methods that have even changed signatures. It has not been maintained for years.
So to support PHPUnit 5.x there will be a new major version of this project, 2.x. The old 1.x branch will remain available but not updated anymore.
2.x will contain:

• Selenium2TestCase

and work with PHPUnit 4.x or 5.x, with correspondent PHP versions.
1.x will contain:

• SeleniumTestCase
• Selenium2TestCase

but will only work with PHPUnit 4.x, with correspondent PHP versions. In general, it will not be updated anymore. 

Supported PHP versions vary from 5.3 (!) to 5.6, according to the PHPUnit's version requirement.
Installation is available through Composer, as before.

Property-based testing primer

I'm a great advocate of automated testing and of finding out your code does not work on your machine, 30 seconds after having written it, instead of in production after it has caused a monetary loss and some repair work to be performed. This is true for many different kind of testing, from the unit level (which has also benefits for internal quality and design feedback) to the acceptance level (which ensures the stakeholders get what they need and documents it for the future). Your System Under Test can be a single class, a project or even a collaboration of [micro]services accessed through HTTP from another process or machine.

However, classic test suites written with xUnit and BDD styles have some scaling problems they hit when you want to exercise more than some happy paths: 

• it is difficult to cover many different inputs by hand-writing test cases, so we stick with at most a dozen of cases for a particular method.
• There are maintenance costs for every new input we want to test: each need some assertions to be written and be updated in the future if the System Under Test changes its API.
• We tend not to test external dependencies such as the language or libraries, since we trust them to do a good job even if their failure is our responsibility. We chose them and we are deploying our project, not the original authors which provided the code "as is", without warranty.

Note that here I define input as the existing state of a system, plus the new input provided by a test (the Given and When part of a scenario) and output as not only the actual response produced by the System Under Test but also the new state it has assumed.

sort()

Let's take as an example a sort() function, which no one implements today except in job interviews and exercises.
Assuming an array (or list, depending on your language), we can produce several inputs for the function like we would do in a kata:

• [1, 2, 3]
• [3, 1]
• [3, 6, 5, 1, 4]

and so on. When do we stop? Maybe we also need some tricky input:

• []
• [1]
• [1, 1]
• [2, 3, 5, 6, 8, 9, 1, 3, 6, 7, 8, 9]

Once we have all the inputs gathered, we need to define what we expect for each of them:

• [1, 2, 3] => [1, 2, 3]
• [3, 1] => [1, 3[
• [3, 6, 5, 1, 4] => [1, 3, 4, 5, 6]
• [] => []
• [1] => [1]
• [1, 1] => [1, 1]
• [2, 3, 5, 6, 8, 9, 1, 3, 6, 7, 8, 9]  => [1, 2, 3, 3, 5, 6, 6, 7, 8, 8, 9, 9]

You can do this incrementally, growing the code one new test case at a time, but you have to do it anyway. Considering this can become boring and error-prone in this toy example makes you wonder what to do when instead of sort() you have a RangeOfMoney class which is key to your business and manages point-like and arbitrary intervals of monetary amounts (true story).

Property-based testing in a nutshell

Property-based testing in an approach to testing coming from the functional programming world. To solve the aforementioned problems (and get new, more interesting ones), it follows these steps:

1. generate a random sample of possible inputs.
2. Exercise the SUT with each of them.
3. Verify properties which should be true on every output instead of making precise comparisons.
4. (Optionally) if the properties verification failed, possibly shrink to find a minimal input that still causes a failure.

How does this work for the sort() function?

We can use rand() to generate an input array:
This array is composed by natural numbers (Gen\nat) and it is long up to 100 elements (Gen\pos(100)), since very long arrays could make our tests slow.

Then, for each of these inputs, we exercise sort() and verify a simple property on the output, which is the order of the elements:
This is not the only property that sort() maintains, but it's the first I would specify. There are possible others:

• every element in the input is also in the output
• every element in the output is also in the input
• the length of the input and output arrays are the same.

Here is the complete example written using Eris, our extension for PHPUnit that automates the generation of many kinds of random data and their verification.

How to find properties?

How do we apply property-based testing to code we actually write every day? It certainly fits more in some areas of the code than in others, such as Domain Model classes.

Some rules of thumb for defining properties are:

• look for inverse functions (e.g. addition and substraction, or doubling an image in size and shrinking it to 50%). You can use the inverse on the output and verify equality with the input.
• Relate input and output on some property that is true or false on both (e.g. in the sort() example than an element that is in one of the two arrays is also in the other)
• Define post conditions and invariants that always hold in a particular situation (e.g. in the sort() example that the output is sorted, but in general you can restrict the possible output values of a function very much saying it is an array, it contain only integers, its length is equal to the input's length.)

[2, 3, 5, 6, 8, 9, 1, 3, 6, 7, 8, 9] makes my test fail

Defining valid range of inputs with generators and the properties to be satisfied is a rich description of the behavior of the System Under Test. Therefore, when a sort() implementation fails we can work on the input in order to shrink it: trying to reduce its complexity and size in order to provide a minimal failing test case.

It's the same work we do when opening a bug report for someone else's code: we try to find a minimal combination that triggers the bug in order to throw away all unnecessary details that would slow down fixing it.

So in property-based testing the [2, 3, 5, 6, 8, 9, 1, 3, 6, 7, 8, 9] can probably be shrinked to [2, 3, 5, 6, 8, 9, 1, 3, 6, 7, 8] and maybe up to [1, 0], depending on the bug. This process is accomplished by trying to shrink all the random values generated, which in our case were the length of the array and the values contained.

Testing the language

So here's some code I expect to work:
This function creates a PHP DateTime instance using the native datetime extension, which is a standard for the PHP world. It starts from an year and a day number ranging from 0 to 364 (or 365) and it build a DateTime pointing to the midnight of that particular day.

Here is a property-based test for this function:
We generate two random integers in the [0. 364] range, and test that the difference in seconds of the two generated DateTime objects is equal to 86400 seconds multiplied by the number of the days passed between the two selected dates. A property of the input (distance) is maintained over the output in a different form (seconds instead of days).

Surprisingly, this test fails with the following message: what happened is we triggered a bug of the DateTime object while creating it with a particular combination of format and timezone. The net effect of this bug could have been that our financial reports (telling daily revenue) would have started showing the wrong numbers starting from February 29th of the next year.

Notice that the input is shrinked to the simplest possible values that trigger the problem: January 1st on one value and March 1st on the other.
Eventually we found a easy work around, as with a couple more lines of code we can avoid this behavior. We could do that only after discovering the bug of course.

In conclusion

Testing an application is a necessary burden for catching defects early and fix them with an acceptable cost instead of letting them run wild on real users. Property-based testing pushes automation also in the generation of inputs for the System Under Test and in the verification of results, hoping to lower the maintanance cost while increasing coverage at the same time.

Given the domain complexity handled by the datetime extension, it's doing a fantastic job and it's being developed by very competent programmers. Nevertheless, if they can slip in bugs I trust that my own code will, too. Property-based testing is an additional tool that can work side by side with example-based testing to uncover problems in our projects.

We named the property-based PHPUnit extension after Eris, the Greek goddess of chaos, since serious testing means attacking your code and the platform it is built on in the attempt of breaking it before someone else does.

References

• Eris on Packagist
• Eris examples on GitHub
• The tools that inspired Eris from other languages: QuickCheck, ScalaCheck, Erlang QuickCheck.

Eris 0.4.0 is out

Eris is a PHPUnit extension for property-based testing, that is to say testing based on generating inputs to a system and check its state and output respect a set of properties. The project is a porting of QuickCheck and Eris is the name of the Greek goddess of chaos, since its aim is to break the System Under Test with unexpected inputs and actions.

I am planning a talk at the PHP User Group Milano and a longer blog post to introduce the general public to how property-based testing works. I held the same talk for a few friends at the phpDay 2015 Unconference.

Meanwhile version 0.4.0 is out, with the following ChangeLog:

• Showing generated input with ERIS_ORIGINAL_INPUT=1.
• names and date (DateTime) new Generator.
• tuple Generator supports variadic arguments.
• Shrinking respects when() clauses.
• Dates and sorting examples.

As for all semantic-versioned projects, the 0.x series should be considered alpha and no API backward compatibility is guaranteed on update.

Image credits

Tabular data in Behat

All of this has happened before, and all this will happen again. -- BSG

I just watched Steve Freeman short talk "Given When Then" considered harmful (requires free login), and I was looking for some ways to cheaply eliminate duplication in Behat scenarios.

Fortunately, Behat supports Scenario Outlines for tabular data which is an 80/20 solution to transform lots of duplicated scenarios:

    Scenario: 3 is Fizz
        Given the input is 3
        When it is converted
        Then it becomes Fizz

    Scenario: 6 is Fizz too because it's multiple of 3
        Given the input is 6
        When it is converted
        Then it becomes Fizz

    Scenario: 2 is itself
        Given the input is 2
        When it is converted
        Then it becomes 2

into a table:

    Scenario Outline: conversion of numbers
        Given the input is <input>
        When it is converted
        Then it becomes <output>

        Examples:
            | input | output |
            | 2     | 2      |
            | 3     | Fizz   |
            | 6     | Fizz   |

Moreover, you can also pass tabular data to a single step with Table Nodes:

    Scenario: two items in the cart
        Given the following items are in the cart:
            | name    | price |
            | Cake    |     4 |
            | Shrimps |    10 |
        When I check out
        Then I pay 14

It takes a few minutes to learn how to do this into an existing Behat infrastructure. There are minimal changes to perform in the FeatureContext in the case of the Table Nodes, while Scenario Outlines are a pure Gherkin-side refactoring.

My code is on Github in my behat-tables-kata repository. If this reminds you of PHPUnit's @dataProvider, try to think of other patterns that can be borrowed from the xUnit world to fast-forward Cucumber and Behat development.

PHPUnit Essentials review

PHPUnit Essentials by Zdenek Machek is a modern and complete book about PHPUnit usage. I've been sent an electronic copy by Packt Publishing and am now reviewing it here.

The first thing that struck me about the book was the breadth of subjects: you start from mocks and command line options, to get even to Selenium usage. You have to know your tools and given PHPUnit being a standard, this is all knowledge that will accompany you for several years.

Every book on PHPUnit must be compared with the wonderful manual, to see what it adds to the picture with respect to the documentation. PHPUnit Essentials, in this respect, looks also at 3rd party libraries such as mocking libraries or "competitors" such as PHPSpec to enlarge the picture to the whole open source PHP landscape. This is something the documentations of single projects cannot do, and where a bit of opinionated advice can be taken.

There is a bit of what may seem outdated information in the book such as how to perform a PEAR-based installation, but it's identified as such (PEAR being deprecated and dismissed by the end of the year.) Another seemingly outdated tool is Selenium IDE, but once upgraded with a formatter for Selenium2TestCase like explained in this book it becomes usable again. This kind of advice demonstrates the real world experience of the author and makes you trust the content.

On the whole by reading this book you go in as a naive tester and you come out with lots of skills on using PHPUnit in different scenarios; so I would recommended it to programmers wanting to dive into testing PHP applications. Probably it's not worth a read for the medium-to-advanced users, for which most of the content is already known from PHPUnit manual or personal experience. After all the book's named Essentials, so it delivers all that you expect from the title in a convenient single package.

Weekly roundup: PHP.TO.START

I'm packing up some gear (and some dinner) for my trip to Torino for the PHP.TO.START. The path from Porta Nuova station to the event is in the most central area of the city, so I'll take a walk in the morning; maybe I'll try the metro for the return trip, which (unlike the Milan one) is very modern and completely automatic.

Here are my original articles published during this week.
Practical PHP Refactoring: Remove Assignments to Parameters is a small scale refactoring to ensure the values of parameters remain invariants of a method.
Testing JavaScript when the DOM gets in the way uses jsTestDriver for testing JavaScript code which manipulated divs, inputs and other HTML elements.
Practical PHP Refactoring: Replace Method with Method Object describes how to extract a too complex method as an object of its own.
The era of Object-Document Mapping is a snapshot of the current Doctrine tools for mapping PHP objects to a NoSQL database.

PHP.TO.START - July 11th

I will be in Torino on Monday for PHP.TO.START, a day-long event oriented to PHP businesses and university students. The event would be in Italian and has been created by Indigeni Digitali and Skuola.net.
Who wants to work with EJBs if you can work with PHP? :)

Subscription is mandatory and closes on Sunday. I will make the trip with the 8:00 AM Freccia Rossa if you're coming from Milan.

DPC 2011 slides

Here are my slides for the Testing in isolation tutorial. A link is included pointing to the Git repository, this time with tags. :)

Testing in isolation

These are the slides for the Domain-Driven Design talk (similar to the phpDay 2011 version). Link to the Git repository is included too.

Pursuing practices of Domain-Driven Design in PHP

Weekly roundup: back from DPC 2011

I am back home from Amsterdam, after having attended the Dutch PHP Conference 2011 and having presented my tutorial on isolation of tests and a talk on Domain-Driven Design.
Which, in a different way from the phpDay 2011 version, was liked by some attendees and also widely criticized by many. Thanks to the many attendees who took the time to give feedback on joind.in, as it's difficult for a speaker to find out the problematic areas which he has to focus on (more on that in the next days). I commonly express myself with writing and addressing a crowd is very different in many aspects.

While I was at the conference, these original articles were published on DZone.
Practical PHP Testing Patterns: Dependency Injection
The 4 rules of simple design
Practical PHP Testing Patterns: Dependency Lookup
Git backups, and no, it's not just about pushing

Packing for DPC 2011

I'm leaving in the afternoon from Milano Linate to Amsterdam in order to participate to the Dutch PHP Conference.

I will present:

• Isolation for your tests, a 3-hour tutorial on how to isolate your code from other objects, libraries or external services, and in particular databases.
• Pursuing practices of Domain-Driven Design in PHP, a 45-minute talk on DDD and how to end the fight between PHP frameworks, ORMs and your object model.

See you at #dpc11!

Weekly roundup: back from phpDay 2011

Yesterday's evening I arrived home from the phpDay, and I have lots of things to share with you.

First of all, the slides of my two presentations.

An year of Pomodoros

Pursuing Domain-Driven Design practices in PHP

And then, four of my articles published this week.
Practical PHP Testing Patterns: Table Truncation Teardown shows the first option for resetting a relational database between tests: deleting everything from the tables but maintaining the schema for speed.
PHP UML generation from a live object graph shows you a bit of code I written in order to generate an Uml class diagram from a group of instantiated PHP objects.
Practical PHP Testing Patterns: Transaction Rollback Teardown shows you another option for resetting a database in tests: rolling back a transaction.
Bleeding edge JavaScript for object orientation contains the new features which will be available in the generation of browsers containing Firefox 4 and Internet Explorer 9.

Interview on Voices of the Elephpant

There is an interview with me, recorded with Cal Evans, on the Voices of the Elephpant podcast. You can enjoy my Italian accent and my random pronunciation of injection. :)

Inside: design for testability, the Shower Methodology, test maintainability.

The new PHP TARDIS Api

Yeah, from PHP 5.4 and further you will be finally be able to access your TARDIS from PHP code:

$t = new Tardis("192.168.0.4"); // the IP address of the TARDIS
$t->configureOwner(11); // only numbers from 1 to 11 supported
                        // but we'll eventually get to 13

You see the innovation here? PHP is providing a new extension with an object-oriented Api instead of a bunch of tardis_*() functions which take as parameters strings, arrays, integers and resources. Indeed we have jumper into the future.
The good news is spreading all over Twitter and in PHP mailing lists, as frameworks and hosting services will never be the same thanks to the use of TARDIS (so much for the Java platform being more powerful for integrating external software.)
What are the advantages of such an Api, directly available as a core extension of PHP? Here's my take.

Translation of pages

Homepage of TARDIS-powered Google Deutsch.

You can access the telepathical hooks of the TARDIS to write your pages only one time, and then translate automatically the content by linking the device to the minds of your users.
Make sure to display an EULA, or some I Accept checkbox, before diving into this feature. However, i18n is finally solved!
Disclaimer: as you can see in the example on the right, it does not work with images for now. It seems that translation of image-contained text is covered by a patent that Gallifreyans couldn't obtain.

Dimensional trascendence
As you know, TARDIS interior is in a different dimension from the exterior appearance. This space is suitable for server clusters and other technology: for example by moving your TARDIS to your server room, you will have an enormous capacity for scaling as it can contain an huge amount of racks.
You may have to install some additional cooling mechanism, however.

Chameleon Circuit

My home servers

You can rent rack space from other companies, but how can you make sure they not fiddle with your servers? You can simply link them to the chameleon circuit of the TARDIS. Their appearance will merge with the background, and no one will be able to find them without your permission.
But that's not all: you can also link newly spawned virtual machines to it via the PHP Api, so that they effectively disapper in the cloud and remain unreachable from attackers!

Teleportation
You can now physically access your production servers in case something is wrong and teleport them to the colocation site for production.
When this PHP script is executed (it can be chained to the failure of tests or to repeated failures of the db connection, or to the detection that someone wants to access your private customer data), the servers linked to the TARDIS are teleported back:

$tardis = new Tardis(...);
$tardis->query('SELECT * FROM servers
        WHERE failed IS NOT NULL')
       ->fetchAll()
       ->teleport('22031', 'Italy'); 

Of course substitute my parameters with your ZIP code and country. And yes, the TARDIS speaks SQL.

My conclusions
PHP is going to be revolutionized and empowered by the TARDIS integration. You'd better hurry up and join a startup based on this new technology - visit the project page for more information on TARDIS-enabled PHP.

Weekly roundup: new semester

I'm starting a new semester at PoliMi tomorrow. I did not really have a choice on the selection of courses for this round, but the subjects seem nevertheless interesting. Still, I have to work on my computer vision project to wrap up the first semester, and my PHP work time will be greatly reduced.
Some topics I will encounter in the next months and that will probably influence my articles are:

• Java and non-Java web development with frameworks (don't yet know which particular technologies)
• Physical architectures (clustering, parallelism, distributed systems and similar)
• Performance evaluation (how many concurrent users can this machine sustain?)
• Machine learning and model estimation (like how do you get a machine to recognize human faces)

Anyway, here are my original articles published last week.
Practical PHP Testing Patterns: Implicit Teardown is the last type of teardown for test suites, which is inserted in hooks provided by PHPUnit.
SOLID for packag... err, namespaces explains the 6 principles which extend the SOLID ones to packages instead of classes. In PHP, I think namespaces will become somewhat equivalent to binary packages in other languages.
Practical PHP Testing Patterns: Test Double introduces the various types of Test Doubles like Stub, Spy and Mock. Each will have its own dedicated article in the series.
What you must know about PHP errors to avoid scratching your forehead when something goes wrong is exactly what it says on the tin.