How to TDD a database application

After the publishing of Failed attempt to apply Tdd response, some readers have asked me how to test a database application, as an example of code which is not suitable for xUnit frameworks.

Let's start with saying that database is not a special case for testing, as it's only a port of your application. Whenever the application interact with an external program I would say it presents a port, and this pattern is called Hexagonal Architecture.
Your effort should be in testing thoroughly the core of the application, building a solid object-oriented Domain Model piece after piece, by writing a test at the time an making it pass. The Domain Model should not have dependency on any infrastructure like database, http requests, twitter adapters, and so on: this refinement of the Domain Model pattern is called Domain-Driven Design and when applied produces easy testable code. The Domain Model would be tested by injecting fake adapters as there should be no logic in the database: this is object-oriented programming and in database object do not exist nor we can test it easily and automatically with Junit.
Though, this approach is very complicated and it is considered when the domain has a rich set of rules and behavior, while many applications have only CRUD capabilities.

Think of your application as existing only in Ram memory and strip out all the unnecessary code. The classes which remains are the core of the Domain Model. For instance if I had to manage the list of the users of a forum, I would write initially only a User class and a generic collection of User objects. User in my point of view is a POJO or POPO, which does not extend anything:
class User { ...

Insulating the database from view will hide this generic collection behind an abstraction, since it is not present at all in memory except for caching. Subsets of it can be reconstituted as needed. This abstraction is called a Repository or, at a low level, a DataMapper. Hibernate for Java and Doctrine 2 for php are example of a DataMapper pattern: they let you work on your objects and then take them and synchronize the database with the new data you have inserted: a change in a User's password, or new Users to be added. To polish the DataMapper Api, which is very generic, a UserRepository class can be created.
Even if you do not have a generic DataMapper, and work with mysql_query, PDO or JDBC queries, you can write a UserRepository which will act as the port for the database (or maybe a text file; since the repository decouples it, the storage mechanism can be anything from a memcache server to a group of monkeys which writes down on paper serializations of objects).
Depending on your architecture, your controllers or other domain classes will now have the UserRepository as a collaborator, and will talk to it and call its methods instead of accessing the database directly; this is a form of Dependency Injection. Obviously if there are other entities which are persisted, like Groups, Posts, etc. they should have their own repository class.

Of course the point of this discussion is how to test this code, since to write it we have to prepare a test before (it's called red green refactor and not code and then try to test). If we manage to write these tests first, the code will be automatically testable, and very decoupled and reusable as this is a characteristic of testable code.
What we need to write are not tests, but unit tests. If you want to check the entire path of data in the application, you can write integration tests which will exercise even the user interface, but Test-Driven Development prescribes to write unit tests: your test methods should have a dependency only on the system under test, and to the interface it uses.
Continuing our example, we need:

• unit tests for User, Group, Post classes (Domain Model entities);
• unit tests for controllers or other classes from the Domain Model which uses the adapters.
  
• unit tests for UserRepository and similar classes (adapters);
• unit tests for DataMapper or (infrastructure);
  

The point of unit testing is you can test singular components separately, and not the entire application as you would do with functional testing. Moreover, you must test each component separately or otherwise it would reveal too difficult to write classes to satisfy an integration test and the TDD benefits of adding a bit of design with every test method would be lost.
With this knowledge, we can say:

• unit tests for Domain Model entities must be written by the developer of this application. If the project is mostly a CRUD application and is data-intensive, these test cases will be very short.
  
• unit tests for controllers and everything that uses the adapters also must be written by the developer, mocking the adapters out. If you use the real adapters in testing, it will be integration testing and it will be heavy and brittle; you won't know if it's the controller or the adapter which does not work after a red test.
• unit tests for the adapters concrete classes: this is the only interaction with the database. Fortunately, we are unit testing for this classes so we can even use a new database for every test method as every class will present a few methods; compare this approach with testing every single feature of the application by using a real database.
• unit tests for infrastructure: these are included in the framework so we shouldn't worry.
  

Note that if you use a DataMapper which abstracts the particular database vendor as infrastructure, you can run the unit tests for the adapters by using a in-memory database like sqlite instead of the real database which will likely be very heavy to manage and recreate every time; this way, you de facto exclude database from your unit tests. Of course some integration tests will be necessary, but they are not part of TDD and of the design process.

I hope to have given you an idea of what unit tests means and how to deal with an application which uses a database for persistence, which is a very common scenario. Feel free to raise questions in the comments if something is not clear.

Do you want more? There's a book for .NET developers which explains DDD and database-independent testing.
The image at the top is a photograph of integrated circuits, which are real world components designed for testability and which are tested indipendently from the card where they will work.

Applying Domain-Driven Design and Patterns review

Applying Domain-Driven Design and Patterns: With Examples in C# and .NETis a book from Jimmy Nilsson which shows you how to begin applying such things as TDD, object relational mapping, and DDD to .NET projects...techniques that many developers think are the key to future software development. Although it is based on .NET technology and features C# examples, it is a valuable resource for DDD adopting in every object-oriented language and one of the few practical books on Domain-Driven Design, along with the original one from Eric Evans.

The main feature of this book is a full description of the process followed in development: previous solutions are presented along with improvements to the object-oriented design and the reasons that have lead to refactoring. For instance, persistence ignorance is not assumed as the basis of a decoupled model but it is the final point of a research path.
This is a panoramic of the topics which the author deals with in the 600+ pages book.

Part 1: Background
The first part of the book is a introduction on low-level techniques like TDD, refactoring, and application of patterns (for example, State pattern). Problem and tradeoffs are discusses regarding databases and their impedance mismatch with an object-oriented model, together with the importance of layering and modelling. Part 1 is important because understanding of the rest of the book strongly depends on these concepts: a responsible programmer should test first and refactor where he expects changes.

Part 2: Applying DDD
In part 2 a full application requirements are discussed along with part of their implementation. Since the title is applying DDD the focus is on building the right model for the example, leaving out infrastructure and security concerns for a moment.
The requirements list comprehends orders and customers management, concurrency conflict detection, validation rules on amount of moneys in orders and customers, atomic saving and validation with an external service. This list is well crafted as it contains the main problems encountered when using a rich Domain Model, so the order management application is a good example to see DDD applied at its best.

Part 3: Applying PoEaa
While part 2 is centered on a persistent-ignorant model, part 3 discuss the infrastructure needed for the example application to work (object/relational mapping in primis) and tradeoffs from the incorporation of database concerns in the pure domain model. PoEaa is Patterns of Enterprise Application Architecture,a famous book from Martin Fowler which presents the principal patterns used in orms and big applications.
PoEaa is full of code examples so it is not important for the author to attach naked code but the discussion on how to persist a rich model is still crucial.

Part 4: What's next?
This part introduces other techniques and topics which are not strictly related to modelling and infrastructure: Service-oriented architecture, Inversion of Control and Dependency Injection, Aspect-oriented programming, Model-View-Controller pattern, user interface development. These are all interesting topics which a smattering of can only help the average developer to enlighten himself and face different ideas and paradigms: TDD for instance needs Dependency Injection to work.

Part 5: Appendices
Last but not least, other styles are proposed in contrast to DDD, such as database-driven development and rich service layers. The same application requirements from part 3 are rediscussed using these new point of views, and the reader gets a comparison between various implementation of the Domain Model pattern.
The appendices contain also a list of the patterns referenced in the book, but the should consider other sources for more information.

Summing up, Applying DDD is a good panoramic on practicing Domain-Driven Design, but also on implementing patterns and using modern tools and approaches like Dependency Injection and Test-Driven Development. It can also be read along or after the original DDD book to improve the understanding of Factories, Repositories and other domain patterns.

Naked objects, DDD and the user interface

The SeparatedPresentation pattern is used to abstract the user interface from the domain model of an application. But it isn't more useful to enrich the user with a view of the domain itself?
The MVC pattern is all about separating the presentation of data. The major frameworks like Zend Framework and Rails are built to take advantage of an MVC stack, which means write your models, controllers and views.
Tipically we have a Post model with a PostController class that governs the operations that can be performed on the model and which views to show.
But there's more than MVC in the world.
What I'm talking about is Naked Objects pattern, that is currently implemented in the Java and .Net world by a framework with the same name. This pattern can be the next paradigm shift in development: in short, the V and C from MVC are completely generic and are built from the M part, the model classes. Also the infrastructure part, like database persistence of entities, are taken care by the framework that includes an Orm in its libraries.
Let's discuss the advantages of this approach:

• there's no mapping between concepts, as the user interface is automatically created on-the-fly or generated and the user gains insight on the model (and thus on the domain). A driver aware of the fact that there is an engine will use the accelerator at its best. This helps an Ubiquitous Language to take shape.
• there's no views and controllers to write, obviously.
• there's no infrastructure coupling, also obviously if we exclude the annotations or xml metadata needed to make Hibernate or another Orm work. ActiveRecord is a reminescence of the past.
• there's no logic in the controllers: they are automatically generate and delegates to the model. Fat model, skinny controller is achieved by force.
• the domain model will be built without caring much of the other parts because it is the only thing to write. This allows a DDD approach, and the only attention is to conform to the conventions of the framework.
• no more repeated validation on the ui: it is simply delegated to the model, where it should reside.
  
• if the generated user interface does not satisfy you, what remains to you is a powerful and complete domain model that can be used to write an MVC application (thus using naked objects pattern only to prototype).
  

There are also issues of giving a complete user interface that cannot be edited by hand: this is not scaffolding. In the next posts I will talk about my journey in learning to use Naked Objects framework for Java, which is distributed as open source software.