Slides from CouchDB university presentation

As part of the Advanced Database course at Politecnico di Milano, I have done a short presentation on CouchDB, a NoSQL database. Here are the slides, in case they are of interest to you.

CouchDB @ PoliMi

View more presentations from piccoloprincipe.

Repositories in Doctrine

Jebb wrote to me asking information about the Repository pattern:

I read your blog at http://giorgiosironi.blogspot.com/and I found it very helpful. I develop applications using DDD and Zend Framework however for persistence ignorance, I use some basic Repository as interface for DAOs. May I ask you how to implement Doctrine ORM in the Repository?

As I explain in depth in my previous post about this pattern, a Repository is an illusion of an in-memory collection of all persistent ignorant entities of the same class. A Repository may be abstracted beyond an interface defining only the mandatory methods that make sense in the domain model.
How an Orm fits in this discussion? Very few repositories accomplish their tasks alone: most of them compose an underlying generic Data Mapper layer which is usually an Object-Relational Mapper which translates classes and objects in tables and rows. A generic Data Mapper such as Doctrine 2 provides all the possibly imaginable operations on the collection of entities, and the Repository abstraction decouples the other parts of the application from knowing anything about an Orm. This is the primary advantage of a Repository.

As you may have thought at the persistence ignorance reference, with Doctrine 1 there are no chances to implement a real Repository. Doctrine 1 does not provide persistence ignorance because it requires entity classes to extend a base abstract Active Record.
In Doctrine 2, repositories can be implemented as Plain Old Php Objects: simply injecting the EntityManager and other collaborators they may need in the constructor will suffice. A Factory can then encapsulate the new operators.
This is the standard Repository pattern: a Plain Old Php Object which aggregates whatever is needed to hide the persistence mechanism of objects.

Doctrine 2 also provides a facility to quickly implement Repositories, but the freedom of implementation of this approach is limited. The process is described in the Doctrine manual and it consists in:

• extending an abstract class, which has a protected member $this->_em you can execute queries with;
• annotating the entity class with the class name of the concrete repository class;
• obtaining an instance with $em->getRepository('EntityClassName'): the EntityManager will create the object and automatically inject itself in the Repository.

Some notes on this architecture:

• there is no entity-specific interface;
• you extend an abstract class, which may be a cling because you want only some find*() methods to be available. However testing won't be affected since a concrete Repository will always involve a persistence mechanism.
• it would be useful for having different repositories referencing each other via the EntityManager; but you may manually inject them instead, maintaining the original solution.
• Doctrine 2 default Repository class is instantiated instead if you do not specify a subclass; if you use a lot of repositories and inject some of them in other service classes, the default implementation will be very handy. This would be a pro for taking advantage of the Orm support instead of using POPOs.

Where is business logic?

Have you ever heard of Multitier architecture? If not, you have probably encountered it without knowing its name while working on web applications.
In a multitier architecture, an application is divided in different horizontal layers, each addressing a different concern. Every layer builds on the one that lies directly under it to perform its work, thus decoupling for example html presentation (upper layer) from sql queries (lower layer).
The number of layers is flexible and there is a high number of variants for a multitier architecture, but the simplest model many web applications fall in is composed of three layers:

• user interface: generates html, handles user input and displays errors.
• Domain Model: objects and classes that represent concepts, such as Post, Thread, Forum, User, Message and so on.
• Infrastructure: usually data access code to a database and, by extension, the Sql schema itself where the relational model is used. External services also qualify as infrastructure.

Thus there are three major approaches to development, which differ in the layer of the application that contains the greater quantity of business logic. Which User can close a particular Thread and in what order the Messages for a particular User are listed? In which Forum a User can add a Post?
The answers to these questions ideally reside in one of the fundamental layers as specifications require (though sometimes they are scattered trough the layers, which is a very effective way to complicate a design.)

Smart Ui
As the name says, this style keeps the logic in the user interface. A Smart Ui example is a folder full of php scripts that move data back and forth from a MySql database.
During maintenance, usually the replication of rules and code in different scripts increase, rendering difficult to change and expand the application; this style is appropriate only for small projects which only shuffle data from tables to html pages.

Smart database
A style primarily taught in database classes, which result in a very accurate schema, full of constraints, triggers and stored procedures to maintain data integrity.
Note that if you want to implement this approach, you probably need an expensive database like Oracle because open source databases do not support all the logic you need. Moreover, Sql is not a programming language, you can stretch Ddl with proprietary extensions and set up many rules but you will be replicating them in the front-end (if there is one at least) for error handling and localization.

Rich Domain Model
The most powerful approach is implementing logic in the domain model layer, which is the type of model that should be able to best represent the real world.
It follows that in such an approach the Ui delegates nearly everything to the domain layer, or it is even automatically generated (Naked Objects). Technology is available for the database to be generated automatically once a mapping from objects to tables is defined (Orms like Hibernate and Doctrine 2). The dependencies are inverted as all other layers mirror the domain model.
The advantages of a rich domain model are multiple:

• testing is simple because infrastructure and Ui do not get in the way; no need to run databases to test business logic or to fill forms with a bot or Selenium.
• no duplication of logic is permitted, because different views of the user interface refer to the same methods in the domain model.
• the model of the application is the model presented to the user; there is no translation between concepts and no need for him to learn a data model along with a presentational one. Often a Presentation Model is needed because the underlying Domain Model is anemic.

Essentially with the tools available today for managing generic layers you can achieve everything by manipulating objects directly in memory and storing the result in the database by pushing a big Save button.

What everybody ought to know about storing objects in a database

Probably during your career you have heard the term impedance mismatch, which is commonly referred to the problems that arise in converting data between different models (or between different cables if you are into electrical engineering).
Usually the complete expression is object-relational impedance mismatch, which indicates the difficulties of the translation process between two versions of the same domain model: the former resides in memory and it consists in an object graph, while the latter is used for storage and it is a relational model stored in a database. The conversion between parts of both models happens many times while an application runs, and in php's case at least once for every http request.

Object-relational mappers like Hibernate and Doctrine are infrastructure applications which deal with the mismatch, doing their best to implement a transparent mechanism and providing the abstracted illusion of a in-memory model, like the Repository pattern. These particular Orms are the best of breed because they do not force your object graph to depend on infrastructure classes like base Active Records.
The connection between the two models is defined by the developer, by providing metadata about its classes properties: for instance you can annotate a private field specifying the column type you want to use to store its value. But what are the translation rules the developers provide configuration for? Here is a basic set of the tasks an Orm performs for you.

• Entity classes are translated to single tables as a general rule, with a one-to-one mapping. The class private or public fields which are configured for storage define the columns of a particular table.
  
• Objects which you pass to the Orm for being stored become rows of the correspondent table. A User class becomes a User table containing one row for every registered user of your website.
  
• A primary key is defined by choosing between the existing fields or inserted ex-novo. Often as a requirement the developer should explicitly define a field.
  
• Repeated single (or multiple) class fields become new tables, and the problem of representing them is shifted to representing relationships; in the domain model this kind of objects are Value Objects, which is semantically different from Entities, but databases only care about homogeneous data and such objects receive no special treatment.
• One-to-one and many-to-one relationships can be represented with a foreign key on the source entity that resembles the original pointer to a memory location.
• One-to-many relationships are a bit trickier because they require a foreign key on what is called the owning side, in this case the target entity. What can seem strange at first glance is that even if in the domain the relationship is unidirectional (pointer to a collection), elements of a collection need to have a reference to the owner to unequivocally identify it. The mutual registration pattern can be used to build a correct Api starting from this constraint; I will write about it in one of the next posts.
• Many-to-many relationships are managed by creating an association table that references with foreign keys the participating entities. Every row constitutes a link between different objects; sometimes it may be the case to use such a table also for one-to-many associations, to avoid having a back reference field on collection elements.
  
• Inheritance is by far the most complex semantic to maintain as it is not supported at all by relational databases: Single/Class/Concrete table inheritance are three famous patterns which organize hierarchical objects in tables, but I prefer to avoid inheritance altogether if not strictly necessary.

And this list is one of the reasons why in your college courses they told you that a model should be as small and simple as possible: a simple model can undergo much more simple transformations for storage and transmission than a complex one.
Note that some contaminations leak from the database side to the object graph, such as the bidirectionality of one-to-many relationships, present even when it is not required by the domain model.
Orms take care for you of this translation process and can even generate the tables from the classes source code, but they only perform automatically the tedious part of object-relational mapping. You should know very well how the mapping works if you plan to use such powerful tools without reducing your database to a list of key/value pair.

The image at the top is an Entity-Relationship model used to design database schemas. I find it not useful anymore as I now prefer to think in classes terms, with Uml diagrams.

Modern database models

The word database is often heard during a developer's education and a decent database knowlege is considered fundamental. Nearly every serious application relies on a database to work and although nowadays the usual choice is a relational database, understanding other models helps to explore new ways of dealing with data and new, emergent products.

The database word indicates the collection of data saved somewhere in the infrastructure: this blog saves posts in a database, Firefox saves preferences in a little database in your filesystem. When we are talking of MySQL and Sql server, the correct term is Database Management System, or DBMS. A DBMS provides all the procedures and libraries needed to access and modify the database content, abstracting away the need to use low-level filesystem functions.
The abstraction a dbms builds upon its storage engine is called model. There is more than one model for representing data: some have become obsolete and other ones are so widely used that they probably will never disappear in the next twenty years no matter what happens in the database scenario.
Let's start talking about database model. The models I am presenting here are logical models, which specify how information is presented to the user or to the application which talks to the database. This model must be distinguished from the physical model, which consists in the way the dbms chooses to persist data on disks, tapes and other mass memory devices.

Flat model
When you open a spreadsheet or a simple ini file, you are using a flat model. Data are organized in one list or table, with similar elements or rows.
The problem with a flat database is representing the relationships between elements. For instance how do you associate two users that are friends or a user with the groups he chooses to belong to? Although this limitations, flat files and databases are useful because of their data access simplicity where there is the need for lightweight and easily implementable systems, such as config files or spreadsheet saved in the comma-separated values format.

Hierarchical model

The first improvement to the flat model, applied for the first time in the 1960s, is the addition of a pointer to every instance of data (which is called record). This pointer establish a child-parent relationship towards another record, where every one of them has at most one parent.
Information is thus presented in a tree structure, which is a good model for many real world entities. For instance, you can represent too much things in Xml, which is a hierarchical model too. The Dns system and Ldap protocol present a hierarchical model, but they are a specific application of this paradigm and not a dbms-like product.
A variation of the hierarchical model is named network model, which removes the limit of one parent and places a set of many pointers into every record. The tree-like model becomes a graph. This concept dates back to the 1965, so there's no buzz around network models and hierarchical general-purpose databases.

Relational model
The relational model is the widely used one which I was referring to at the start of this post. Relational dbms like MySQL, Microsoft SQL Server, PostgreSQL and even Sqlite constitute the majority of today's applications storage mechanisms.
The relational model describes data in various tables, where each row has a fixed set of fields that form the table's columns. Continuing with our example, a User and a Group tables, with their fields lists, are a relational model.
In this model, relationships between entities are established with the equality of some columns, often named primary keys and foreign keys. If you have not previously lived under a rock, you probably have used these databases a lot so I won't bore you anymore.

Object model
This is when the problem becomes interesting. Data that were presented in a relational model yesterday is being substituted by an object model, built with classes and instances. Maintaining the whole object graph in memory is usually too expensive to be feasible, since it requires enormous amount of resources like memory and cpu cycles to search objects in the mess of a 2-million-objects graph.
To preserve the object model and persists data at the same time, various solution have been proposed in the years:

• Serialize the objects and put the binary stream on disk. Simple, but how do you search a User instance by his nick when he signs in?
  
• Mapping the objects to a relational database, more or less saving every object as a row of a table which corresponds to the class of choice. An Object-Relational Mapper is the tool used for these operations, but it has some limits, for instance in dealing with class inheritance; these limits are known as the impedance mismatch. The Orm is also used for retrieval, ideally abstracting away the relational storage from the application.
• Put the objects in an object database.

So, an object database is a tool built from scratch to persist an object model and to allow retrieval of small parts of the overall graph with procedures similar to the relational one (ordering, selection).
An object database would be great to use in real world, but currently going the Orm route is the standard since relational databases are at the world's center. Data typically don't fall from the sky, and there is a need for synchronization between applications and machines in a relational database. Thus, different object models can work on the same data and even with applications which don't use an object-oriented paradigm.

Document model
Instead of presenting a fixed structure, a dbms can instead show a semi-structured model, where records have no enforced lists of attributes. These type of entities are called documents, and an application or middleware which relies on it can store nearly everything as a document property: the advantage of this technique is that you'll never have to update a schema. An example of a document-oriented, open source database is CouchDB.

The schema-less novelty is one of the last buzzword in the database world, and it's still not clear what will the future of these solutions be. Relational databases are probably here to stay as there is a lock-in from applications all over the world to their data model. Object and document models are often presented as a panacea to improve scalability and simplicity, but they are not a standard at the moment. Try to explore new persistence solution, as the technology changing pace is slow in this field, but it exists.

In the image at the top, a typical relational model for an employees table, with the specification of primary key, fields and foreign keys.

How to TDD a database application

After the publishing of Failed attempt to apply Tdd response, some readers have asked me how to test a database application, as an example of code which is not suitable for xUnit frameworks.

Let's start with saying that database is not a special case for testing, as it's only a port of your application. Whenever the application interact with an external program I would say it presents a port, and this pattern is called Hexagonal Architecture.
Your effort should be in testing thoroughly the core of the application, building a solid object-oriented Domain Model piece after piece, by writing a test at the time an making it pass. The Domain Model should not have dependency on any infrastructure like database, http requests, twitter adapters, and so on: this refinement of the Domain Model pattern is called Domain-Driven Design and when applied produces easy testable code. The Domain Model would be tested by injecting fake adapters as there should be no logic in the database: this is object-oriented programming and in database object do not exist nor we can test it easily and automatically with Junit.
Though, this approach is very complicated and it is considered when the domain has a rich set of rules and behavior, while many applications have only CRUD capabilities.

Think of your application as existing only in Ram memory and strip out all the unnecessary code. The classes which remains are the core of the Domain Model. For instance if I had to manage the list of the users of a forum, I would write initially only a User class and a generic collection of User objects. User in my point of view is a POJO or POPO, which does not extend anything:
class User { ...

Insulating the database from view will hide this generic collection behind an abstraction, since it is not present at all in memory except for caching. Subsets of it can be reconstituted as needed. This abstraction is called a Repository or, at a low level, a DataMapper. Hibernate for Java and Doctrine 2 for php are example of a DataMapper pattern: they let you work on your objects and then take them and synchronize the database with the new data you have inserted: a change in a User's password, or new Users to be added. To polish the DataMapper Api, which is very generic, a UserRepository class can be created.
Even if you do not have a generic DataMapper, and work with mysql_query, PDO or JDBC queries, you can write a UserRepository which will act as the port for the database (or maybe a text file; since the repository decouples it, the storage mechanism can be anything from a memcache server to a group of monkeys which writes down on paper serializations of objects).
Depending on your architecture, your controllers or other domain classes will now have the UserRepository as a collaborator, and will talk to it and call its methods instead of accessing the database directly; this is a form of Dependency Injection. Obviously if there are other entities which are persisted, like Groups, Posts, etc. they should have their own repository class.

Of course the point of this discussion is how to test this code, since to write it we have to prepare a test before (it's called red green refactor and not code and then try to test). If we manage to write these tests first, the code will be automatically testable, and very decoupled and reusable as this is a characteristic of testable code.
What we need to write are not tests, but unit tests. If you want to check the entire path of data in the application, you can write integration tests which will exercise even the user interface, but Test-Driven Development prescribes to write unit tests: your test methods should have a dependency only on the system under test, and to the interface it uses.
Continuing our example, we need:

• unit tests for User, Group, Post classes (Domain Model entities);
• unit tests for controllers or other classes from the Domain Model which uses the adapters.
  
• unit tests for UserRepository and similar classes (adapters);
• unit tests for DataMapper or (infrastructure);
  

The point of unit testing is you can test singular components separately, and not the entire application as you would do with functional testing. Moreover, you must test each component separately or otherwise it would reveal too difficult to write classes to satisfy an integration test and the TDD benefits of adding a bit of design with every test method would be lost.
With this knowledge, we can say:

• unit tests for Domain Model entities must be written by the developer of this application. If the project is mostly a CRUD application and is data-intensive, these test cases will be very short.
  
• unit tests for controllers and everything that uses the adapters also must be written by the developer, mocking the adapters out. If you use the real adapters in testing, it will be integration testing and it will be heavy and brittle; you won't know if it's the controller or the adapter which does not work after a red test.
• unit tests for the adapters concrete classes: this is the only interaction with the database. Fortunately, we are unit testing for this classes so we can even use a new database for every test method as every class will present a few methods; compare this approach with testing every single feature of the application by using a real database.
• unit tests for infrastructure: these are included in the framework so we shouldn't worry.
  

Note that if you use a DataMapper which abstracts the particular database vendor as infrastructure, you can run the unit tests for the adapters by using a in-memory database like sqlite instead of the real database which will likely be very heavy to manage and recreate every time; this way, you de facto exclude database from your unit tests. Of course some integration tests will be necessary, but they are not part of TDD and of the design process.

I hope to have given you an idea of what unit tests means and how to deal with an application which uses a database for persistence, which is a very common scenario. Feel free to raise questions in the comments if something is not clear.

Do you want more? There's a book for .NET developers which explains DDD and database-independent testing.
The image at the top is a photograph of integrated circuits, which are real world components designed for testability and which are tested indipendently from the card where they will work.

Pagination is dead

Pagination is the feature for displaying a long list of entities in a web application: a division of them per page and a list of link to the various pages. Today there are better solutions to this classical problem, and some of them were always available even in the first days of the web.

The typical scenario solved by pagination is to allow the search of a particular entity from a list, by displaying it a chunk at the time. Particularly in web applications, where page size is limited by bandwidth, the maximum amount of items contained in a page is fixed in less than an hundred:

• programming languages query on Google
  
• motorcycle on Wikimedia Commons
• dance research on Youtube
  

The problem with pagination is how often do you look to page 2?
I google many times a day, so many that I now use the search bar of firefox instead of loading the homepage and entering the query in the input field. I usually found the first or the second result to be the most reliable resource for the query I entered since Google ranking is legendary: it's Google that decides how popular an article on this website will be and the only thing that competes in popularity with Google ranking is social network one.
Thus, I never went to the 2nd page of a Google search result. I bet you neither have done the same more than once or twice this month, and probably refining your search terms would have put the link you were looking for in the first position of the first page. Since the first link is almost always what you will be clicking some seconds later, Google main page even feature the I'm Feeling Lucky button which does this work for you.
In my opinion, Google pagination is rather useless.

In the early years of the web, pagination was the killer feature: LIMIT clauses for databases were everywhere and calculation of its argument were spread all over an application. This blog, hosted on the Blogger platform, also implements pagination: but do you prefer to scan my archives five posts at the time or to use the search box on the right?
Although all the content is available in a list of pages, a blog is not a book and it is not sequential: articles are often found by visiting a particular label or by a Google result. Honestly I sometimes look to the page 2 or 3 of a blog to form an idea on what content is posted there and decide whether to subscribe to the atom feed, but I think the author would rather have me look to a search on a tag, to a collection of popular posts or to its about page.

What about different kind of lists to paginate? Wikipedia lists are often very long: sometimes pagination is not adopted, like in the link, and the result is a unfocused and difficult to navigate page. But if you refuse to paginate there are other ways to manage this big pack of data:

• showing results on demand a la Dzone: thanks to ajax requests, when an user reaches the end of the list or is at the last items, another chunk is lazy loaded to fill the empty space between the list and the end of the page.
• better search system: as we have discussed earlier, Google does not need pagination since it is the best search system and you'll find your desired result in the first 10 links. Provide a mean to search a big list instead of spitting it all out, leaving the burden on the end-user.
  
• real time filtering: a dojo grid presents a pagination similar to the Dzone one, but different filters can be attached to modify the query. The result is similar to google suggestion while typing in the text field, as when you add characters to your search string the filtering is performed instantly.

These are only examples of what can be done if you force yourself to not paginate. The question which I repeat here is always the same:

Who will have the patience to look at page 2?

If you keep in mind this problem, finding another user interface to substitute pagination will be at the top of your todo list.

The scroll at the top of the page shows you a continuos source of pages that reminds of pagination sliders where you can go only to the next and previous page. It was very inefficient, but ancient monks did not have Google search capabilities...

10 orm patterns: components of a object-relational mapper

An Orm is a complex and generic tool which stores objects such as Entities and Value Objects (Customer, Groups, Money classes) in a relational database like MySQL or Sql Server, using metadata provided along with the Domain Model. The internals of an Orm usually follow some useful patterns that a developer should know to understand what is going on under the hood and here's a list of the most famous ones.

• Table Data Gateway: an object that represent a table of the database, on a one-to-one basis. It is usually built as a generic class which can be subclasses or instantiated for any physical table.
  
• Active Record: the most common approach, transforms a row of a table in an object. It strictly couples the object structures to the database tables by making the domain object subclassing an abstract implementation.
• Data Mapper: a real Orm is an instance of a Data Mapper, a tool that stores objects which are ignorant of the environment where they will be kept, thus decoupling them from persistence concerns.
  
• Unit Of Work: maintains a list of dirty objects and writes out the changeset. The purpose of this object is to keep track modified data on the entity object that it knows and its flushing capabilitities substitute the save() method of the Active Record implementations. It is a more resilient and sophisticated pattern than Active Record since it strives for persistence ignorance.
• Repository: the persistent-ignorant equivalent of the Table Data Gateway. While a single repository implementation is aware of the database backend, a generic interface is placed between service classes and object retrieval mechanisms to aid decoupling.
  
• Identity Map: a map of objects organized by class and primary key. It is a first level cache that contains every initialized object, so it can be used to prepare a changeset of database queries on flushing.
  
• Lazy Loading: substituting a domain object or collection with a subclass that loads data on the fly when methods are requested, before forwarding the original call.
• Query Object: a class that represents a query for retrieving objects, encapsulating Sql or other high level languages.
  
• Criteria Object: a class that represents a set of criteria for selection of objects of a particular model.
  
• Single Table Inheritance / Class Table Inheritance / Concrete Table Inheritance: patterns implemented to represent class inheritance in relational tables. Favoring composition over inheritance is a must because neither of these patterns does a perfect job in persisting data efficiently and in a clean way.

I hope you'll check out more information on the patterns you are using, maybe without even know that they exist. Feel free to add other useful patterns and approaches to the list in the comments.