A year of mob programming, part 5: methodology

I wouldn't call mob programming a methodology, but rather a practice that can be adopted. Like methodologies, however, it only surfaces problems and questions faster rather than providing solutions.

For example, a team might have a low bus factor due to the specialisms of its members, and only one person might be used to edit code in a certain language or application. Other team members might have gaps, such as not being set up with the tools and credentials to work on infrastructure and operations; or lacking the knowledge to perform screen reader testing for accessibility.

Besides exposing bus factors and gaps, mob programming brings design conflicts and opinions into the open. It takes little effort to turn the other way when looking at a pull request, or to switch to busy work or generic "refuctoring" when alone; disagreements and questions about value can often prop up under your nose inside a mob session.

However, that doesn't mean that the practice automatically gives the team the psychological safety to talk about those issues out loud, and to resolve them. That's like asking an hammer to walk to the shop and buy the right nails.

At the last part of this post, I thought I'd compare my mob programming experience with some of the theory; the theory being famous Agile software development methodologies and their values or principles.

The serene predictability of waterfall software development

Within Extreme Programming values:

• Communication inside a mob is very high: synchronous conversations supported by a visual medium such as code or a digital whiteboard.
• Simplicity is fostered by people in a diverse team reminding others that You Aren't Gonna Need It, but also needing to understand what's going on at all times.
• Feedback from the rest of the team, or even from a customer proxy, is immediate.
• Courage is needed to be a driver in an unfamiliar setting, or even a navigator asking the mob for help on a task we are uncomfortable with.
• Respect is necessary to work face to face for prolonged periods of time.
  

Within Lean values, mob programming:

• eliminates waste such as partially done work: there should ideally be no pull request open or branch, as code is produced and integrated in the mob continuously;
• eliminates waste such as outdated features or gold plating: when you include a product owner in the mob, it's easy to steer and change the scope upon feedback rather than plowing on with the original idea;
• eliminates waste such as handoffs: there is no packaging up of code for review, or of pixel-perfect Photoshop design to front end developers, or front end requests to back ends to be implemented, and so on.
• reduces task switching: since everyone that is needed for progress is present, you often don't need to switch task because of waiting on someone, but can just carry on.
• amplifies learning: ideas are tried out in code from the get go, and most of the team can learn from the results including the product owner.
• optimizes the whole: most of all, we are not optimizing how much we can use everyone's time, but how fast we can flow a single piece from idea to customer value.

Image by Yann.

A year of mob programming, part 4: the remoteness of it

I don't have a frame of reference for mob programming in a co-located environment, besides conference (or team exercise) workshops that didn't involve production code.

On a video call, the driver is still always explicitly defined as the person sharing their screen and tools. This is similar to the person physically having the keyboard when you are all sitting around a table.

The navigator can either be more implicit, move around or be strictly defined by a process. If an implicit navigator doesn't emerge for the current commit, it should be considered normal to quickly nominate one as needed. Over time, we normalize the phrase "can you navigate [that change]?" in response to a proposal.

Due to latency, in a video call environment there is a sort of Lorentz time where you may hear overlapping voices when someone else doesn't, and vice versa. It all depends on how much latency each video link is experiencing, and the adjustments that software makes can be jarring, such as video freezes or audio being delivered 10-20 seconds later.

"Sorry, two people talking over each other" "Really??"

I'm sure though everyone has already experienced people talking over each other in a physical room: it isn't exclusively a remote working problem and relies on team interactions more than technology.

We also want to communicate and especially give feedback through voice, as people's faces are not always visible to a driver looking at code; nodding doesn't necessarily help.

Perhaps a bigger problem is the fact that video calls don't allow side conversations to happen, as there is a single audio channel that our ears cannot separate into directions. This is likely to impact a massively parallel EventStorming around a whiteboard, less so an ensemble of 3-4 people trying to get to a consensus on their next move. The side conversations however include those famous water cooler talks, bumping into people in the kitchen or while on a break.

Your theory of mind might have more trouble cutting through the limited signal, and understand in what mental state other people in the team are; and whether today's problem is more due to Internet connection trouble, or a real conflict between team members. The daily retrospective helps to let people put tools down, and reflect on the day; possibly proposing experiments for the next session. 

Looking at the world around me, I'm quite sure I would be as depressed by remote work as many, if it consisted of me being alone for most of the day. So due to mob programming I have a different perspective on how incredibly engaging working from home can be.

Stay tuned for the next (and last) part of this post, Methodology.

Image by Duesentrieb.

A year of mob programming, part 3: a laboratory for team dynamics

Our implementation of mob programming consists in a permanent Google Meet videocall, with camera on, in which we rotate a developer sharing a screen containing IDE and browser. So there are two unusual practices to get used to: being in a group all the time, and being on a camera all the time.

I was originally surprised on how both practices went from exhausting to being a normal work day. This is my experience, and we have to consider neurodiversity and personal preferences in the team on having a long screen time and continuous social interaction. It's almost obvious to say, but sometimes people hate inconclusive meetings in which they have no say; not being together with other human beings.

Don't underestimate your capacity to adapt, but recognize that your energy is not going to always be the same; whether because you got a cold, or something is going on at home, or some task is particularly draining. 

The Flemish inscription at bottom reads: because feedback is perfidious, I will go code in my cave

Gold cards and spike branches give people the ability to work on their own when they request to. Mechanical work such as upgrade of dependencies or investigating logs can benefit from focused time from one person. If you were in an office, you could go to an isolated room; working remotely, it's even easier as it just consists in leaving the video call and coming back refreshed later.

The opposite can also happen, when someone like me can manifest Fear Of Missing Out and the mob making lots of decisions in their absence. As the team gets to a norming phase, we should expect fewer surprises for members coming back to the mob after some time off.

Being together with the rest of the team can make your motivation higher due to the group support in getting started or unstuck from a particular problem; but of course the whole group can struggle on occasion.

Every time there is a new composition, though, there are changes in how the mob is working and what it pays more attention to. And from the point of view of a growing team leader, being embedded in a mob environment means being positively inundated with information. The challenge is to make sense of what we see to understand where team members are struggling and what are they finding most effective.

"I wish I could see players only in 121 meetings" -- No football coach ever

This requires a lot of (cognitive) bandwidth, and I usually can't at the same time focus on technical architecture and on social roles. But maybe this is just an argument to work in a team where the different hats can be rotated; as much as there is always someone writing code, there can always be someone checking if we have started to talk over each other, or we are falling into a rabbit hole of an hour without committing.

Stay tuned for the next part of this post, The remoteness of it.

Images by Michael Kranewitter and in the public domain.

A year of mob programming, part 2: Collective Code Ownership

With respect to a team assigning tasks to developers by their function (frontend, backend, infrastructure, and so on), mob programming fosters Collective Code Ownership.

This is more in the sense that everyone should be able to contribute through the mob in any area, rather than everyone being able to prioritize on their own what needs to be improved. The pact is that our time belongs to the team, and the team decides what is important.

A result of this is an healthy prevention of knowledge silos, where you can exercise your own creativity just because no one else knows how to work in that area anymore.

Code review happens continuously in the mob on small changes, and there are no huge feature branches to go through. In large changes such as those, it's too late for review to suggest a valuable but completely different approach; there is too much investment of time and energy into code. If review dares to suggest groundbreaking changes, it causes extensive rework instead.

There is a larger picture on ownership: team members tend to keep track of what they care about, whether that is consistency in architecture, front end approaches to styling, performance issues, and so on.

Sometimes I found myself being talked out of a design I had in mind, counteracting bias I could have for the first or most familiar solution. I started working on this team with an object-oriented mindset, but we have transitioned to functional programming in TypeScript.

Sometimes, there are hills to die on: hard to reverse decisions on architecture, or programming language choices. It's a job of a psychologically safe team to discuss these choices collectively without drama or oversimplification like JavaScript being the the death of computer science. 

"I told you we should have written the code in English, not Latin!"

It tooks three different iterations to get a repeatable pattern for accepting Commands (as in CQRS). But it was more fruitful to focus on the Domain Events design, as a difficult choice to revisit, than on the shape of the code itself which can be refined at any time.

Working outside of the mob

The counterpart to the XP mantra of writing all production code in pairs (or a mob) is to allow team members to contribute when they are working alone due to other necessities, such as an unstable Internet connection or a flexible day where they can't align completely with the core hours of the team.

I have learned to ask the team to be commissioned something to do, or at least to give them a choice. This keeps the prioritization in the hands of the group, again reducing bias.

It's helpful to assign problems that have only a constrained possible solution such as renaming, or propagating a refactoring through the codebase for consistency's sake.

It also helps to report back what you have learned during a solo coding exercise when rejoining the mob; unexpected issues or decisions you had to take on the spot and you feel unsure about.

You're allowed to stop at a certain point, as making progress alone at all costs is valued less than maintaining collective code ownership and a shared understanding of how our application works.

Technical spikes can also be chosen by a single person to work on, either because they want to individually prioritize them to demonstrate an idea; or because the amount of investigation required makes it difficult or frustrating to collaborate.
Spikes can be built on throwaway branches, being optimized for learning rather than for delivering production quality code. Once an idea has been demonstrated and approved, the mob can implement it pretty quickly on the trunk, and if we believe in the practice, with an higher level of quality. I constantly find feature branches worked by a single person to be a dead end (especially if they are my own).

In the end, people are frequently in meetings, researching or even just on holiday for a given day. Hence there's always someone missing that will catch up on the progress when they come back into the mob. But the group itself never stops even as the components change, so some of that progress will happen every day, other things being equal.

Stay tuned for the next part of this post, A laboratory for team dynamics.

Image by Johann Jaritz.

A year of mob programming, part 1: metaphors

I've been practicing remote mob programming in my team for more than a year, writing more than 90% of production code inside a video call with multiple developers looking at the same screen.

A bit of context

I work in the scientific publishing domain, on web applications oriented to help scientists throughout their day.

My current team has been formed from the start as a remote team, and has been working with mob programming for more than a year. It is a cross-functional team that is including a product manager and often a designer into the (virtual room) for co-creation.

Our set of technologies includes TypeScript, lots of semantic HTML, and minimal client-side JavaScript. Container-based infrastructure and databases are pulled in as required by the evolution of the product.

This is an experience report, not advice that can be applied blindly to your team or organization.

Metaphorical definitions

All the brilliant people working on the same thing, at the same time, in the same space, and on the same computer -- Woody Zuill

Like pair programming, mob programming can be simplistically explained through the metaphor of driving a car in a rally. Or at least, that's how I always understood the driver and navigator pair.

• A driver shares their screen and has sole access to the keyboard. They are an intelligent IDE capable of executing mechanical refactoring and fixes, but not of deciding a design direction for the code.
• A navigator verbalizes what the driver should be doing, making decisions that reach down to what test or line of code to write.
• The rest of the group is on the back seat of the car, and can speak at will, volunteering information to the navigator or answering questions. Are we there yet?
  

The roles of driver and navigator are rotated often, in fixed time or scope increments such as on every commit. The navigator can implicitly move continuously from one person to another, as long as there isn't more than one navigator at once.

Mob as a name

Speaking Italian as a first language, the term mob doesn't necessarily evoke an emotional reaction from me. But I can recognize a term that can be used to describe organized crime might not allow team members to feel comfortable. My understanding is that mob has to be intended as crowd rather than mafia.

"This is so wasteful. Why aren't they all in separate rooms playing 4 different songs?"

Nevertheless, we looked for more metaphors and terms that can be used to talk about the group of people that work together:

• orchestra emphasizes the coordination required by a group of people, and their simultaneous presence. I'm not sure they need a conductor though.
• ensemble is a similar musical metaphor, that focuses on a small group of complementary roles closely working together.
• swarm refers to a group of bees all converging towards a single task to get it done.
• the team itself ideally coincides with the mob, though there can be non-technical roles that find difficult to contribute all the time; the mob can also be temporarily reduced from the full set of developers in the team, or multiple mobs can appear in a relatively large team.
• a womble is a fictional furry creature from children's books that helps the environment by cleaning up rubbish. It has no relation to groups whatsoever and is an unrelated word, with no unfortunate or fortunate connotations attached. We ended up using this word for a while for disambiguation.

These are all the metaphors that I've seen used to describe mobbing. Stay tuned for the next part of this post, Collective Code Ownership.

Image by Princess Ruto.

Book review: The 5 Dysfunctions of a team

This is a spur-of-the-moment review of The 5 Dysfunctions of a Team, a business novel on team health that I've read today as part of the quarterly Professional Development Days I take as part of working at eLife.

As a follow-up to my role evolution into Software Engineer in Tools and Infrastructure, I am looking again more into the people skills side of my job (as opposed to purely technical skills). I have done this cyclically during my career, as the coder hat becomes too restrictive and you have to pick up other tools to achieve improvement. In particular, I am working on eLife's Continuous Delivery platform and it is crucial to work with multiple product-oriented teams to have them adopt your latest Jenkins pipelines and Github reports.

Dysfunctions

Patrick Lencioni's model of team dysfunctions (or of blessed behaviors if you flip all definitions) is a pyramid where each dysfunction prevents the next level from being reached. It would be a disservice to how well and quickly this is got across in the novel to just try to list them here, but if I had to summarize this in a long paragraph it would look like:

Building trust between team members allows constructive conflicts, which enable people to commit to action and hold each other accountable for what has been decided; all in the service of results. -- not really a quote

The dysfunctions are the flip side of these positive behaviors, for example lack of trust or fear of conflict. The definitions of some of these terms are more precise than what you find in many Agile and business coaching books; so don't dismiss trust as just a buzzword, for example.

Some context

The case being treated in the book, which comes from the author's management consulting firm, is that of a CEO turning around a team of executives. This makes for a somewhat more fascinating view of results as it's talking about a (fictional) company's IPO or eventual bankruptcy. Despite not being a clear parallel to a software development team, I do think this is applicable in every situation where professionals are paid to work daily together, with some caveats.
In fact, I suspect the level of commitment to the job that you see in the book would be typical of either high stakes roles (executives) or a generally healthy organization that has already removed common dysfunctions at the individual level. If in your organization:

• people are primarily motivated by money
• they look forward to 5 PM
• they browse Facebook and Twitter for hours each day

then there are personal motives that have to be addressed before teams can start thinking about collective health.

Yes but, what can I do in practice?

After the narrative part, an addendum to the book contains a self-administered test to zoom in on which possible dysfunctions your team may exhibit at the moment. It continues with a series of exercises and practices that address these topics, with an estimation of their time commitments or how difficult they would be to run. I definitely look forward to anonymously try out the test with my technical team, out of curiosity for other views.

My conclusions

I think most of the dysfunctions are real patterns, that can only be exacerbated by the currently distorted market for software developers and CV-driven development. The last dysfunction, Inattention to results, is worth many books on its own on how to define those results as employees at all levels are known for optimizing around measurable goals to the detriment of, for example, long-term maintenance and quality.
So don't start a crusade armed by this little book, but definitely keep this model in your toolbox and share it with your team to see if you can all identify areas for collective improvement; it is painfully obvious to say you can't work on this alone!
The author is certainly right when writing that groups of people truly working together can accomplish what any assembly of single individuals could never dream of doing.

Pipeline Conf 2017

Not liquorice

Post originally shared on eLife's internal blog, but in the spirit of (green) open access here it is.

Last month I have attended the PIPELINE conference in London, 2017 edition. This event is a not-for-profit day dedicated to Continuous Delivery, the ability to get software changes into the hands of users, and to do so safely, quickly, and in a sustainable way. It is run by practitioners for practitioners, everyone on different sides of the spectrum like development, operations, testing, project management, or coaching.

The day is run with parallel tracks, divided into time slots of 40-minute talks and breaks for discussions and, of course, some sponsor pitches. I have been picking talks from the various tracks depending on their utility to eLife's testing and deployment platform, since our tech team has been developing every new project with this approach for a good part of 2016.

The conceptual model of Continuous Delivery and of eLife's implementation of it is not dissimilar to the scientific publishing process:

• there is some work performed into an isolated environment, such as a laboratory, but also someone's laptop;
• which leads to a transferable piece of knowledge, such as a manuscript, but also a series of commits, roughly speaking some lines of code;
• which is then submitted and peer reviewed. We do so through pull requests, which perform a series of automated tests to aid human reviewers inside the team; part of the review is also running the code to reproduce the same results on a machine which is not that original laptop.
• after zero or more rounds of revisions, this work gets accepted and published...
• which means integrating it with the rest of human knowledge, typesetting it, organizing citations and lots of metadata about the newly published paper. In software, the code has to be transformed into an efficient representation, or virtual machines have to be configured to work with it.
• until, finally, this new knowledge (or feature) is in the hands of a real person, who can read a paper or enjoy the new search functionalities

Forgive me for the raw description of scientific work.

In software, Continuous Delivery tries to automate and simplify this process to be able to perform it on microchanges multiple times per day. It aims for speed to be able to bring a new feature live in tens of minutes; it aims for safety to avoid breaking the users work on new changes; and to do all of this in a sustainable way, not to sacrifice tomorrow's ability to evolve for a quick gain today.

Even without the last mile of real user traffic, the 2.0 software services have been running in production or production-like servers from the first weeks of their development. A common anti-pattern in software development is to say "It works on my machine" (imagine some saying "It reproduces the results, but only with my microscope"); what we strive for is "It works on multiple machines, that can be reliably created; if we break a feature we know within minutes and can go back the latest version known to work."

Dan North: opening keynote

Dan North started to experiment with Continuous Delivery in 2004, at a time when builds were taking 2 days and a half to run in a testing environment contended by multiple teams. He spoke about several concepts underpinning Continuous Delivery:

• conceptual consistency: the ability of different people to make similar decisions without coordination. It's an holy grail for scaling the efforts of an organization to more and more members and teams.
• supportability: championing Mean Time To Repair over Mean Time Between Failures. The three important questions for facing a problem as what happened? Who is impacted? How do we fix it?
• operability: what does it feel like to build your software? To deploy it? Test it? Releasing it? Monitor it? Support it? Essentially, developer experience in additio to user experience.

Operability is a challenge we have to face ourselves more and more as we move from running our own platform to provide open source software for other people to use. Not only reading an article has to be a beautiful experience, but publishing one should be.

John Clapham: team design for Continuous Delivery

This talk was more people-oriented, I agree with the speaker that engagement of workers is what really drive profits (or value in case of non-profits).
Practically speaking:

• reward the right behaviors to promote the process you want;
• ignore your job title as everyone's job is to deliver value together;
• think small: it's easier to do 100 things 1% better than to do 1 thing 100% better (aka aggregation of marginal gains)

Abraham Marin: architectural patterns for a more efficient pipeline

The target for a build is for it to take less than 10 minutes. The speaker promotes the fastest builds as the one you don't have to run, introducing a series of patterns (and the related architectural refactorings) to be executed, safely, to simplify your software components:

• decoupling an api from implementation: extracting an interface package to reduce the dependencies to a component to a dependency to an interface;
• dividing responsibiliteis vertically or horizontally trying to isolate the most frequent changes and minimizing cross-cutting requirements;
• transform a library into a service;
• transform configuration into a service.

Some of these lessons are somewhat oriented to compiled languages, but not limited to them. My feeling is that even if you reduce compile times, you still have to test some components in integration, which is a large source of delay.

Steve Smith: measuring Continuous Delivery

How do you know whether a Continuous Delivery effort is going well? Or more pragmatically,  which of your projects is in trouble?
The abstract parameters to measure in pipelines are speed (throughput, cycle time) and stability. Each declines differently depending on the context.
In deployment pipelines that go from a commit to a new version release in production, lead time and the interval of new deployments can be measured. But also failure rate (how many runs fail) and failure recovery time are interesting. In more general builds or test suites, execution time is a key parameter but a more holistic view includes interval (how frequent are builds executed).
I liked some of these metrics so much that they are now in my OKRs for the new quarter. Simplistic quote: you can't manage what you can't measure.

Alastair Smith: Test-driving your database

To continuously deploy new software versions, you need an iterative approach to evolve your database and the data within it. When you evolve, you also have to test every new schema change. Even in the context of stored procedures for maximum efficiency (and lock-in), Alastair showed how to write tests that can reliably run on multiple environments.

Rachel Laycock: closing keynote, Continuous Delivery at Scale

Rachel Laycock is the Head of technology for North America at Thoughtworks, the main sponsor of the conference. The keynote however had nothing to do with sales pitches. Here are some anti-patterns:

• "We have a DevOps team" is an oxymoron, as that kind of team doesn't exist; what often happens is that the Ops team gets renamed."
• Do we chose Kubernetes or Mesos?" as in getting excited about the technology before you understand the problem to solve.

The "at scale" in the title pushes for seeing automation as a way to build a self-serving platform, where infrastructure people are not bottlenecks but enablers for the developers to build their own services.
The best quote however really was "yesterday's best practice becomes tomorrow's anti-pattern". What we look for is not to be the first to market but to have an adaptable advantage, a product that can evolve to meet new demands rather than being a dead end.

The Shower Methodology and Kanban

I'm still reading (really slowly, in order to absorb informations better) Test-Driven Development by example by Kent Beck.The author cites Paul Ungar's Shower Methodology for writing code:

• If you know what to, type it.
• If you don't know what to type, go take a shower.

 Many teams would be happier, more productive, and smell a whole lot better if they took his advice. -- TDD by Example

The shower is indeed the place where many difficult problems are solved. When you have a difficult problem, and it becomes and stay your top priority, your brain daemons (background processes) will work on it for you, and present a fresh solution in an unexpected moment, when you stumble on the problem in an unfamiliar environment such as the shower. When you are parking your car, or walking, or showering, these processes will leverage your free CPU.
Sometimes raising money becomes your shower top priority:

I realized recently that what one thinks about in the shower in the morning is more important than I'd thought. I knew it was a good time to have ideas. Now I'd go further: now I'd say it's hard to do a really good job on anything you don't think about in the shower. [...] Money matters are particularly likely to become the top idea in your mind. The reason is that they have to be. It's hard to get money. It's not the sort of thing that happens by default. It's not going to happen unless you let it become the thing you think about in the shower. And then you'll make little progress on anything else you'd rather be working on. -- The Top Idea in Your Mind

Test-Driven Development is an extension of Ungar's Shower Methodology. Given a red (failing) test, you'll have to make it pass:

• If you know what to type, type it: always start from the Obvious Implementation if you know what it is.
• If you don't know how to achieve the result, Fake It: return 0, return 4 and other hard-coded test-specific production code are ubiquitous as TDD starting points.
• If you still can't generalize after faking the response, try Triangulation: write another test which feeds the system other values and expects a different response. This test, when passing, will ensure there is no hard-coding anymore.
• If with triangulation you still don't know how to go on, take a Shower.

For example, given the test for extracting the square root of 2:

$this->assertEquals(1.4142..., sqrt(2));

I would code an Obvious Implementation for sqrt(), if I already know the algorithm.
If I'm unsure, I would insert return 1.4142 to get a green bar initially, and then go on writing the algorithm.
If I do not know the algorithm at all, I will insert another test for sqrt(3) to triangulate it (even more than one test if it's the case).
If now I wonder what the algorithm is and I have no idea on how to go on, I will take a shower and think about it.

Disclaimers: TDD in this version is a methodology for implementing a single feature (it doesn't cover prioritization, or estimation, or everything else). It acts as a low level of abstraction, which can become higher if you can write end-to-end and functional tests.
Note also that taking a shower may be metaphorical: you can go work on another project, or another component, or to a meeting, or google for some resource on algorithms and data structures.

Our Kanban board

After all this talking about breaks, the comparison that does not always fit is with Kanban and Work-In-Progress limits. If you have to build a car, indeed you will achieve a very short lead time if you limit the buckets of semi-finished parts: there will be less inventory to transport and store, they will be exchanged more frequently and so on...
If you have to develop software, well, it depends: if you always perfectly know what to type, you can place a WIP limit of one or two user stories and work only on one feature at the time. Scrum and every Agile methodology does exactly this with iterations; Kanban does it at a finer level by limiting the tasks that can be moved forward in each phase (analysis/development/testing/deployment/...).
If I am on a PHP web application, Usually I perfectly know what to type: no problem with WIP limits.
Yet if learning is involved, you may juggle different stories and tasks in the same time period, working for example for one day on each, rotating them. I am tackling a difficult computer vision project where I have to solve problems that range from tracking vehicles in a video to estimate their positions and understand if they will collide under certain conditions. I'm putting in it 4 hours a week: there's no way I will be able to do more due to the research and experiments involved (half of the time I do not know how to go on; and it's normal.) Multitasking is the only way to go if I do not want to lose the other 36 work week hours.
So if I had an hard WIP limit on the projects I can tackle, I would be screwed. You know what works with very large WIP limits? Universities. Normally in Italy you can have 6 courses at the same time. If you try to do one course at the time (20 hours a week for three weeks), you will shoot yourself. Formally, there are almost no limits in the engineering faculties: you can complete your studies following the order you prefer.

Thus I think WIP limits are important as they avoid attacking too many things at the same time, and never complete any. Yet if creativity is involved, shuffling projects and go taking a shower when you need it will help you more.

I went for a shower after writing a draft of this article, and I thought also about the analogy of reading books, which is an example of where Kanban is really useful (summing it up: don't buy books if you do not already read all the ones from the last order.) In the cases of user stories, car parts or books we have a set of items which is useless until it goes through all the stages: a user story is deployed, a car part is on the road, a book is ordered, unpackaged and read. You don't get value from a still wrapped book sitting on your shelf, a story you can't deploy yet or an engine not mounted on a car.
With university, it's beneficial to have related courses taken at the same time instead of serially: you can often apply in practice a concept coming from another course, or compare different approaches (mathematical one vs. physics one vs. the engineer's attitude to linearize and model everything as a Gaussian; or the operations researcher attitude towards a problem vs. the computer scientist goal to find the best data structure to solve it).
With research projects like the computer vision one, the end result is unclear and there are no subparts that can be defined earlier. That's how it works in scientific research. I often don't have user story to transport from one side to the other of the board: I have challenges to overcome with creative thinking, and creativity is not a very repeatable, optimizable process.

Note that there is always a bit of creativity in every software project: that's what makes it fun.

Italian Agile Day 2010

Tomorrow I'm going to Italian Agile Day in Genoa. It is not a long trip, but I will wake up at 5 am.
If you want to chat, code or insult buzzword-obsessed professors together, look for me there.
Twitter stream: #iad2010

Learning the Pomodoro technique

During a workshop of Francesco Trucchia I've been introduced to the Pomodoro technique for time management. I've known the existence of this methodology for years, but I never gave it a shot, being unsure if it was adequate for programming tasks.
It seems however that in the Agile world the Pomodoro technique has a good following. I won't go into the details of explaining this technique as the official site can do this much better than me (I started timeboxing with pomodoros on Tuesday morning.)
Basically, the atomic unit of time is an uninterruptible pomodoro, which lasts 25 minutes and which comprehends 5 minute pause at the end of period of work. Pomodoros can't be sliced up nor you can do other tasks different from the planned ones during a pomodoro, and the pause of 5 minutes is mandatory. During the pause, you can't continue thinking about the problem you were tackling (this has been difficult to grasp, but if you clear your mind during a pause 5 minutes have a value of much more).

Timeboxing has been very useful for writing articles, which is a task that commonly can suck up hours. I got a feedback on how much I was completing, and been forced to improve my process and avoid distractions (like looking for a word on Wiktionary or insisting on a particular construction).

For programming, which is the critical task, I am still experimenting after three days. With fast computers or platforms, this technique works very well. But sometimes I work with Java and I can get a cup of coffee before a test finishes running. I can't help multitasking with two machines to get something done in this environment.
The most difficult part for now has been estimating, since I have no experience with these short timeboxed. However, these estimations may become more accurate than
The rapid feedback of a pomodoro finishing is definitely a psychological pro, since it gives me back confidence on the task I complete and notice me if something is scattering into the next timebox (and so may be re-estimated, or maybe I should take a wider look at documentation before jumping into coding).

The forced interruption, in fact, has proven very useful, despite the worries of interrupting the flow.
First, Test-Driven Development helps me keeping my flow when I resume my work. I take a look at the last unit or acceptance test that does not pass and I know immediately where I was working on.
Second, it is often the case that I'm taking too much time on a single task: for example I am perform fine refactoring on code that I may end up throwing away, or I am insisting on integrating an external library now and fighting with runtime errors when I should switch to a different task with higher ROI.
Thus, the basic 25-5 minutes cycle is not a danger for programmers. Enter often in a state where we lose track of time it's not something we should be proud of...

Small steps

Today I was working with a Java framework which runs on top of OSGi (another framework, and yes, Java is verbose even from the platform stack point of view.)
After having prepared a (hopefully) failing acceptance test which used httpunit to extract the content of a dynamic page and verify that the world "Hello" is contained in it, I was ready to run the test, see the assertion failing and start writing our beloved production code.
Unfortunately, the test failed not because of the assertion, but because the JUnit Plugin runner, which takes a JUnit test and run it over a configured OSGi framework, was saying "No tests found." How that was possible?

When tests that should drive the development leave you without a clue on what is happening, chances are that you are taking big steps when you should walk carefully. I was taking big steps because it takes two minutes to run such an acceptance test due to all the infrastructure noise, but I'm not an expert in OSGi applications and I should have taken smaller steps towards my goal of a failing test. The same is true for production code: if I can't get a green bar maybe my failing tests are not fine-grained enough.

Thus I run a git reset and perform the following baby steps to build confidence:

• create an empty bundle (jargon for package), and write an empty test in it. Run and see Warning: no assertions, but nothing explodes.
• Add a test method with a assertTrue(true) call. Run, and nothing explodes still.
• Add httpunit in lib/ and set it up with the Bundle-ClassPath directive, which basically says where the runtime can find the library JARs in the bundle. 

Ok, it was that. After a quick search on Google, I found out that I was specifying the classpath as I've always done via command line or ant scripts:
Bundle-ClassPath: .:lib/httpunit.jar
while OSGi has a different opinion of classpaths:
Bundle-ClassPath: ., lib/httpunit.jar
So I lost even the current directory (.) from the classpath when introducing the first, malformed directive.

After having located the problem with a test-first approach, I was able to (slowly) run my red test and return writing production code.
Automated testing and incremental development save the day again - imagine what would have happened if I had to debug this mess manually, with a framework that takes some minutes to start, and no idea of where the error was...

Velocity

There's an old joke that goes like this:

One day a pilgrim was walking towards Santiago and, during the way, he encountered a philosopher. He asked the supposedly wise man:
"How much does it take to arrive to Santiago?"
But the philosopher did not answer.
Then the pilgrim began to walk again, and only when he was fifty meters away, the philosopher shouted:
"Half a day!"
The pilgrim looked back and said "But... Couldn't you tell me while I was there with you?"
And the philosopher: "I did not know what was your velocity."

One of the points of Agile methodologies is in approaching estimation without random guesses. We are often asked to prepare a schedule and a deadline without knowing enough about a software problem.
Agile prescribes to estimate relative size of user stories (which can be think of features with an acceptance test) in points instead of estimating time. The time spent to complete the user stories varies, because it is dependent on the team or programmer's velocity.
Velocity in physical definitions is a different quantity than speed. Velocity is defined as the derivative of displacement in respect to time, while speed is the derivative of path length in respect to time. While the shortest path between two points in space is a straight line, following a non rectilinear path results in an average velocity lower than the equivalent average speed, which could thus be a biased progress metric. If I go to Tokyo and return home in an hour, my speed would have been amazing while my velocity is exactly zero.

Velocity focuses on the effective work which has been completed, not on the effort nor on the hours spent coding. Only the finished stories are taken into consideration to calculate velocity.
Velocity is best estimated by gathering data from some short iterations (some weeks long). After the first iteration, which accomplishes a certain number of points from several completed stories, we are able to extrapolate a range for the remaining number of iterations required.

Consider this example: during the first two-week iteration of a project, a solo developer finished two stories, which are worth 5 and 2 points respectively. The velocity of this single-person team is 7 points per iteration. It does not matter how much hour he spent on this project as long as he can commit to the same effort in the subsequent iterations.
If the remaining stories' points amount to 40, we know that, on average, the number of remaining iteration will be 40 / 7 ~ 6. We may give a range of 5 to 7 or 4 to 8 iterations as the time estimate.
After more iterations, the velocity estimate is adjusted as the sample which we measure it on increases in size. The average velocity can be used to re-estimate the number of remaining iterations, providing a stricter range and updating the project schedule with newer informations. The team and the iteration length should be consistent during the project to take advantage of these metrics.

If you want to deepen your knowledge of this subject, I reviewed a book which may be propedeutic to this goal: Agile estimating and planning.

Agile estimating and planning review

Agile Estimating and Planning by Mike Cohn is a masterpiece on Agile management techniques, especially in dealing with schedules and application features. I just finished reading it and it gave me a very positive perspective on classical development conundrums like schedule and scope.
Agile does not solve problems for us, nor promises to eliminate every issue. The 300+ pages cover the majority of the topics in the, like the title says, estimation and planning field for an Agile team. The author writes in an honest style and anticipate reader's questions and objections.
There are many concepts scattered trough the book:

• The Agile planning approach: we don't know much at the start of a new project, but after every iteration we get to know more about the domain and the application. Thus, we can improve our estimation of remaining work, while changing scope and release dates to deliver the maximum value. So we should keep planning, but be ready to throw away the plans.
• Estimating size and estimating time are two different processes: velocity is the parameter that links them. Size is described by different, relative variables than actual time needed (like story points or ideal days).
  
• What's a story point? And a release burndown chart? We often use Agile terms without referencing read formal definitions and they can seem mumbo jumbo to the uninitiated developer. Actually, Agile is not complicated if you take a bit of time to learn; you probably already know nearly all of the math involved in this book, but a glance at probability theory could help.
• Tools like questionnaires and charts for tracking progress explained from the ground up. Back at the first chapter, I had not an Agile theorical foundation, but I still found the book exciting to read and very accurate.
• Common practices for stories management can help you to mix up, split and join user stories. Estimation can be a difficult process but in this context it is not a random guess.
  
• Prioritization of stories, along with iteration and release planning: the 1,000 and 10,000 feet views on your project life and scope.
• Plenty of practical examples are spreaded throughout the chapters, and the author reports how to implement the techniques described in a real project, by consistently taking a swimmer statistics management application as the main example. This consistency helped me to get the overall picture.
• Finally, a fictional case study is presented at the end of the book, to pull all things together and see an Agile project worked out from the initial requirements gathering to the deliver date.

After having read this book, every project now seems a big opportunity to apply an Agile approach. I strongly recommend it if you want to get started with story points, iteration and other great Agile concepts.

Mistakes of a freelancer

I have been inspired by a post by Soon Hui to write about my mistakes. I have evolved much on the programming side during these years, but my biggest mistakes have been social and economic: dealing with other people. Thus, I have collect a list of my errors committed as a php freelance developer.

• Giving out your personal phone number: no matter what, use separate phone numbers for clients and friends. People have the tendency to call in awkward hours, and having a single number you can shut off after the workday has finished helps your work-life balance .
  
• Lack of tests: every application you write will be maintained in the future, often by you. Even small tweaks (that you can't honestly charge for) can break an application and the safety net of a test suite will free you from the burden of manual testing.
• Providing fixed estimates: estimates should be given in the form of a range, and the whole process of estimation and planning in software development is more complex than the average person thinks. Counting billable hours just does not work and an application's size should be assessed during the requirements gathering.
  
• Tasks instead of features: one pillar of Agile processes is that features are the metric of success and accomplishment, and not tasks. Even if you're working on fixed-price waterfall projects, focus on giving out the features requested because no customer has the time to comprehend technical and infrastructure tasks such as "database modelling".
• Thinking that a client knows what he needs: even in porting legacy applications, no customer really understands the kind of software he wants. It is our job to interact with him to distinguish between mandatory and exciting features and providing the highest value in an application, since we have great  programming skills but little domain knowledge. Often emphasis is put on gold plated features which are really not worth their cost and can cause disasters in the long run (and maybe you are even forced to prioritize features with high risk and little reward). Dialogue, dialogue, dialogue.
  
• Not defining economic terms early: when you write the first line of code you should have an agreement on your reward. This rule of thumb can seem obvious to us, but remember that customers usually come from a whole different world.
  

It's a long list, but I feel that I have grown for more than five years and since I started my journey in computer science even before, I'm approaching the 10000 hours as a developer but still gaining basic experience in business. These errors are something I really had to try by myself.
Have you some freelance experience to share? What do you feel you could have done differently during your career?

Validity of development tactics

Many software development practices and methodologies are presented as panaceas and silver bullets, seeming to be valid in every domain and situation. But a responsible developer must be pragmatic (and a successful book and series started from this term) about where and when he applies his preferred technology, knowledge, or workflow.
I blogged some weeks ago about the overusing of a specific technology, but now I am focusing on the more general methodologies and paradigms like object-orientation, particularly if they involve killing a fly with a bazooka.
Let's start with a series of examples regarding field of application:

• Test-Driven Development might be the best and most controversial XP practice and is widely applied for producing robust and maintainable software. Though, it is obviously not applicable to application which are not object-oriented as you cannot easily isolate piece of functionality in structured programming; moreover, there are specifical tasks where TDD is an obstacle, like creating a graphical user interface or a throwaway prototype.
• Limitations are intrinsic in Design Patterns: Ralph Johnson, one of the author of the original Design Patterns book, affirmed along with his fellow authors in a recent interview that functional programming requires for instance different patterns than the ones presented in the book.
• Digging further, even object-oriented programming is not applicable in every domain and delopyment node, mainly where there is not the infrastructure to provide polymorphism and inheritance, like in embedded applications which typically employ the C programming language or in low-level operating systems routines. Some zealots will say that an object-oriented website will never scale, but this is an exaggeration.
  
• Agile software development is a great methodology for delivering value to a customer, but it is not suitable if your organization does not support it, or software development is not your job. One could argue that software development should be managed by professionists, but this is not usually true in the real world, where in-house programmers have more than one responsibility.
• At the programming level, using a Factory to build your objects is a good choice if they require external dependencies, but sometimes this is not the case. For instance objects that has to be serialized commonly are required to not have external dependencies, as you can always pass any service class trough the stack while calling a method. These objects are often declared newables.
• Version control is great and even solo developers should give it a try. I put nearly everything I produce in a Subversion repository, but I do not store binaries in it for example, since they can be generated from source code stored in the repository and they would only slow down the server while performing enormous diffs. In the previous post I said that Subversion and similar applications are general purpose systems, but even here there is a limitation in what they are meant for.

I can go on for hours and I'm sure you can find a flaw in every single practice I can mention here. As Fred Brooks wrote in his famous essay:

There is no single development, in either technology or management technique, which by itself promises even one order of magnitude [tenfold] improvement within a decade in productivity. --Fred Brooks

And it is indeed true that every practice we implement has limits in field of application and in the productity improvement it can give back to us. There is the temptation to apply straight our new knowledge or technology to every problem at hand, but instead of perform well the technique usually jumps the shark and produces an horrible result like the ones I listed previously.
Consider the TDD example: test-first programming helps us to produce decoupled components for our applications, and shrinks the time required for bug fixing. So should we apply this technique to user interfaces? I would say a big No as it would slow down our development, cluttering the codebase with brittle tests that have to change every day. So the improvement in productivity is not gained on all the production code, but only in the domain layer, which is fortunately the most important one. You also may have to write other infrastructure code which serves as the glue between layers. Is it useful to test-first this code? I think it is not. The effort spent for automating the tests and discover wiring or user interface bugs is often not worth the value they provide.
You also have to test the wiring of your application, and unit tests like the ones prescribed by TDD are not useful here.

Take care of what you learn and also research and discover when you can use it. Like in physics, no formula is always valid and you must put it in context before starting to tackle a problem with your swiss-army knife.

Links that made me think

Here's a roundup of the posts that made me think the most last week in the blogosphere and that deserve to be recommended. I also wrote a brief summary of the topic they talk about to give them a context.
I reserve some time every day to evaluate my Google Reader entries from an hundred feeds, and I think the ones I have listed here are very interesting readings. Do you use a feed aggregator? Do you find quality links in it?
I hope these authors feel appreciated for their good work.

Getting started with Scrum: a panoramic on this agile methodology, even if towards the end it digresses in marketing. If you want a brief introduction to agile estimating and planning this is a must read and there are links to explanation of terms like backlog, stories, etc.. though, there is no need for full-featured project management applications if you just want to give a shot to agile. Just organize the process with plain text files under version control instead.


Coding Simplicity: How to avoid feature creep in your life is an interesting parallel between life and programming. In both software development and personal development we should think before pulling in new features and activities: they may cost more than the value they provide, and your life it's not a toy project you can throw away.

Your company is insane because what they wish is that you could somehow get all the benefits of Agile without making any difficult or scary changes. We fight for unit testing, agile estimates and continuos integration every day, while people say "It's a waste of time, I just want to get it done". This post is inspiring and shows the reality: you cannot think that reading a book and attending a seminar will make you an agile programmer or a good unit tester instantly. There is managerial and personal work that has to be done before becoming gurus.

Design for testability talk is a one-hour video of a Google Tech Talk where Misko Hevery (Agile coach at Google) explains the pillars of designing an object-oriented application to allow ease of testing. The key concept is there is no silver bullet (also known as testing magic) in the testing world, that can be applied after classes are already coded. Testability must be included in the design and, not surprisingly, it improves its quality by forcing high decoupling.

A more practical link for Php developers: "Micro" Optimizations that matter where Brandon Savage points out the real techniques that cost nearly nothing in time and money, and can vastly improve a php application's performance. If your profiling result says that you need to speed up a large number of things, try Apc for example before refactoring and eliminate your framework because of its overhead. Often Apc cuts the time for displaying a page in half, without making any changes to your code.

Let me know if you have read other interesting articles on software development and engineering recently. Feel free to add similar links in the comments.

What drives development nowadays

Many blog posts (also of mine) contain acronym and abbreviations which have spreaded in the agile world. Every now and then words like TDD, BDD enter feed readers which might have still to learn what they mean.
To help fellow readers and to make a bit of humour on the number of methodologies which are growing, I made a research on all the techniques that shares the last part of the name.


Here is the list:

• TDD - Test-driven Development is a development technique based on prewritten test case and short coding iterations (5 minutes). The mantra of TDD is red-green-refactor, where red means that a test fails and green that it passes.
• BDD - Behavior Driven Development is a response to TDD which involves the definition of scenarios and tests involving different business contexts and units. The mantra of BDD is given-when-then.
• FDD - Feature Driven Development is the typical agile approach: develop some features at the time, in incremental steps. An iteration is the basic unit of time.
• PDD - Process Driven Development is a methodology for constructing non-trivial custom software applications by analyzing the business processes in which they will be used.
• DDD - Domain-Driven Design merits a mention although it is not a type of Development. The focus of this methodology is to not worry about the infrastructure layer which comprehends persistence, authenthication, etc. but to develop a powerful Domain Model which has no dependency on frameworks or libraries.

There are also less serious approaches, which are anti-pattern:

• ADD - Asshole Driven Development: a methodology where the biggest jerk makes all the big decisions.
• RDD - Resume Driven Development: a situation where programmers wanted to use tools, techniques, or technology to improve their resume instead of solving the customer’s problems.
• FDD - Fear Driven Development: there are different definitions of this anti-pattern, but I prefer when you have an amount of LegacyCode and maintaining the code is hard because you don't know what unintended effects your changes have.

This is a summary of all driven development approaches which I have encountered around the Internet. Do you know any other?

5 books to change your point of view

For a software developer that face the challenge of creating maintainable and extensible applications, which provide value to users, there is the need to learn something every day to keep up with innovation. Programming books are a resource to explore in depth a subject.
Books which treat a topic at a great extent are the simple tool to teach yourself practices, skills and technologies; but there are also classics (or wannabe in the future): each of these books paradigm-shifted my mind and I hope you can find in them some of the answers you're looking for, as they challenge the naive assumptions of software development.

1. The Pragmatic Programmer: From Journeyman to Master: this is one of the most practical books that I've ever read. It is free from mumbo-jumbo buzzwords and focus on a pragmatic approach to software development, a compromise between purity of the design and getting things done. It also explains Extreme Programming/Agile practices and is a complete read despite being only 300+ pages long; it does not cover in full depth its topics but it goes as far as possible to tackle the advantages of source control, refactoring, testing, plain text, and more. It was the first book for me that gave a sense to the agile word.
2. The Mythical Man-Month: Essays on Software Engineering: a classic from Fred Brooks. The man-month is the unit of measure for the work needed in a project: Brooks was the first to say that adding manpower to a software project would slow down the process. Even the Halloween documents commentaries cite The Mythical Man-Month; Plan to throw one away comes from here.
   
3. Patterns of Enterprise Application Architecture by Martin Fowler. While the original GOF Design patterns book it's about common repeatable object structures in everyday applications, Fowler's one talks of the infrastructure layer: ActiveRecord, DataMapper, Unit Of Work and so on, fighting the object-relational mismatch, presenting distribution and layering. After reading this book you will feel a compelling desire to write an Orm, and you will be more or less capable of doing it.
4. xUnit Test Patterns: Refactoring Test Code is a writing in all the patterns that a developer encounters when writing unit tests with an xUnit tool (phpunit, junit, nunit...). Who watches the watchmen? applies also to the test vs. production code duality and the maintainability of a project test suite is crucial for having it run by developers. The term fixture started to have a sense when reading this book.
   
5. Dive into Python, an introduction to the Python language, the one that Google uses. This is a free ebook and explains the basics of this duck typing total-object-oriented language, where 2 and True are objects and magic methods are born.
   

Maybe you have already a copy of some of these next to (or in) your development box.
What books made a great impression on you? I'd like to read what changed your way of thinking. Let me know in the comments.