A map metaphor for architectural diagrams

It is a (two-dimension) representation of a pipe.

The map is not the territory, but in software engineering terms they are models of it. A frequent remark about models is that they are all wrong, as they eliminate many aspects of what they refer to. Big efforts can certainly go very wrong if they don't consider something that turns out to be very relevant, but models in general try to focus on a few aspect that are relevant for a particular job to be done. That's what makes them useful despite their limitations. 

If this philosophy was more established, we would spare software engineers from many an effort to keep a visual model completely up to date with code that is changing at a rapid pace. The continuous invalidation of a diagram leads to the conclusion that it's not abstract enough, or that it's the symptom of other larger problems.

Coming back to the map metaphor, geographical maps emphasize one primary aspect, generally mapped across two dimensions. For example, elevation or depth with respect to sea level:

or political borders:

 

Like in Wardley maps, architectural maps can use meaningful spatial dimensions, even if they don't correspond to a physical location like the squares on a chess board or a projection of the world. In a Wardley map one key dimension is visibility (to the user) as they capture a value chain where products use components which use other components or utilities.

Unlike the modeling happening in Domain-Driven Design, I'm referring mostly to models of the software itself here rather than models of the domain; domain models in diagram and code format are just one specific example of the activity.

Which maps does a team need?

To answer this question, consider the goals for a team to:

• identify and capture what "good" looks like for them in this project
• easily see divergence from that to correct it
• take fewer decisions over and over; reuse patterns that have emerged, and have been tried and tested multiple times already in similar changes

Some would call this documentation, or diagrams, and they would try to fit into some formal notation to be completely unambiguous if these were published in some application for funding. These maps are lightweight, they are only meant to be internally used by a team, and there is no formal notation as the barrier to entry (much like Eventstorming involves everyone in a room with only a set of orange stickies).

I've seen a basic Miro palette emerging depending on what the team is comfortable with:

• boxes and arrows, with a couple of style of arrows if necessaries
• stickies to capture particular decisions (text)
• color coding to track status (e.g. adopted as best known practice, team fully endorsed it, deprecated/suspicious)

Substitute your favourite digital whiteboard products. I have not attempted to do this in a office setting. I suspect Miro lowers the cost of change to make this feasible, both in a sense of not churning through paper and markers and in a sense of having a simple enough UX that can be picked up in a day to allow contribution.

The first one of this maps replaced an attempt at maintaining Architectural Decision Records. Software engineering involves continuously taking hundreds of decisions, in different places, at different scales. I suspect ADRs cover the very high level perspective, or consequential decisions; but they don't scale to a high number of decisions that delve into the inner workings of a smaller module without reaching line-of-code level. Different abstractions for different purposes.

ADRs are also immutable and go through a deprecation and superseding process. These maps are supposed to be mutated all the time, for refinement. The continuous application of new user stories applies the pressure to revisit decisions to better suit what we now one or two years after a product was created.

Some real world maps covering the same project

To keep scaling to a large number of decisions, I started classifying them into various different maps segregated by a specific aspect. There's no right choice on how to organize information into a hierarchy, but there always was a cognitive load limit on how many decisions can be quickly grasped or considered when looking at a particular component.

In more formal approaches these separate aspects are called views or viewpoints. In any situation they might arise to support a specific discussion rather than just because there wasn't enough space in another map. The hard step is possibly to move from one model to multiple models, while maintaining the same attention and ownership from the team.

Here are some examples. The main dimension has usually been user visibility, but from left to right rather than from top to bottom like in Wardley maps.

The context around $productName

Includes no details about the internals of the system being worked on, but only other projects or organizations that the current system integrates with. Many things are out of the control of the team in this map. They are also often not visible in code at all, including for example a list of clients of our API.

I took this name from the C4 model's system context diagram.

Bounded Contexts and their languages

Where does a vocabulary apply consistently? For example, do we have a UX language to cater to users and a separate, symmetrical and consistent language for the underlying domain model? And how do they differ from the languages used or imposed upon us by third parties? Are we conforming to another team's language, or introducing anti-corruption layers?

I took this name from strategic Domain-Driven Design.

Static architecture

The dependencies between different modules of the codebase. Imports, requires, use statements depending on your language of choice will ultimately define this. It's a static map because this information can be detected and distilled into the map without running any code. There are also additional decisions that are not represented in code: which committed folders are modules with a strong interface, and which is just a folder?

A key map to foster cohesion (inside a component) and keep coupling under control (as it makes high-level dependencies visible).

Testing map (or testability map)

What testing strategies are used consistently in any application layer? Unit testing classes or functions? Screenshot reference testing for the UI? What integrated tests are we using and what choices have been made for their setup or assertions?

Observability map

What modules produce useful logs and how can I access them? What is the difference between an error or a warning level log? What dashboards (sometimes disparate) we should link to? An entry point more than something that can visualize data directly, often linking to disparate tools from Grafana to Kubernetes dashboards.

Technology stack

Answers questions on specific of various programming languages and tools. Does not need to capture what can be enforced via into linting rules instead, but the set of decisions includes more than just conventions:

• what safe subset of JavaScript or Typescript are we endorsing for usage?
• What we will use to represent URLs or dates?
• How do we mark deprecated code intended to be replaced?

The choices made here often have security, performance or other non-functional implications.

Process and ownership

This was not a complete list! The set of maps should be owned by the whole team that uses them, not by its tech lead only. Decisions on retirement of maps can then be taken together. 

Often, though the tech lead role has their senses oriented to detecting when a new map could be helpful; proposing its adoption at the right moment; and intentionally not filling it in completely to co-create it with the rest of the team. 

As part of the development process, the team self-organizes themselves to pick up user stories, and often has a refactoring checklist that they want to achieve before closing off their unit of work and move onto something else. Conceptual breakthrough might also have happened as part of delivering value, like a new data structure having been identified and tested successfully. While most of the notes will be deleted as the team moves on, some can be captured by refactoring the code to follow our understanding. And some are hard to fit at that level and can be captured by maps. 

Often items are marked with a specific color, indicating a pair or a subset of the team has stored these decisions but there is some catch up to do with the other team members so that they are all aware of the direction. 

Speculations

A gap I would have liked to be filled regards explicitly referring to maps at the beginning of a new unit of work, at some granularity. For example, consulting them when a new epic or new user story is prioritized. 

The results might be various:

• speeding up as fewer decisions have to be taken, and maps constitute an enabling constraint. I've seen this happen empirically more by referencing existing code. There is a trap waiting as developers might pick an outdated item to copy from, and they rely on memory to come up with a recent reference rather than the old button no one has not touched in the last couple of years.
• invalidation of decisions. New business requirements may require a change in architecture to support them, hopefully infrequently.
• refinement. Some aspects we might find out of date, or obsolete, and maps could be simplified as a result.

One way to look at this process could be evidence-based: can we work within the existing architecture to deliver, or will we fail to do so? A spike can help us understanding which one is the case. However, failure is not binary: we might never encounter an absolute blocker to delivery, and yet spend sweat and tears before deciding a new approach is needed.

Team learning session: surviving legacy code by J.B. Rainsberger

This is the description, and experience report, of an exercise I picked up many years ago and then used in a team I lead. I'll describe this in the present tense as I imagine applying the exercise in similar contexts. Credit to J.B. Rainsberger for coming up with this diabolical codebase and make us work on it!

Context

A team has been formed and has been working on a single product, for example a TypeScript monolith comprising a frontend and APIs.

The team is now tasked with taking over a set of services and frontends written in PHP (or some other programming language and ecosystem). It turns out this set of services is the flagship product of the organization, and it needs to support innovative business change. Yet, normal operations and incremental improvements had been mostly outsourced as part of the technical strategy for the last few years. 

The focus of innovation was elsewhere. Testing coverage is redundant, confusing or missing, depending on the area you are working on. Very few people in house know the traps of these codebases: we are now in the realm of legacy code.

Learning goal

The original exercise by J.B. Rainsberger is oriented to learning patterns to take over legacy code: inherited codebases that exhibit a lot of business value. If they don't have business value, then why working on it?

Specifically, the patterns regard:

• characterizing and understanding code
• testing it, at various level of scope from whole application to small units
• refactoring, with a safety net in place to support those changes without introducing regressions.

There isn't an emphasis on shipping new features, and there is no product owner.

To the original goals, we add a separate one here with its own new difficulty level: learn a new programming language that most of the team has not worked professionally in before, or has not picked up for a long time. With a new programming language, there's also knowledge of a new set of tools for running, testing, or linting the code. For PHP alone, in this example, the toolset ranges from Composer to PHPUnit, PHPStan, or PHP-CS-Fixer.

The learning at all these levels translate in being ready to use these patterns and tools on real projects, making it much easier and safer to deliver business value on those.

Activity

The original problem statement gives this guidance:

• Refactor this code to understand it
• [Decide] when to refactor and when to rewrite, and how to do that safely.
• lResolve the central conflict of legacy code: I need tests to refactor safely, but I need to refactor to write tests effectively.

I've been running these workshops back in the days of the Legacy Code Retreats, which is where the name came from. The setting diverges from a whole-day session, where iterations are handle by various pairs. In this case, the repository is forked at the beginning of the activity and several sessions can take place, for example with one or more hour per week reserved. The participants work together as an ensemble, but they could split into multiple rooms working independently if there are too many.

11? 12? What?!

A loose plan to follow can be:

1. run the application. Does everyone understand the problem domain and what problem the code solves?
2. introduce golden master testing for characterization. This step involves figuring out seeding and reproducibility, and how to achieve the isolation of automated tests from the outside world and its changing conditions.
3. introduce further testing at lower levels. This step involves introducing tooling for running code easily, testing it, or performing static analysis; manipulating the folders and file structure safely; and refactoring the code to isolate the units under test.

I act as a language expert here, but not with an agenda in mind. What to learn is decided by what moves the participants want to take, and what they are missing to be able to do so. It also pays off to prompt the team to research rather than trusting anyone's memory or word: tests and experiments are the source of truth.

The fact that this is a completely new, fictional, and ugly codebase should help with the feeling of safety when raising lack of understanding. The code is supposed to be hard to understand and fragile. Once that is established we can then work on our own learning, directed to improve our situation. It's a very different framing that delivering features on real legacy code with a deadline in mind.

Resources

The original code repository used in legacy code retreats.

The canonical explanation of a golden master approach to characterization testing.

Specifically for PHP, it helps to understand how seeding random number generation works to achieve reproducibility.

Retrospective

Add to a board two prompts to allow reflections to appear concurrently. For example:

• what did we learn today? Anything from a language construct that did not map easily to something I already knew, to a tool's use cases.
• any ideas for next time? There are lots of potential directions of exploration, and we want to crowdsource the gaps the participants are starting to see so we can fill them. This helps getting into context quickly when we start a new hourly session at another date.

Team learning session: the (in)famous types folder

This is the description, and experience report, of an exercise I used in a team I lead. I'll describe this in the present tense as I imagine applying the exercise in similar contexts.

Context

The team has been working on a Typescript monolith application; Typescript is used exclusively on the backend and compiled with Node.js as a target.

Encapsulation and modularization have been promoted as important software engineering concepts, and some high-level architecture and modules are emerging. For example:

• a clear write and a read side in a CQRS application
• a third-parties module to isolate from external dependencies
• a top-level evolving http layer to isolate the Domain Model and the rest of the code from HTTP and other delivery mechanisms.

The team is oriented to functional-programming, modeling some domain types but also keeping related functions close to those types for cohesion. Or they could be working within the object-oriented paradigm, similarly organizing the domain model and other code with types and classes.

For the rest of the discussion, the word module applies to units of encapsulation of varying sizes. In a TypeScript monolithic codebase, this could be a folder with a index.ts file; depending on its size, it could contain other sub-modules following the same structure. This exercise is also focusing on compile-time dependencies, mostly import statements.

The types folder refers to a (anti-)pattern where shared types are collected in a top level, or very high level, folder. The types can then too easily be imported anywhere, with a thick tree of dependencies towards the types folder being established, inadvertently coupling together disparate parts of the codebase.

Learning goal

The team is picking up encapsulation and information hiding at the small scale, encapsulating functions in a folder behind a index.ts and its export statements.

We want the team to expand this process to larger scale modules, sized at 1K to 10K lines of code: how much code can be hidden inside within these modules, or added to their public interfaces to elevate it as a contract between modules?

We also want the team to think inside a monolith as we are not sure there is enough capacity to address the overhead of separating repositories and services.

Activity

The original text of the exercises follows, edited for clarity:

Our src/types/ folder acts as a catch-all for types and related functions when we do not find an encapsulated place for them to be placed into. While our architecture is often changing for the better, over time these global types remain visible everywhere in the monolith's codebase even if hiding them could help working on the code that does not need to know about them.

For each of the .ts files linked here, work as an ensemble to ask the following questions before deciding any refactoring move:

• Where are the contents of this file currently used?
• Are all the usages legitimate?
• Are these file contents addressing too many responsibilities?
• Do the names reflect the intended responsibilities or usage?

Once you decided on a new location for the file (or for part of it), ask:

• Do the resulting import rules agree with our architectural maps?
• If we change a decision within the newly encapsulated file, how far does the change propagate?

This is not a backlog to clear: do not rush to cover them all (there are 32 in total anyway). The goal is to identify patterns for architectural refinement.

Some example of problematic types we found when running this exercise, with real named and an general explanation:

• RecordedEvaluation, Group: used only in module A and module B that depends on A. Yet globally visible.
• GroupId: used in multiple modules that relate to writing and reading to a database, yet visible to the HTTP layer.
• EvaluationType, ArticleServer: union types of a few strings, global for convenience but also only used in a few modules.
• CommandResult: a return type and part of the contract between module C and D. Module C depends on module D. Should this be defined by C, D, or a third module to break the dependency completely?
• DescriptionPath:  a branded and validated string. The same problem could be solved by ensuring validation happens in the right modules. Should this type be simplified to a string?

A map of (compile-time) dependencies between high-level modules, intentionally zoomed out. Dependencies are generated by import statements originating in one module and reaching out for a type or a function contained in another module at any level of nesting.

Resources

A small guide to barrel files (index.ts files containing only export statements) can be useful. They are often mentioned as having performance implications, but if there are any measurable drawbacks they should be traded-off with maintenance.

dependency-cruiser can be used to help map out the current dependencies of a codebase, generally at a very fine-grained level rather than top-level modules.

Retrospective

A couple of questions can be asked here in a round robin, or for lack of time written as prompts on a digital board for participants to brainstorm about for a few minutes.

As often in quick retros, one question is looking backwards into what has happened and one is looking forwards to improve our practices:

• did you see a type which looked reasonable before today, but now is a target for refactoring?
• what would you be differently tomorrow when you create a new type?

The reflections that emerge from all the participants help set a shared direction for the architecture that you want to see. Enjoy!

Ensemble programming roles and cues

In the last few years I've been leading a team that has adopted ensemble programming and pair programming as the main practice to deliver production code. We have evolved cues that have a precise meaning for team members and that make communication more effective, especially when some of these phrases are used every few minutes.

I considered various terms to indicate this set of phrases, and discarded most of them:

• mantra (a slogan repeated frequently,  but often to yourself, and with spiritual implications),
• shibboleth (implies a closed group having secret handshakes and hiding from an external force)
• catchprase (a signature phrase associated with a specific character)
• formula (works in other Romance languages but has many other meanings in a English context)

Formula could be considered a loan word from Italian, but I think its original etimology fits:

1630s, "words used in a ceremony or ritual" (earlier as a Latin word in English), from Latin formula "form, draft, contract, regulation;"

To keep with the metaphor of an ensemble of actors or musicians performing, I found cue was fitting too:

the trigger for an action to be carried out at a specific time

A few roles in ensemble programming

The rotation of roles visualized on a Miro board

This experience is in the context of remote ensemble programming, where there is a single audio medium shared across a virtual room, and limited surface space for visual cues such as body language or facial expression. One of two monitors, or part of a monitor, would usually be dedicated to 2 or 3 other camera feeds, with a digital whiteboard or an IDE on a shared screen being the place of operation. This isn't to say that paying attention to your colleagues isn't prioritized; rather than making use of continuously-improved verbal language is one way to achieve that.

Within this setting, a few roles are assumed by the team members participating into a particular ensemble:

• the Driver inputs all changes into a working copy of the codebase.
• the Navigator has the responsibility to coordinate the group into the next decision it needs to take, and verbalize all changes for the Driver to enact.
• other ensemble members assume roles as needed and without even noticing, actively contributing to the discussion. They might help with diagrams or note-taking, or notice code smells or other people dynamics since they can dedicate their attention to something different than the next code change.

In this team, a rotation based on committing and pushing the last change emerged. A strict rotation where a navigator remains in that role for the whole duration of the cycle also emerged, mostly to keep a level playing field across all levels of seniority and giving the space to everyone to lead the next change.

The cycle described here is the one of starting a new screen share session; pulling the latest version of the code; making a change and pushing it to make it available to the rest of the group. This micro-iteration has an associated cycle time, which is where I believe the term came from. After each cycle, the previous Navigator would rotate in to become a Driver; someone else would become a Navigator.

Many of these cues are commonly uttered from the Navigator or the Driver, but they are not limited to them. Consider this list harvested in a design patterns sense: extracted from repeated real world experience; no claim to completeness. It is however limited to a single team, operating in various context and changing its membership over time. Crediting Kevin Rutherford for introducing ensemble programming when this team was newly formed at the beginning of 2020, and helping the team honing in on particular solutions over time.

At the beginning of a cycle

"What do you want to see?"

The Driver may start a cycle asking this to the Navigator, to relieve pressure on deciding a direction quickly. To ease into the role of Navigator, there is little downside in spending time observing or understanding some area of the code, rather than attempting to change it immediately.

In the context of new ensemble members, this cue also relinquishes control explicitly to the Navigator. If you are used to the person with control of the keyboard showing what they mean, you will instead seen a Driver waiting for instructions.

"I'm just a pair of hands" or "Waiting for instructions"

Indeed, the Driver uses these or other cues to remark how they are not taking decisions just because they happen to hold the keyboard or the mouse that can perform changes. The Navigator has the responsibility to coordinate. The driver can focus on efficiently executing a move.

"Let's go to the IDE/"the code"/Github/maps/Miro"

At the beginning of a new cycle, or triggering a new cycle, an ensemble member suggests to change the visual support to fit the discussion. This might involve not sharing a screen anymore if that medium supports collaboration natively e.g. Miro as a digital whiteboard.

For example, Github could be useful to navigate unfamiliar repositories; a whiteboard is vital for remote Eventstorming or for diagrams; maps indicate a set of long-lived architectural diagrams for context, dependencies or separation of concerns.

During a cycle

I think this is where the phrase comes from. But lack of verbalization of intention should be the exception, not the rule.

"Make it so"

The group has shared understanding of a change that needs to be applied. For example, we slice a change into many small steps; there is low risk because we already performed a few of these slices; uncertainty might lie more related to what can be discovered from the compiler or a test suite reacting to the change; or a refactoring move may need to be applied to a different part of the codebase, in what we suspect will not require new design decisions.

After having stated the change they'd like to see, the Navigator hands over control to the Driver with this cue, and lets them loose in achieving it with their preferred tooling: VSCode, Vim, grep, even an LLM. The separation of concerns is between the destination, and how to get there.

"Rename the ... class/function/method/variable and all references"

The Navigator directs the Driver at a high level of abstraction. The Driver is a intelligent IDE and applies this change predictably as they both understand what the outcome should look like. In case the Driver runs into trouble such as unforeseen situations, they stop and ask for more precise direction.

"Can we scribe on the board something about ...?"

The Driver or the Navigator asks other members of the ensemble to take a note about something that we should look at later. They could just ask to take a note, but we picked the word scribe from Eventstorming practices to identify whoever is currently responsible for creating artifacts to a shared digital board.

Assigning this role, for example, to the third ensemble member in the rotation helps avoiding concurrency clashes, with multiple people trying to take the same note.

"From the back of the room, ..."

Other roles than Driver or Navigator hand out suggestions or considerations, without wanting to interrupt or override the navigator's intention. The Navigator still coordinates making a decision, but the other ensemble member can contribute specific knowledge or cues that help the group make progress without taking away the opportunity for the Navigator to exercise their skills.

The expression comes loosely from the phrase "leading from the back of the room"; I've seen this role referred to as the Rear Admiral in some ensemble programming literature, turning the Driver and Navigator rallying metaphor into a naval one where everyone helps running the ship.

"... can be a rabbit hole"

A problem we face is recognized as something that might be taking away from the momentum of the group, or its overall importance is controversial. It might be more productive to mitigate it now, rather than digging into its ultimate causes. The group can agree to postpone the discussion of this problem, to focus on the next test to write or to make green. If the problem makes it very difficult to make progress, it will keep emerging. Rabbit hole is just internet slang for an engrossing and time-consuming topic such as programming language trade-offs; a troublesome library upgrade; a non-deterministic test; or large-scale architectural changes that require a lot of evidence collection and consideration.

At the end of a cycle

Group peer pressure helps doing better than that

"Anything else for this cycle?"

The Driver is asking the Navigator about whether the intended scope has been achieved. Confirmation will start a commit process if no other checks or changes are necessary.

Sometimes the Driver acts as a preemptive time keeper, reminding everyone uncommitted code is work in progress and the bigger it gets, the higher the risk of progress being lost if it triggers inadvertent behavior changes in the codebase.

"End of cycle" or "Let's rotate"

The Navigator decides it's time to rotate the roles, considering what has been learned in this cycle enough to perform a switch without loss of context. The cycle might have been focused on investigation or reading code, given in this case it ends without a commit. Even if there is an attempted change at play, the knowledge gathered can led to the next cycle to regenerate the same git diff quickly, or to find a safer or easier path through that change.

"make check and commit" 

The navigator communicates it's time to commit what we have done. Substitute make check with your local testing command of choice; it's just a team convention to capture a set of compilation, static analysis and tests that we deem acceptable to run before every commit. Other more specific or intensive testing can be left to a Continuous Integration build as a safety net that will constitute a longer feedback loop. Normally, the commit is followed by a push and a default end of the current cycle.

"What [commit] message would you like?"

A Driver understand we are ready to commit a change, and invites the Navigator to summarize and describe it. This is an opportunity, if the team wishes so, to try other cues to repeatedly nudge towards a certain behavior and getting into a new habit. For example, you could experiment with having Drivers ask "Why are we doing this change?" for a whole day.

I just want to run a container...

 ...is a developer-centric point of view?

Generic data center image to exemplify a place many of us have never set foot into.

After a few hours spent on upgrading the toolchain that starts from Terraform, goes through a few AWS-maintained modules and reached the Elastic Kubernetes Service APIs, my team was entertaining the thought of how difficult it is to set up infrastructure. Infrastructure that, for this use case takes a container image of a web application that Docker generated and run it somewhere for users to access.

Thinking through this, either Kubernetes is the new IBM (no one ever get fired for choosing it), or there is more to a production web application than running a container which is what Kubernetes is often sold as: a tool to run containers in the cloud without the necessity to set up specific virtual machines, but treating them as anonymous, sacrificeable nodes where the aforementioned containers can be tightly packed and sharing CPU, memory and other resources.

What exactly is the infrastructure doing for us here? There are a few concerns about operations, the other side of that magic DevOps collaboration. For example:

• the container image is exposing an HTTP service on port 80. This is not useful for a modern browser and its padlock icons, so to achieve a secure connection a combination of DNS and Let's Encrypt generated and automatically renews certificates after verifying proof of ownership of our domain name.
• the container image produces logs in the form of JSON lines. Through part of the Grafana stack, these lines are annotated with a timestamp, the container that generated them and various labels such as an environment or the name of a component of the application (think frontend or prod or staging). After these lines are indexed, further software provides the capability to query these logs, zooming in on problems or prioritizing errors during investigation.
• if we get too much of these logs at the level of error for a particular message, we'd like to receive an alert email that triggers us into action.
  
• invariably applications benefit from scheduled processes that run periodically and independently from the request/response lifecycle. It's such a useful architectural pattern that Kubernetes even named a resource after the cron daemon, introduced in 1975.
• it's also very useful for applications to maintain state and store news rows in a relational database. Provided turnkey by every cloud, a set of hostname, username and password allows access from any combination of programming language and library. No need to worry about rotating the logs of Postgres anymore, or that we are running out of space.
• the data contained in the application and generated by user also lends itself to timely analysis, so that we know whether to kill a feature or to invest in it. This means somehow taking out recent updates (or whole tables) into a data pipeline that transforms it into something that can be analyzed by a data scientist. All with an acceptable speed.
• Those credentials nevertheless need to be provided to the application itself, hopefully without accidentally disclosing them into chats, logs, or emails. No human should need to enter them into an interface.
• Configuration is slightly easier to manage than credentials due to the lack of sensitive values, but we'd still like the capability to generate some of these values such as a canonical URL or to pass in different numbers for different environments.
  
• And of course, whenever we commit, we'd like to deploy the change and start the latest version of our application, check that it is responding correctly to requests, and then stop the old, still running one.

This is all in the context of a single team, running operating a single product. Part of this is achieved through running off-the-shelf software on top of Kubernetes; part of it by paid-for cloud services; part of it by outsourcing to a specialized software as a service.

However, when we think about software development we don't necessarily think about software operations. Many of us started our careers by writing code, testing and releasing it without a further look to what was happening afterwards. DevOps as a philosophy meant bridging the gaps; You Build It You Run It is the easiest summary for me.

Yet what I see and what I hear from experienced people is that silos in infrastructure seriously slow teams down, and create fears of insurmountable problems. Abandoned projects, "finished" from a development point of view, but now having unclear ownership and running in production supporting the main revenue stream of an organization.

So I don't want to just run a container. I'd rather deploy many incrementally improved versions of a container, monitor its traffic and user activity, and close the feedback loop that links usage to the next development decision.

In other words: I can always build simple code if it doesn't have to run in production.

A year of mob programming, part 5: methodology

I wouldn't call mob programming a methodology, but rather a practice that can be adopted. Like methodologies, however, it only surfaces problems and questions faster rather than providing solutions.

For example, a team might have a low bus factor due to the specialisms of its members, and only one person might be used to edit code in a certain language or application. Other team members might have gaps, such as not being set up with the tools and credentials to work on infrastructure and operations; or lacking the knowledge to perform screen reader testing for accessibility.

Besides exposing bus factors and gaps, mob programming brings design conflicts and opinions into the open. It takes little effort to turn the other way when looking at a pull request, or to switch to busy work or generic "refuctoring" when alone; disagreements and questions about value can often prop up under your nose inside a mob session.

However, that doesn't mean that the practice automatically gives the team the psychological safety to talk about those issues out loud, and to resolve them. That's like asking an hammer to walk to the shop and buy the right nails.

At the last part of this post, I thought I'd compare my mob programming experience with some of the theory; the theory being famous Agile software development methodologies and their values or principles.

The serene predictability of waterfall software development

Within Extreme Programming values:

• Communication inside a mob is very high: synchronous conversations supported by a visual medium such as code or a digital whiteboard.
• Simplicity is fostered by people in a diverse team reminding others that You Aren't Gonna Need It, but also needing to understand what's going on at all times.
• Feedback from the rest of the team, or even from a customer proxy, is immediate.
• Courage is needed to be a driver in an unfamiliar setting, or even a navigator asking the mob for help on a task we are uncomfortable with.
• Respect is necessary to work face to face for prolonged periods of time.
  

Within Lean values, mob programming:

• eliminates waste such as partially done work: there should ideally be no pull request open or branch, as code is produced and integrated in the mob continuously;
• eliminates waste such as outdated features or gold plating: when you include a product owner in the mob, it's easy to steer and change the scope upon feedback rather than plowing on with the original idea;
• eliminates waste such as handoffs: there is no packaging up of code for review, or of pixel-perfect Photoshop design to front end developers, or front end requests to back ends to be implemented, and so on.
• reduces task switching: since everyone that is needed for progress is present, you often don't need to switch task because of waiting on someone, but can just carry on.
• amplifies learning: ideas are tried out in code from the get go, and most of the team can learn from the results including the product owner.
• optimizes the whole: most of all, we are not optimizing how much we can use everyone's time, but how fast we can flow a single piece from idea to customer value.

Image by Yann.

A year of mob programming, part 4: the remoteness of it

I don't have a frame of reference for mob programming in a co-located environment, besides conference (or team exercise) workshops that didn't involve production code.

On a video call, the driver is still always explicitly defined as the person sharing their screen and tools. This is similar to the person physically having the keyboard when you are all sitting around a table.

The navigator can either be more implicit, move around or be strictly defined by a process. If an implicit navigator doesn't emerge for the current commit, it should be considered normal to quickly nominate one as needed. Over time, we normalize the phrase "can you navigate [that change]?" in response to a proposal.

Due to latency, in a video call environment there is a sort of Lorentz time where you may hear overlapping voices when someone else doesn't, and vice versa. It all depends on how much latency each video link is experiencing, and the adjustments that software makes can be jarring, such as video freezes or audio being delivered 10-20 seconds later.

"Sorry, two people talking over each other" "Really??"

I'm sure though everyone has already experienced people talking over each other in a physical room: it isn't exclusively a remote working problem and relies on team interactions more than technology.

We also want to communicate and especially give feedback through voice, as people's faces are not always visible to a driver looking at code; nodding doesn't necessarily help.

Perhaps a bigger problem is the fact that video calls don't allow side conversations to happen, as there is a single audio channel that our ears cannot separate into directions. This is likely to impact a massively parallel EventStorming around a whiteboard, less so an ensemble of 3-4 people trying to get to a consensus on their next move. The side conversations however include those famous water cooler talks, bumping into people in the kitchen or while on a break.

Your theory of mind might have more trouble cutting through the limited signal, and understand in what mental state other people in the team are; and whether today's problem is more due to Internet connection trouble, or a real conflict between team members. The daily retrospective helps to let people put tools down, and reflect on the day; possibly proposing experiments for the next session. 

Looking at the world around me, I'm quite sure I would be as depressed by remote work as many, if it consisted of me being alone for most of the day. So due to mob programming I have a different perspective on how incredibly engaging working from home can be.

Stay tuned for the next (and last) part of this post, Methodology.

Image by Duesentrieb.

A year of mob programming, part 3: a laboratory for team dynamics

Our implementation of mob programming consists in a permanent Google Meet videocall, with camera on, in which we rotate a developer sharing a screen containing IDE and browser. So there are two unusual practices to get used to: being in a group all the time, and being on a camera all the time.

I was originally surprised on how both practices went from exhausting to being a normal work day. This is my experience, and we have to consider neurodiversity and personal preferences in the team on having a long screen time and continuous social interaction. It's almost obvious to say, but sometimes people hate inconclusive meetings in which they have no say; not being together with other human beings.

Don't underestimate your capacity to adapt, but recognize that your energy is not going to always be the same; whether because you got a cold, or something is going on at home, or some task is particularly draining. 

The Flemish inscription at bottom reads: because feedback is perfidious, I will go code in my cave

Gold cards and spike branches give people the ability to work on their own when they request to. Mechanical work such as upgrade of dependencies or investigating logs can benefit from focused time from one person. If you were in an office, you could go to an isolated room; working remotely, it's even easier as it just consists in leaving the video call and coming back refreshed later.

The opposite can also happen, when someone like me can manifest Fear Of Missing Out and the mob making lots of decisions in their absence. As the team gets to a norming phase, we should expect fewer surprises for members coming back to the mob after some time off.

Being together with the rest of the team can make your motivation higher due to the group support in getting started or unstuck from a particular problem; but of course the whole group can struggle on occasion.

Every time there is a new composition, though, there are changes in how the mob is working and what it pays more attention to. And from the point of view of a growing team leader, being embedded in a mob environment means being positively inundated with information. The challenge is to make sense of what we see to understand where team members are struggling and what are they finding most effective.

"I wish I could see players only in 121 meetings" -- No football coach ever

This requires a lot of (cognitive) bandwidth, and I usually can't at the same time focus on technical architecture and on social roles. But maybe this is just an argument to work in a team where the different hats can be rotated; as much as there is always someone writing code, there can always be someone checking if we have started to talk over each other, or we are falling into a rabbit hole of an hour without committing.

Stay tuned for the next part of this post, The remoteness of it.

Images by Michael Kranewitter and in the public domain.

A year of mob programming, part 2: Collective Code Ownership

With respect to a team assigning tasks to developers by their function (frontend, backend, infrastructure, and so on), mob programming fosters Collective Code Ownership.

This is more in the sense that everyone should be able to contribute through the mob in any area, rather than everyone being able to prioritize on their own what needs to be improved. The pact is that our time belongs to the team, and the team decides what is important.

A result of this is an healthy prevention of knowledge silos, where you can exercise your own creativity just because no one else knows how to work in that area anymore.

Code review happens continuously in the mob on small changes, and there are no huge feature branches to go through. In large changes such as those, it's too late for review to suggest a valuable but completely different approach; there is too much investment of time and energy into code. If review dares to suggest groundbreaking changes, it causes extensive rework instead.

There is a larger picture on ownership: team members tend to keep track of what they care about, whether that is consistency in architecture, front end approaches to styling, performance issues, and so on.

Sometimes I found myself being talked out of a design I had in mind, counteracting bias I could have for the first or most familiar solution. I started working on this team with an object-oriented mindset, but we have transitioned to functional programming in TypeScript.

Sometimes, there are hills to die on: hard to reverse decisions on architecture, or programming language choices. It's a job of a psychologically safe team to discuss these choices collectively without drama or oversimplification like JavaScript being the the death of computer science. 

"I told you we should have written the code in English, not Latin!"

It tooks three different iterations to get a repeatable pattern for accepting Commands (as in CQRS). But it was more fruitful to focus on the Domain Events design, as a difficult choice to revisit, than on the shape of the code itself which can be refined at any time.

Working outside of the mob

The counterpart to the XP mantra of writing all production code in pairs (or a mob) is to allow team members to contribute when they are working alone due to other necessities, such as an unstable Internet connection or a flexible day where they can't align completely with the core hours of the team.

I have learned to ask the team to be commissioned something to do, or at least to give them a choice. This keeps the prioritization in the hands of the group, again reducing bias.

It's helpful to assign problems that have only a constrained possible solution such as renaming, or propagating a refactoring through the codebase for consistency's sake.

It also helps to report back what you have learned during a solo coding exercise when rejoining the mob; unexpected issues or decisions you had to take on the spot and you feel unsure about.

You're allowed to stop at a certain point, as making progress alone at all costs is valued less than maintaining collective code ownership and a shared understanding of how our application works.

Technical spikes can also be chosen by a single person to work on, either because they want to individually prioritize them to demonstrate an idea; or because the amount of investigation required makes it difficult or frustrating to collaborate.
Spikes can be built on throwaway branches, being optimized for learning rather than for delivering production quality code. Once an idea has been demonstrated and approved, the mob can implement it pretty quickly on the trunk, and if we believe in the practice, with an higher level of quality. I constantly find feature branches worked by a single person to be a dead end (especially if they are my own).

In the end, people are frequently in meetings, researching or even just on holiday for a given day. Hence there's always someone missing that will catch up on the progress when they come back into the mob. But the group itself never stops even as the components change, so some of that progress will happen every day, other things being equal.

Stay tuned for the next part of this post, A laboratory for team dynamics.

Image by Johann Jaritz.

A year of mob programming, part 1: metaphors

I've been practicing remote mob programming in my team for more than a year, writing more than 90% of production code inside a video call with multiple developers looking at the same screen.

A bit of context

I work in the scientific publishing domain, on web applications oriented to help scientists throughout their day.

My current team has been formed from the start as a remote team, and has been working with mob programming for more than a year. It is a cross-functional team that is including a product manager and often a designer into the (virtual room) for co-creation.

Our set of technologies includes TypeScript, lots of semantic HTML, and minimal client-side JavaScript. Container-based infrastructure and databases are pulled in as required by the evolution of the product.

This is an experience report, not advice that can be applied blindly to your team or organization.

Metaphorical definitions

All the brilliant people working on the same thing, at the same time, in the same space, and on the same computer -- Woody Zuill

Like pair programming, mob programming can be simplistically explained through the metaphor of driving a car in a rally. Or at least, that's how I always understood the driver and navigator pair.

• A driver shares their screen and has sole access to the keyboard. They are an intelligent IDE capable of executing mechanical refactoring and fixes, but not of deciding a design direction for the code.
• A navigator verbalizes what the driver should be doing, making decisions that reach down to what test or line of code to write.
• The rest of the group is on the back seat of the car, and can speak at will, volunteering information to the navigator or answering questions. Are we there yet?
  

The roles of driver and navigator are rotated often, in fixed time or scope increments such as on every commit. The navigator can implicitly move continuously from one person to another, as long as there isn't more than one navigator at once.

Mob as a name

Speaking Italian as a first language, the term mob doesn't necessarily evoke an emotional reaction from me. But I can recognize a term that can be used to describe organized crime might not allow team members to feel comfortable. My understanding is that mob has to be intended as crowd rather than mafia.

"This is so wasteful. Why aren't they all in separate rooms playing 4 different songs?"

Nevertheless, we looked for more metaphors and terms that can be used to talk about the group of people that work together:

• orchestra emphasizes the coordination required by a group of people, and their simultaneous presence. I'm not sure they need a conductor though.
• ensemble is a similar musical metaphor, that focuses on a small group of complementary roles closely working together.
• swarm refers to a group of bees all converging towards a single task to get it done.
• the team itself ideally coincides with the mob, though there can be non-technical roles that find difficult to contribute all the time; the mob can also be temporarily reduced from the full set of developers in the team, or multiple mobs can appear in a relatively large team.
• a womble is a fictional furry creature from children's books that helps the environment by cleaning up rubbish. It has no relation to groups whatsoever and is an unrelated word, with no unfortunate or fortunate connotations attached. We ended up using this word for a while for disambiguation.

These are all the metaphors that I've seen used to describe mobbing. Stay tuned for the next part of this post, Collective Code Ownership.

Image by Princess Ruto.

How is software like cooking?

Time for a light-hearted post. After my move to the UK and having had my share of fish and chips, I have become by reaction more interested in Italian culinary history and practice. So I started diving into the science and the tradition of cooking, reading books such as the science of meat which combine chemistry and good taste, and I have now cooked enough lasagne to build a statistically significant sample.

Disclaimer: this post is full of meat references as that's culturally significant as a metaphor to transmit the concepts I have in mind. You may find this distasteful if you have chosen to follow a different path.

So here's 5 ways in which software development and cooking are alike...

Feedback loops

"to serve man"

There's a joke in a Futurama episode about Bender (being a robot) not having a sense of taste and hence playfully disgusting the humans in the team with too much salt. The joke works because in cooking you need a continuous feedback loop to conform to your taste, for example adding salt and pepper at the end of a preparation until it tastes right.

We are no strangers to this process in software development: most of the practices I preach about lead to getting working software in front of someone that will use it as soon as possible, to better steer future development with the feedback.

There are shorter, inner feedback loops than tasting: the speed with which meat is browning will make you adjust your source of heat for that phase of the preparation to avoid charring the exterior surface to a pitch black color. Not too different from your unit tests failing and informing you of an issue well before it gets to an actual customer that will send the steak back to the kitchen.

Quality is in the eye of the beholder

Cacio e pepe: simple&tasty or poor?

Taste has lots of different components, including not just what your tongue perceives but also smells, presentation, and expectations. But for most of these aspects, quality is in the eye of the beholder and we can't avoid coming to grips with the variety of people and cultures.

Therefore, despite how good you think your burgers are, some people just don't like fat, mince meat. I appreciate Indian curries but I have some physical limits on the spiciness levels that make me almost always choose mild korma. And imagine the cultural shock of discovering I have been wrongly putting lemon in tea all my life instead of milk as the only acceptable choice.
(I know, tea doesn't even grow in Europe, but I grew up with British tea as the standard.)

We all have good intentions in thinking hard about what a user will enjoy or be productive with; but we have to recognize there is a vast variety of users and we have to design (or cook) for each of them.

Control the process, rather than micromanaging the material

This cake was fluorescent on purpose

Convection ovens are a great example of a controlled process for cooking uniformly. In the context of large pieces of meat or fish, this mainly means getting them to a uniform, high-but-not-too-high temperature to avoid overcooking. The oven fan pushes hot air all around them, heating the surface evenly. Air transfers less heat than, for example, water; so there is time for the temperature to rise across your roasting chicken rather than overcooking the outside while leaving some parts dangerously raw.

Thus for large cuts this is pretty much impossible to achieve in a pan, unless you literally cut everything into slices thin enough that they can cook quickly. The oven-based process is much more convenient as you literally abandon your tray in there, checking from time to time if it's ready with a thermometer.

Generally speaking of enterprise applications and websites, I favor a process in which we catch bugs with multiple safety nets (up to user experimentation if possible) than overdesigning for every possible problem. While you can think of possible scenarios to test endlessly, bugs are always going to happen and it's more important to have a process in place for which they will be fixed and never regress because of new automated tests. That makes your software converge to a steady, stable state like a perfectly cooked chicken.

You need to measure

That's definitely cold, not just for meat

If you want to consistently cook meat to your liking, there is no escape from using a thermometer to understand when it's ready - its center reaching a set temperature that corresponds to medium-rare (for a steak) or well-cooked (for poultry) or something else. Looking at the external color? No relation with the inside. Checking how hard it has become? Too subjective. Roast for a certain amount of time? Ignores the variability of both the ingredients and the heat source.

When we perceive part of an application as slow, we need to use a profiler to find out what functions or methods are taking the most time to execute. Making as few assumptions as possible, we collect data to point us in the right direction. Opinions don't count: your browser timings and other metrics do (if collected correctly).

You can substitute ingredients, to a certain extent

Focaccia genovese

Cornstarch and flour are used in small quantities in many recipes, with the goal of thickening a liquid. This is due to their starch content, as this carbohydrate granules swell up with water creating enough friction to transform a liquid with the viscosity of water into something that feels like cream.

If you try to use cornstarch to make bread however, you won't be able to get an elastic product as it lacks the proteins that would build gluten. Even if you use the wrong flour (cake flour as opposed to bread flour, to keep it simple), this will greatly affect the result due to the smaller percentage of proteins that it contains. Baking, both for sweet and savoury goods, require much more precision.

In software development, we have grown up with Lego bricks as a metaphor and we continuously try to swap out pieces, hiding details behind a useful abstraction that sometimes leaks. Nowadays relational databases can be queried interchangeably if you stick to standard SQL queries. But the data types for columns can be pretty different in the range they support, especially if they are somewhat more exotic like JSON and XML fields rather than integer and strings. A wise decision is still required to understand when substituting components is possible, or where some combinations will never work.

And here are 5 ways in which software development and cooking are very different...

Cooking is a repeatable process

Lots of Dutch cheese around Amsterdam

Recipes (at least the good ones) are literally the codification of a process that should be robust to external variations to get a consistent result. It's the mark of a good cook to be able to deal with variations in ingredients or tools, but unless you are up on a mountain water boils at pretty much the same temperature, and the physical transformation that your carrots undertake when they are heated is well established.

In software, every new feature is a new design to make rather than the execution of a plan. Even porting software or reimplementing it bears surprises as the platform it is running on is now different. And no one understands how long it took to produce the original version, no matter give an estimation for the new one that is being created. We have processes for understanding what a feature should do, and safely implementing it and rolling it out; but there are always land mines waiting on the path.

Cooking has some precise physical quantities you can rely on

Peppered pork fillet steak

Understood: measurement is needed in both fields. But as much as your oven oscillates around its target temperature, it is still much more precise than a developer's effort. Even without meetings and other time variables, how fast and precise we are in a certain day varies: humans aren't robots. Just how knowledgeable we are about a technology influences greatly the designing and testing phases. The Mythical Man-Month remains, well, mythical.

In the food industry, the right tools can even measure the strength of a flour, to check whether it's good for the bread you want to obtain. If you look at a technology team, measuring how many tasks per week we have completed is probably as good as it gets. There's humans involved and applying social science to a very small group probably doesn't get you very far in terms of collecting data and drawing inferences.

You can still measure other times objectively, like time to deploy: how long it takes for a commit on master to reach the production environment. We partly do this because it's important but also because it's feasible to measure. What most project managers would care about will be time from idea to complete implementation instead. But that requires estimating the length of a queue that changes all the time, and is just the first step of a creative development process with its own variations.

Determinism of the digital world

Blackberries from the garden

It's pretty difficult to get the same tomatoes, courgettes or grapes as last week, and pretty much impossible to get the same ones in-season and off-season. You can ship them in from South Africa or Australia but travel time and refrigeration can modify their contents, and thus their taste.

If you look at a physical server, it's much more similar to laboratory equipment than to a living product: you can run programs and see them always taking a similar amount of time to complete, controlling the randomness of the operating system around it. This gets eroded a bit in the cloud, where performance may be affected by your neighbors due to co-tenancy.

Timing in the kitchen

Very unstable crochembouche

Whether it is simply changing the temperature of a meat joint, or a more complex transformation like baking a cake, timing should be one of the concerns if you want to obtain a good result. I formalize this concept by thinking that it's not possible to stop time, in many cases.

Cooks know tricks like cooking eggs or rice to a certain degree, than cooling it down and finish the process later when the food has to be served; or simply reheat it if fully cooked. This works for various categories of products, but it's an ad-hoc process.

Consider the power we have in a digital world: firing up a debugger literally stops execution at some point in the life of the program, allowing us to take a look at what we want in the right context. Since the state of the program is the Matrix, we can slow it down, speed it up, and change things causing a déjà vu to your objects.

If you want to reproduce some computation, you have the tools available to build a Docker image containing all sorts of dependencies and store it for future usage. If you want to reproduce your perfect croissants, the only tools you have are a recipe and your own memories. Add the variation of ingredients and even temperature and humidity in your kitchen, and you can understand why scientific exploration needs a laboratory with its controlled conditions to be able to make progress.

Cooking equipment makes a difference

Now, grate parmesan without this...

Besides basic tools like appropriately shaped knives, a pressure cooker would make you able to reach certain results that would take a long time with an ordinary pot of boiling water. A temperature bath (I don't own one of these) can help cooking meat evenly only to then finish the process with a 2-minute searing. Even a scale is just necessary for baking, as measuring ingredients like flour by volume has a 50% margin of error due to its compressibility.

Consider how you can write code on your old laptop from the beach instead. You target an open source interpreter, and the end product will run on the same server that could accept strictly regulated banking software. As long as you can literally string bytes together, you can produce running software: everything else helps. The ephemeralization of software tools due to virtualization and the large availability of open source platforms make digital startups a reality, whereas opening a restaurant remains a capital-intensive operation.

But there's more...

The power of metaphors

Metaphors can foster understanding of a new system, or lead us ashtray. They are powerfully transmitting a mental model, but that model has its limitations and may even be less precise than a more formal model like a math analogy. But especially in complicated fields like cryptography, terms such as key and signature have popularized concepts to generation of students that would have otherwise found them very hard to think about.

I wrote this post for fun, but I stand behind most of the comparisons: that's all for now. You'll find me using an Helm to ship my containers...