I just want to run a container...

 ...is a developer-centric point of view?

Generic data center image to exemplify a place many of us have never set foot into.

After a few hours spent on upgrading the toolchain that starts from Terraform, goes through a few AWS-maintained modules and reached the Elastic Kubernetes Service APIs, my team was entertaining the thought of how difficult it is to set up infrastructure. Infrastructure that, for this use case takes a container image of a web application that Docker generated and run it somewhere for users to access.

Thinking through this, either Kubernetes is the new IBM (no one ever get fired for choosing it), or there is more to a production web application than running a container which is what Kubernetes is often sold as: a tool to run containers in the cloud without the necessity to set up specific virtual machines, but treating them as anonymous, sacrificeable nodes where the aforementioned containers can be tightly packed and sharing CPU, memory and other resources.

What exactly is the infrastructure doing for us here? There are a few concerns about operations, the other side of that magic DevOps collaboration. For example:

• the container image is exposing an HTTP service on port 80. This is not useful for a modern browser and its padlock icons, so to achieve a secure connection a combination of DNS and Let's Encrypt generated and automatically renews certificates after verifying proof of ownership of our domain name.
• the container image produces logs in the form of JSON lines. Through part of the Grafana stack, these lines are annotated with a timestamp, the container that generated them and various labels such as an environment or the name of a component of the application (think frontend or prod or staging). After these lines are indexed, further software provides the capability to query these logs, zooming in on problems or prioritizing errors during investigation.
• if we get too much of these logs at the level of error for a particular message, we'd like to receive an alert email that triggers us into action.
  
• invariably applications benefit from scheduled processes that run periodically and independently from the request/response lifecycle. It's such a useful architectural pattern that Kubernetes even named a resource after the cron daemon, introduced in 1975.
• it's also very useful for applications to maintain state and store news rows in a relational database. Provided turnkey by every cloud, a set of hostname, username and password allows access from any combination of programming language and library. No need to worry about rotating the logs of Postgres anymore, or that we are running out of space.
• the data contained in the application and generated by user also lends itself to timely analysis, so that we know whether to kill a feature or to invest in it. This means somehow taking out recent updates (or whole tables) into a data pipeline that transforms it into something that can be analyzed by a data scientist. All with an acceptable speed.
• Those credentials nevertheless need to be provided to the application itself, hopefully without accidentally disclosing them into chats, logs, or emails. No human should need to enter them into an interface.
• Configuration is slightly easier to manage than credentials due to the lack of sensitive values, but we'd still like the capability to generate some of these values such as a canonical URL or to pass in different numbers for different environments.
  
• And of course, whenever we commit, we'd like to deploy the change and start the latest version of our application, check that it is responding correctly to requests, and then stop the old, still running one.

This is all in the context of a single team, running operating a single product. Part of this is achieved through running off-the-shelf software on top of Kubernetes; part of it by paid-for cloud services; part of it by outsourcing to a specialized software as a service.

However, when we think about software development we don't necessarily think about software operations. Many of us started our careers by writing code, testing and releasing it without a further look to what was happening afterwards. DevOps as a philosophy meant bridging the gaps; You Build It You Run It is the easiest summary for me.

Yet what I see and what I hear from experienced people is that silos in infrastructure seriously slow teams down, and create fears of insurmountable problems. Abandoned projects, "finished" from a development point of view, but now having unclear ownership and running in production supporting the main revenue stream of an organization.

So I don't want to just run a container. I'd rather deploy many incrementally improved versions of a container, monitor its traffic and user activity, and close the feedback loop that links usage to the next development decision.

In other words: I can always build simple code if it doesn't have to run in production.

Pipeline Conf 2017

Not liquorice

Post originally shared on eLife's internal blog, but in the spirit of (green) open access here it is.

Last month I have attended the PIPELINE conference in London, 2017 edition. This event is a not-for-profit day dedicated to Continuous Delivery, the ability to get software changes into the hands of users, and to do so safely, quickly, and in a sustainable way. It is run by practitioners for practitioners, everyone on different sides of the spectrum like development, operations, testing, project management, or coaching.

The day is run with parallel tracks, divided into time slots of 40-minute talks and breaks for discussions and, of course, some sponsor pitches. I have been picking talks from the various tracks depending on their utility to eLife's testing and deployment platform, since our tech team has been developing every new project with this approach for a good part of 2016.

The conceptual model of Continuous Delivery and of eLife's implementation of it is not dissimilar to the scientific publishing process:

• there is some work performed into an isolated environment, such as a laboratory, but also someone's laptop;
• which leads to a transferable piece of knowledge, such as a manuscript, but also a series of commits, roughly speaking some lines of code;
• which is then submitted and peer reviewed. We do so through pull requests, which perform a series of automated tests to aid human reviewers inside the team; part of the review is also running the code to reproduce the same results on a machine which is not that original laptop.
• after zero or more rounds of revisions, this work gets accepted and published...
• which means integrating it with the rest of human knowledge, typesetting it, organizing citations and lots of metadata about the newly published paper. In software, the code has to be transformed into an efficient representation, or virtual machines have to be configured to work with it.
• until, finally, this new knowledge (or feature) is in the hands of a real person, who can read a paper or enjoy the new search functionalities

Forgive me for the raw description of scientific work.

In software, Continuous Delivery tries to automate and simplify this process to be able to perform it on microchanges multiple times per day. It aims for speed to be able to bring a new feature live in tens of minutes; it aims for safety to avoid breaking the users work on new changes; and to do all of this in a sustainable way, not to sacrifice tomorrow's ability to evolve for a quick gain today.

Even without the last mile of real user traffic, the 2.0 software services have been running in production or production-like servers from the first weeks of their development. A common anti-pattern in software development is to say "It works on my machine" (imagine some saying "It reproduces the results, but only with my microscope"); what we strive for is "It works on multiple machines, that can be reliably created; if we break a feature we know within minutes and can go back the latest version known to work."

Dan North: opening keynote

Dan North started to experiment with Continuous Delivery in 2004, at a time when builds were taking 2 days and a half to run in a testing environment contended by multiple teams. He spoke about several concepts underpinning Continuous Delivery:

• conceptual consistency: the ability of different people to make similar decisions without coordination. It's an holy grail for scaling the efforts of an organization to more and more members and teams.
• supportability: championing Mean Time To Repair over Mean Time Between Failures. The three important questions for facing a problem as what happened? Who is impacted? How do we fix it?
• operability: what does it feel like to build your software? To deploy it? Test it? Releasing it? Monitor it? Support it? Essentially, developer experience in additio to user experience.

Operability is a challenge we have to face ourselves more and more as we move from running our own platform to provide open source software for other people to use. Not only reading an article has to be a beautiful experience, but publishing one should be.

John Clapham: team design for Continuous Delivery

This talk was more people-oriented, I agree with the speaker that engagement of workers is what really drive profits (or value in case of non-profits).
Practically speaking:

• reward the right behaviors to promote the process you want;
• ignore your job title as everyone's job is to deliver value together;
• think small: it's easier to do 100 things 1% better than to do 1 thing 100% better (aka aggregation of marginal gains)

Abraham Marin: architectural patterns for a more efficient pipeline

The target for a build is for it to take less than 10 minutes. The speaker promotes the fastest builds as the one you don't have to run, introducing a series of patterns (and the related architectural refactorings) to be executed, safely, to simplify your software components:

• decoupling an api from implementation: extracting an interface package to reduce the dependencies to a component to a dependency to an interface;
• dividing responsibiliteis vertically or horizontally trying to isolate the most frequent changes and minimizing cross-cutting requirements;
• transform a library into a service;
• transform configuration into a service.

Some of these lessons are somewhat oriented to compiled languages, but not limited to them. My feeling is that even if you reduce compile times, you still have to test some components in integration, which is a large source of delay.

Steve Smith: measuring Continuous Delivery

How do you know whether a Continuous Delivery effort is going well? Or more pragmatically,  which of your projects is in trouble?
The abstract parameters to measure in pipelines are speed (throughput, cycle time) and stability. Each declines differently depending on the context.
In deployment pipelines that go from a commit to a new version release in production, lead time and the interval of new deployments can be measured. But also failure rate (how many runs fail) and failure recovery time are interesting. In more general builds or test suites, execution time is a key parameter but a more holistic view includes interval (how frequent are builds executed).
I liked some of these metrics so much that they are now in my OKRs for the new quarter. Simplistic quote: you can't manage what you can't measure.

Alastair Smith: Test-driving your database

To continuously deploy new software versions, you need an iterative approach to evolve your database and the data within it. When you evolve, you also have to test every new schema change. Even in the context of stored procedures for maximum efficiency (and lock-in), Alastair showed how to write tests that can reliably run on multiple environments.

Rachel Laycock: closing keynote, Continuous Delivery at Scale

Rachel Laycock is the Head of technology for North America at Thoughtworks, the main sponsor of the conference. The keynote however had nothing to do with sales pitches. Here are some anti-patterns:

• "We have a DevOps team" is an oxymoron, as that kind of team doesn't exist; what often happens is that the Ops team gets renamed."
• Do we chose Kubernetes or Mesos?" as in getting excited about the technology before you understand the problem to solve.

The "at scale" in the title pushes for seeing automation as a way to build a self-serving platform, where infrastructure people are not bottlenecks but enablers for the developers to build their own services.
The best quote however really was "yesterday's best practice becomes tomorrow's anti-pattern". What we look for is not to be the first to market but to have an adaptable advantage, a product that can evolve to meet new demands rather than being a dead end.

Book review: Site Reliability Engineering

The overwhelming majority of a software system’s lifespan is spent in use, not in design or implementation. So, why does conventional wisdom insist that software engineers focus primarily on the design and development of large-scale computing systems? -- the book tagline

Site Reliability Engineering - How Google runs production systems is a 2016 book about the ops side of Google services, and the set of principles and practices that underlie it.

A Site Reliability Engineer is a software engineer (in the developer sense) that designs and implement systems that automated what would otherwise be done manually by system administrators. As such, SREs have a directive for employing a minimum 50% of their time in development rather than firefighting and maintenance of existing servers (named toil in the book).

The book really is a collection of chapters, so you don't have to be scared by its size as you don't necessarily need to read it cover to cover. You can instead zoom in on the interesting chapters, being them monitoring, alerting, outage tracking or even management practices.

Principles, not just tools

I like books that reify and give names to concepts and principles, so that we can talk about those concepts and refer to them. This book gives precise definitions for the Google version of measures such as availability, Service Level Objectives and error budgets.
This is abstraction at work: even if the examples being used show Google specific tools like Borgmon or Outalator, the solutions are described at an higher abstraction level that makes them reusable. When load balancing practices are made generic enough to satisfy all of Google services, you can bet that they are reusable enough to be somewhat applicable to your situation.

Caveat emptor

Chances are that you're not Google: the size of your software development effort is order of magnitudes smaller, and even when it is of a comparable size it's not easy to turn an established organization into Google (and probably not desirable nor necessary.)
However, you can understand that Google and the other giants like Facebook and Amazon are shaping our industry through their economically irresistible software and services. Angular vs React is actually Google vs Facebook; containers vs serverless is actually Kubernetes vs Lambda which is actually Google vs Amazon. The NoSQL revolution was probably started by the BigTable and Dynamo papers... and so on: when you deployed your first Docker container, Google engineers had already been using similar technology for 10 years; as such, they can teach you a lot at a relatively little cost through the pages of a book. And it's better to be informed on what may come to a cloud provider near you in the next years.

Conclusions

It took some time to get through this book, but it gives a realistic picture of running systems that undergo a large scale of traffic and changes at the same time. Besides the lessons you can directly get from it, I would recommend it to many system administrators and "devops guys" as a way to think more clearly about which forces and solutions are at play in their datacenters (or more likely, virtual private clouds).