tag:blogger.com,1999:blog-365471682024-03-05T08:46:03.696+01:00Invisible to the eyeA journey in web development, [computer] science, engineering:<br>
getting to know what lies under the hood <em>-- Giorgio Sironi</em>Giorgiohttp://www.blogger.com/profile/03558287012747987157noreply@blogger.comBlogger415125tag:blogger.com,1999:blog-36547168.post-77935688721639957862023-05-11T20:37:00.002+02:002023-05-11T20:37:25.860+02:00I just want to run a container...<p> ...is a developer-centric point of view?</p><p><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipDsBf2pmANoen3jtGf8iHUAh8V98WMHJH-ZfeqyVBBaS-PYdN6gZcj5uuvArjEhHAZs11cUbXeWpatkgYc05kFM2lb_S_TFwi416C9idaPKWLkjtNDkYxFmw21E6Wvgiu9FTDtv2nkjRTQIFHbH89LO1_4oGbAe53C5INjYq9pTh_XePiovc/s4408/imgix-klWUhr-wPJ8-unsplash.jpg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="2628" data-original-width="4408" height="191" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipDsBf2pmANoen3jtGf8iHUAh8V98WMHJH-ZfeqyVBBaS-PYdN6gZcj5uuvArjEhHAZs11cUbXeWpatkgYc05kFM2lb_S_TFwi416C9idaPKWLkjtNDkYxFmw21E6Wvgiu9FTDtv2nkjRTQIFHbH89LO1_4oGbAe53C5INjYq9pTh_XePiovc/s320/imgix-klWUhr-wPJ8-unsplash.jpg" width="320" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Generic data center image to exemplify a place many of us have never set foot into.<br /></td></tr></tbody></table></p><p>After a few hours spent on upgrading the toolchain that starts from Terraform, goes through a few AWS-maintained modules and reached the Elastic Kubernetes Service APIs, my team was entertaining the thought of how difficult it is to set up infrastructure. Infrastructure that, for this use case takes a container image of a web application that Docker generated and run it somewhere for users to access.</p><p>Thinking through this, either Kubernetes is the new IBM (no one ever get fired for choosing it), or there is more to a production web application than running a container which is what Kubernetes is often sold as: a tool to run containers in the cloud without the necessity to set up specific virtual machines, but treating them as anonymous, sacrificeable nodes where the aforementioned containers can be tightly packed and sharing CPU, memory and other resources.</p><p>What exactly is the infrastructure doing for us here? There are a few concerns about operations, the other side of that magic DevOps collaboration. For example:</p><ul style="text-align: left;"><li>the container image is exposing an HTTP service on port 80. This is not useful for a modern browser and its padlock icons, so to achieve a <b>secure connection</b> a combination of DNS and Let's Encrypt generated and automatically renews certificates after verifying proof of ownership of our domain name.</li><li>the container image produces <b>logs</b> in the form of JSON lines. Through part of the Grafana stack, these lines are annotated with a timestamp, the container that generated them and various labels such as an environment or the name of a component of the application (think <i>frontend</i> or <i>prod</i> or <i>staging</i>). After these lines are indexed, further software provides the capability to query these logs, zooming in on problems or prioritizing errors during investigation.</li><li>if we get too much of these logs at the level of error for a particular message, we'd like to receive an <b>alert</b> email that triggers us into action. <br /></li><li>invariably applications benefit from <b>scheduled processes</b> that run periodically and independently from the request/response lifecycle. It's such a useful architectural pattern that Kubernetes even <a href="https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/">named a resource</a> after the <i>cron</i> daemon, introduced in 1975.</li><li>it's also very useful for applications to maintain state and store news rows in a relational <b>database</b>. Provided turnkey by every cloud, a set of hostname, username and password allows access from any combination of programming language and library. No need to worry about rotating the logs of Postgres anymore, or that we are running out of space.</li><li>the data contained in the application and generated by user also lends itself to timely analysis, so that we know whether to kill a feature or to invest in it. This means somehow taking out recent updates (or whole tables) into a <b>data pipeline</b> that transforms it into something that can be analyzed by a data scientist. All with an acceptable speed.</li><li>Those <b>credentials</b> nevertheless need to be provided to the application itself, hopefully without accidentally disclosing them into chats, logs, or emails. No human should need to enter them into an interface.</li><li><b>Configuration</b> is slightly easier to manage than credentials due to the lack of sensitive values, but we'd still like the capability to generate some of these values such as a canonical URL or to pass in different numbers for different environments.<br /></li><li>And of course, whenever we commit, we'd like to <b>deploy</b> the change and start the latest version of our application, check that it is responding correctly to requests, and then stop the old, still running one.</li></ul><p>This is all in the context of a single team, <strike>running</strike> operating a single product. Part of this is achieved through running off-the-shelf software on top of Kubernetes; part of it by paid-for cloud services; part of it by outsourcing to a specialized software as a service.</p><p>However, when we think about software development we don't necessarily think about software operations. Many of us started our careers by writing code, testing and releasing it without a further look to what was happening afterwards. DevOps as a philosophy meant bridging the gaps; <a href="https://www.equalexperts.com/blog/our-thinking/what-is-you-build-it-you-run-it/">You Build It You Run It</a> is the easiest summary for me.<br /></p><p>Yet what I see and what I hear from experienced people is that silos in infrastructure seriously slow teams down, and create fears of insurmountable problems. Abandoned projects, "finished" from a development point of view, but now having unclear ownership and running in production supporting the main revenue stream of an organization.<br /></p><p>So I don't want to just run a container. I'd rather deploy many incrementally improved versions of a container, monitor its traffic and user activity, and close the feedback loop that links usage to the next development decision. <br /></p><p>In <a href="https://wiki.c2.com/?QuotesOnProgramDevelopment">other words</a>: I can always build simple code if it doesn't have to run in production.<br /></p>Giorgiohttp://www.blogger.com/profile/12689416577856305650noreply@blogger.com0tag:blogger.com,1999:blog-36547168.post-26267320510178891882022-01-20T19:22:00.001+01:002022-01-20T19:22:30.613+01:00A year of mob programming, part 5: methodology<p>I wouldn't call mob programming a methodology, but rather a practice that can be adopted. Like methodologies, however, it only surfaces problems and questions faster <a href="https://cacm.acm.org/magazines/2000/10/7556-the-five-orders-of-ignorance/fulltext">rather than providing solutions</a>.<br /><br />For example, a team might have a low bus factor due to the specialisms of its members, and only one person might be used to edit code in a certain language or application. Other team members might have gaps, such as not being set up with the tools and credentials to work on infrastructure and operations; or lacking the knowledge to perform screen reader testing for accessibility.</p><p>Besides exposing bus factors and gaps, mob programming brings design conflicts and opinions into the open. It takes little effort to turn the other way when looking at a pull request, or to switch to busy work or generic "<a href="https://www.waterfall2006.com/Refuctoring.pdf">refuctoring</a>" when alone; disagreements and questions about value can often prop up under your nose inside a mob session.</p><p>However, that doesn't mean that the practice automatically gives the team the psychological safety to talk about those issues out loud, and to resolve them. That's like asking an hammer to walk to the shop and buy the right nails.</p><p>At the last part of this post, I thought I'd compare my mob programming experience with some of the theory; the theory being famous Agile software development methodologies and their values or principles.</p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgtgvsRplVn7qY8PRBq6T5UrZNZDKivGo3lbccglCufS4VwlG5Kw5u7S9gKC-LDtUpWb8JFuYscgsxP0eLdWnpUoV-7iObMwlU1eOVCHBLXlBjHmcy8rAdZBT7ZFoMHgONg4Kth6pc6RpNGKyxP2hXZHlR5DOdUHehNYRLNX49F9my9xUafCD4=s3888" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="2592" data-original-width="3888" height="213" src="https://blogger.googleusercontent.com/img/a/AVvXsEgtgvsRplVn7qY8PRBq6T5UrZNZDKivGo3lbccglCufS4VwlG5Kw5u7S9gKC-LDtUpWb8JFuYscgsxP0eLdWnpUoV-7iObMwlU1eOVCHBLXlBjHmcy8rAdZBT7ZFoMHgONg4Kth6pc6RpNGKyxP2hXZHlR5DOdUHehNYRLNX49F9my9xUafCD4=s320" width="320" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">The serene predictability of waterfall software development<br /></td></tr></tbody></table><h4 style="text-align: left;">Within <a href="http://www.extremeprogramming.org/values.html">Extreme Programming values</a>:</h4><ul style="text-align: left;"><li><b>Communication</b> inside a mob is very high: synchronous conversations supported by a visual medium such as code or a digital whiteboard.</li><li><b>Simplicity</b> is fostered by people in a diverse team reminding others that You Aren't Gonna Need It, but also needing to understand what's going on at all times.</li><li><b>Feedback</b> from the rest of the team, or even from a customer proxy, is immediate.</li><li><b>Courage</b> is needed to be a driver in an unfamiliar setting, or even a navigator asking the mob for help on a task we are uncomfortable with.</li><li><b>Respect</b> is necessary to work face to face for prolonged periods of time.<br /></li></ul><h4 style="text-align: left;">Within <a href="https://en.wikipedia.org/wiki/Lean_software_development">Lean</a> values, mob programming:</h4><ul style="text-align: left;"><li><b>eliminates waste</b> such as <b>partially done work</b>: there should ideally be no pull request open or branch, as code is produced and integrated in the mob continuously;</li><li>eliminates waste such as <b>outdated features</b> or gold plating: when you include a product owner in the mob, it's easy to steer and change the scope upon feedback rather than plowing on with the original idea;</li><li>eliminates waste such as <b>handoffs</b>: there is no packaging up of code for review, or of pixel-perfect Photoshop design to front end developers, or front end requests to back ends to be implemented, and so on.</li><li><b>reduces task switching</b>: since everyone that is needed for progress is present, you often don't need to switch task because of waiting on someone, but can just carry on.</li><li><b>amplifies learning</b>: ideas are tried out in code from the get go, and most of the team can learn from the results including the product owner.</li><li><b>optimizes the whole</b>: most of all, we are not optimizing how much we can use everyone's time, but how fast we can flow a single piece from idea to customer value.</li></ul><p>Image by <a href="https://commons.wikimedia.org/wiki/File:Waterfall_on_Johilla_river,_Umaria_dist,_Madhya_Pradesh,_India.jpg">Yann</a>. <br /></p>Giorgiohttp://www.blogger.com/profile/12689416577856305650noreply@blogger.com0tag:blogger.com,1999:blog-36547168.post-46057652413686305592022-01-08T11:29:00.002+01:002022-01-20T19:23:18.008+01:00A year of mob programming, part 4: the remoteness of it<p>I don't have a frame of reference for mob programming in a co-located environment, besides conference (or team exercise) workshops that didn't involve production code.</p><p>On a video call, the <i>driver</i> is still always explicitly defined as the person sharing their screen and tools. This is similar to the person physically having the keyboard when you are all sitting around a table.</p><p>The <i>navigator</i> can either be more implicit, move around or be strictly defined by a process. If an implicit navigator doesn't emerge for the current commit, it should be considered normal to quickly nominate one as needed. Over time, we normalize the phrase <i>"can you navigate [that change]?"</i> in response to a proposal.<br /></p><p>Due to latency, in a video call environment there is a sort of <a href="https://en.wikipedia.org/wiki/Relativity_of_simultaneity">Lorentz time</a> where you may hear overlapping voices when someone else doesn't, and vice versa. It all depends on how much latency each video link is experiencing, and the adjustments that software makes can be jarring, such as video freezes or audio being delivered 10-20 seconds later.</p><p></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEge4W3NYVz2IunLKyZGh6i1Wan7SOUt79_9Cm-XLahRkfBoU-PYoHiDs3kjvw2NRCTaoDX2itfBEICtTQWEph9kQbkgN_proFo8z0CQOBhFy1Rv5OrRewxo6f_v9i4c2Aug7UFaix96ZNTW0SctSMOm5Y-ryQnIigGIx8Gfrynbc2AowRU1XLc=s744" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="438" data-original-width="744" height="188" src="https://blogger.googleusercontent.com/img/a/AVvXsEge4W3NYVz2IunLKyZGh6i1Wan7SOUt79_9Cm-XLahRkfBoU-PYoHiDs3kjvw2NRCTaoDX2itfBEICtTQWEph9kQbkgN_proFo8z0CQOBhFy1Rv5OrRewxo6f_v9i4c2Aug7UFaix96ZNTW0SctSMOm5Y-ryQnIigGIx8Gfrynbc2AowRU1XLc=s320" width="320" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">"Sorry, two people talking over each other" "Really??"<br /></td></tr></tbody></table><p></p><p>I'm sure though everyone has already experienced people talking over
each other in a physical room: it isn't exclusively a remote working
problem and relies on team interactions more than technology.</p><p>We also want to communicate and especially give feedback through voice, as people's faces are not always visible to a driver looking at code; nodding doesn't necessarily help.<br /></p><p>Perhaps a bigger problem is the fact that video calls <b>don't allow side conversations to happen</b>, as there is a single audio channel that our ears cannot separate into directions. This is likely to impact a massively parallel <a href="https://blog.avanscoperta.it/2020/03/26/eventstorming-in-covid-19-times/">EventStorming</a> around a whiteboard, less so an ensemble of 3-4 people trying to get to a consensus on their next move. The side conversations however include those famous water cooler talks, bumping into people in the kitchen or while on a break.<br /></p><p>Your theory of mind might have more trouble cutting through the limited signal, and understand in what mental state other people in the team are; and whether today's problem is more due to Internet connection trouble, or a real conflict between team members. The daily retrospective helps to let people put tools down, and reflect on the day; possibly proposing experiments for the next session. </p><p>Looking at the world around me, I'm quite sure I would be as depressed by remote work as many, if it consisted of me being alone for most of the day. So due to mob programming I have a different perspective on how incredibly engaging working from home can be.</p><p>Stay tuned for the next (and last) part of this post, <a href="https://www.giorgiosironi.com/2022/01/a-year-of-mob-programming-part-5.html"><i>Methodology</i></a>.</p><p>Image by <a href="https://commons.wikimedia.org/wiki/File:Lamport-Clock-en.svg">Duesentrieb</a>. <br /></p>Giorgiohttp://www.blogger.com/profile/12689416577856305650noreply@blogger.com0tag:blogger.com,1999:blog-36547168.post-5496518304825329252021-11-17T20:45:00.003+01:002022-01-08T11:29:51.898+01:00A year of mob programming, part 3: a laboratory for team dynamics<p>Our implementation of mob programming consists in a permanent Google Meet videocall, with camera on, in which we rotate a developer sharing a screen containing IDE and browser. So there are two unusual practices to get used to: being in a group all the time, and being on a camera all the time.<br /></p><p>I was originally surprised on how both practices went from exhausting to being a normal work day. This is my experience, and we have to consider neurodiversity and personal preferences in the team on having a long screen time and continuous social interaction. It's almost obvious to say, but sometimes people hate inconclusive meetings in which they have no say; not being together with other human beings.<br /></p><p>Don't underestimate your capacity to adapt, but recognize that your energy is not going to always be the same; whether because you got a cold, or something is going on at home, or some task is particularly draining. </p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2RO5Hyj6NgxPx4oleYdw1ORJu77SnkpZTGKXZJHisDfjwa_hHF6KKSRIm12kpv7PauY6qRnX-XFaD80zGfKxUQ60pkEvei_DxzfVWnwigygS79Ri8qQiOkp9tyHWbkwgYntVgIA/s2048/Pieter_Bruegel_d._%25C3%2584._035.jpg" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="2023" data-original-width="2048" height="316" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2RO5Hyj6NgxPx4oleYdw1ORJu77SnkpZTGKXZJHisDfjwa_hHF6KKSRIm12kpv7PauY6qRnX-XFaD80zGfKxUQ60pkEvei_DxzfVWnwigygS79Ri8qQiOkp9tyHWbkwgYntVgIA/s320/Pieter_Bruegel_d._%25C3%2584._035.jpg" width="320" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">The Flemish inscription at bottom reads: <i>because feedback is perfidious, I will go code in my cave</i><br /></td></tr></tbody></table><p><a href="https://www.researchgate.net/publication/2394870_Innovation_and_Sustainability_with_Gold_Cards">Gold cards</a> and <a href="http://www.extremeprogramming.org/rules/spike.html">spike branches</a> give people the ability to work on their own when they request to. Mechanical work such as upgrade of dependencies or investigating logs can benefit from focused time from one person. If you were in an office, you could go to an isolated room; working remotely, it's even easier as it just consists in leaving the video call and coming back refreshed later.<br /></p><p>The opposite can also happen, when someone like me can manifest Fear Of Missing Out and the mob making lots of decisions in their absence. As the team gets to a <a href="https://blog.trello.com/form-storm-norm-perform-stages-of-team-productivity">norming</a> phase, we should expect fewer surprises for members coming back to the mob after some time off. <br /></p><p>Being together with the rest of the team can make your motivation higher due to the group support in getting started or unstuck from a particular problem; but of course the whole group can struggle on occasion. <br /></p><p>Every time there is a new composition, though, there are changes in how the mob is working and what it pays more attention to. And from the point of view of a growing team leader, being embedded in a mob environment means being positively inundated with information. The challenge is to make sense of what we see to understand where team members are struggling and what are they finding most effective. </p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAAwVJB3_0PupBxMUjAwZATc_I8iTLGpe7zUA_qXEZdNNCITTn0N9gj8IGDTkVU9S5QXSxup0E5aIiN4T2PUtW4GCSpbN0v5SkDkPc9IxfpkMua2a6cfIb0RpCVDBVjyk_P_gfIQ/s2048/FIFA_WC-qualification_2014_-_Austria_vs_Ireland_2013-09-10_-_Giovanni_Trapattoni_04.JPG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1360" data-original-width="2048" height="213" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAAwVJB3_0PupBxMUjAwZATc_I8iTLGpe7zUA_qXEZdNNCITTn0N9gj8IGDTkVU9S5QXSxup0E5aIiN4T2PUtW4GCSpbN0v5SkDkPc9IxfpkMua2a6cfIb0RpCVDBVjyk_P_gfIQ/s320/FIFA_WC-qualification_2014_-_Austria_vs_Ireland_2013-09-10_-_Giovanni_Trapattoni_04.JPG" width="320" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i>"I wish I could see players only in 121 meetings"</i> -- No football coach ever<br /></td></tr></tbody></table><p>This requires a lot of (cognitive) bandwidth, and I usually can't at the same time focus on technical architecture and on social roles. But maybe this is just an argument to work in a team where the different hats can be rotated; as much as there is always someone writing code, there can always be someone checking if we have started to talk over each other, or we are falling into a rabbit hole of an hour without committing.<br /></p><p>Stay tuned for the next part of this post, <a href="https://www.giorgiosironi.com/2022/01/a-year-of-mob-programming-part-4.html">The remoteness of it</a>.</p><p><i>Images by </i><i><a href="https://commons.wikimedia.org/wiki/File:FIFA_WC-qualification_2014_-_Austria_vs_Ireland_2013-09-10_-_Giovanni_Trapattoni_04.JPG">Michael Kranewitter</a> and in the <a href="https://commons.wikimedia.org/wiki/File:Pieter_Bruegel_d._%C3%84._035.jpg">public domain</a>.</i><br /></p>Giorgiohttp://www.blogger.com/profile/12689416577856305650noreply@blogger.com0tag:blogger.com,1999:blog-36547168.post-80778044232743354022021-11-09T17:27:00.002+01:002021-11-17T20:46:29.437+01:00A year of mob programming, part 2: Collective Code Ownership<p>With respect to a team assigning tasks to developers by their function (frontend, backend, infrastructure, and so on), mob programming fosters <a href="http://www.extremeprogramming.org/rules/collective.html">Collective Code Ownership</a>.</p><p>This is more in the sense that everyone should be able to contribute through the mob in any area, rather than everyone being able to prioritize on their own what needs to be improved. The pact is that our time belongs to the team, and the team decides what is important.<br /></p><p>A result of this is an <b>healthy prevention of knowledge silos</b>, where you can exercise your own creativity just because no one else knows how to work in that area anymore.<br /></p><p>Code review happens continuously in the mob on small changes, and there are no huge feature branches to go through. In large changes such as those, it's too late for review to suggest a valuable but completely different approach; there is too much investment of time and energy into code. If review dares to suggest groundbreaking changes, it <a href="https://chelseatroy.com/2019/12/18/reviewing-pull-requests/">causes extensive rework</a> instead.<br /></p><p>There is a larger picture on ownership: team members tend to keep track of what they care about, whether that is consistency in architecture, front end approaches to styling, performance issues, and so on. <br /></p><p>Sometimes I found myself being talked out of a design I had in mind, counteracting bias I could have for the first or most familiar solution. I started working on this team with an object-oriented mindset, but we have transitioned to functional programming in TypeScript. <br /></p><p>Sometimes, there are hills to die on: hard to reverse decisions on architecture, or programming language choices. It's a job of a psychologically safe team to discuss these choices collectively without drama or oversimplification like JavaScript being the the death of computer science. </p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj26_c66aohnZp2kMSvmOpdD7HO6DJhkXbRw5F48zuuAAeeGrDt0z_08MbncDknQCGRonB-73HmcED8ZTss1TVhpp2EtOikjdcMrBV9_DVLdCWUma6m-EUVB5lHiBCTLt69gSpoCQ/s1024/Maria_Saal_Zollfeld_Virunum_Arena_Podium_r%25C3%25B6mische_Bauinschrift_04102017_1336.jpg" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="696" data-original-width="1024" height="218" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj26_c66aohnZp2kMSvmOpdD7HO6DJhkXbRw5F48zuuAAeeGrDt0z_08MbncDknQCGRonB-73HmcED8ZTss1TVhpp2EtOikjdcMrBV9_DVLdCWUma6m-EUVB5lHiBCTLt69gSpoCQ/s320/Maria_Saal_Zollfeld_Virunum_Arena_Podium_r%25C3%25B6mische_Bauinschrift_04102017_1336.jpg" width="320" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i>"I told you we should have written the code in English, not Latin!"</i><br /></td></tr></tbody></table><p style="text-align: left;">It tooks three different iterations to get a repeatable pattern for accepting Commands (as in CQRS). But it was more fruitful to focus on the Domain Events design, as a difficult choice to revisit, than on the shape of the code itself which can be refined at any time.<br /></p><h3 style="text-align: left;">Working outside of the mob</h3><p style="text-align: left;">The counterpart to the XP mantra of writing all production code in pairs (or a mob) is to allow team members to contribute when they are working alone due to other necessities, such as an unstable Internet connection or a flexible day where they can't align completely with the core hours of the team. <br /></p><p style="text-align: left;">I have learned to ask the team to be commissioned something to do, or at least to give them a choice. This keeps the prioritization in the hands of the group, again reducing bias.<br /></p><p style="text-align: left;">It's helpful to assign problems that have only a constrained possible solution such as renaming, or propagating a refactoring through the codebase for consistency's sake. <br /></p><p style="text-align: left;">It also helps to report back what you have learned during a solo coding exercise when rejoining the mob; unexpected issues or decisions you had to take on the spot and you feel unsure about. <br /></p><p style="text-align: left;">You're allowed to stop at a certain point, as <b>making progress alone at all costs is valued less than maintaining collective code ownership</b> and a shared understanding of how our application works. <br /></p><p style="text-align: left;">Technical <a href="http://www.extremeprogramming.org/rules/spike.html">spikes</a> can also be chosen by a single person to work on, either because they want to individually prioritize them to demonstrate an idea; or because the amount of investigation required makes it difficult or frustrating to collaborate.<br />Spikes can be built on throwaway branches, being optimized for learning rather than for delivering production quality code. Once an idea has been demonstrated and approved, the mob can implement it pretty quickly on the trunk, and if we believe in the practice, with an higher level of quality. I constantly find feature branches worked by a single person to be a dead end (<span style="font-size: x-small;">especially if they are my own</span>). <br /></p><p style="text-align: left;">In the end, people are frequently in meetings, researching or even just on holiday for a given day. Hence there's always someone missing that will catch up on the progress when they come back into the mob. But the group itself never stops even as the components change, so some of that progress will happen every day, other things being equal.</p><p style="text-align: left;">Stay tuned for the next part of this post, <a href="https://www.giorgiosironi.com/2021/11/a-year-of-mob-programming-part-3.html">A laboratory for team dynamics</a>.<br /></p><p><i>Image by <a href="https://commons.wikimedia.org/wiki/File:Maria_Saal_Zollfeld_Virunum_Arena_Podium_r%C3%B6mische_Bauinschrift_04102017_1336.jpg">Johann Jaritz</a>.</i><br /></p>Giorgiohttp://www.blogger.com/profile/12689416577856305650noreply@blogger.com0tag:blogger.com,1999:blog-36547168.post-79887114467624393652021-10-20T22:18:00.001+02:002021-11-09T17:28:36.964+01:00A year of mob programming, part 1: metaphors<p>I've been practicing remote mob programming in <a href="https://sciety.org/">my team</a> for more than a year, writing more than 90% of production code inside a video call with multiple developers looking at the same screen.<br /></p><h2 style="text-align: left;">A bit of context</h2><p>I work in the scientific publishing domain, on web applications oriented to help scientists throughout their day.</p><p>My current team has been formed from the start as a remote team, and has been working with mob programming for more than a year. It is a cross-functional team that is including a product manager and often a designer into the (virtual room) for co-creation.</p><p>Our set of technologies includes TypeScript, lots of semantic HTML, and minimal client-side JavaScript. Container-based infrastructure and databases are pulled in as required by the evolution of the product.</p><p>This is an experience report, not advice that can be applied blindly to your team or organization.<br /></p><h2 style="text-align: left;">Metaphorical definitions</h2><p style="margin-left: 40px; text-align: left;"><i>All the brilliant people working on the same thing, at the same time, in the same space, and on the same computer </i>-- <a href="https://mobprogramming.org/">Woody Zuill</a> <br /></p><p>Like pair programming, mob programming can be simplistically explained through the metaphor of driving a car in a rally. Or at least, that's how I always understood the driver and navigator pair.<br /></p><ul style="text-align: left;"><li>A <b>driver</b> shares their screen and has sole access to the keyboard. They are an intelligent IDE capable of executing mechanical refactoring and fixes, but not of deciding a design direction for the code.</li><li>A <b>navigator</b> verbalizes what the driver should be doing, making decisions that reach down to what test or line of code to write.</li><li>The rest of the group is on the back seat of the car, and can speak at will, volunteering information to the navigator or answering questions. Are we there yet?<br /></li></ul><p style="text-align: left;">The roles of driver and navigator are rotated often, in fixed time or scope increments such as on every commit. The navigator can implicitly move continuously from one person to another, as long as there isn't more than one navigator at once.<br /></p><h2 style="text-align: left;"><i>Mob</i> as a name</h2><p style="text-align: left;">Speaking Italian as a first language, the term <i>mob</i> doesn't necessarily evoke an emotional reaction from me. But I can recognize a term that can be used to describe organized crime might not allow team members to feel comfortable. My understanding is that <i>mob</i> has to be intended as <i>crowd</i> rather than <i>mafia</i>.</p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjazmV7Gca81xs9ii4uIfoDrqMzupCk2Xl8WoJ9M4QP2PXB3yVPul7WoC7e6NkpNKvOGjzFSQJX-iqpnNxvRncyT7X1iOsFZiIaUM_DWiEMiIltb7dI8qeD0N1ZDGikqE_z0j82TQ/s2581/Arianna_String_Quartet_%252814193354501%2529.jpg" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1218" data-original-width="2581" height="151" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjazmV7Gca81xs9ii4uIfoDrqMzupCk2Xl8WoJ9M4QP2PXB3yVPul7WoC7e6NkpNKvOGjzFSQJX-iqpnNxvRncyT7X1iOsFZiIaUM_DWiEMiIltb7dI8qeD0N1ZDGikqE_z0j82TQ/s320/Arianna_String_Quartet_%252814193354501%2529.jpg" width="320" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i>"This is so wasteful. Why aren't they all in separate rooms playing 4 different songs?"</i><br /></td></tr></tbody></table><p style="text-align: left;">Nevertheless, we looked for more metaphors and terms that can be used to talk about the group of people that work together:<br /></p><ul style="text-align: left;"><li><b>orchestra</b> emphasizes the coordination required by a group of people, and their simultaneous presence. I'm not sure they need a conductor though.</li><li><b>ensemble</b> is a similar musical metaphor, that focuses on a small group of complementary roles closely working together.</li><li><b>swarm</b> refers to a group of bees all converging towards a single task to get it done.</li><li>the <b>team</b> itself ideally coincides with the mob, though there can be non-technical roles that find difficult to contribute all the time; the mob can also be temporarily reduced from the full set of developers in the team, or multiple mobs can appear in a relatively large team.</li><li>a <b>womble</b> is a <a href="https://en.wikipedia.org/wiki/The_Wombles">fictional furry creature from children's books</a> that helps the environment by cleaning up rubbish. It has no relation to groups whatsoever and is an unrelated word, with no unfortunate or fortunate connotations attached. We ended up using this word for a while for disambiguation.</li></ul><p>These are all the metaphors that I've seen used to describe mobbing. Stay tuned for the next part of this post, <a href="https://www.giorgiosironi.com/2021/11/a-year-of-mob-programming-part-2.html">Collective Code Ownership</a>.</p><p><i>Image by <a href="https://commons.wikimedia.org/wiki/File:Arianna_String_Quartet_(14193354501).jpg">Princess Ruto</a>.</i><br /></p>Giorgiohttp://www.blogger.com/profile/12689416577856305650noreply@blogger.com0tag:blogger.com,1999:blog-36547168.post-36543463508579623022019-02-03T18:26:00.003+01:002019-02-03T18:42:51.329+01:00How is software like cooking?Time for a light-hearted post. After my move to the UK and having had my share of fish and chips, I have become by reaction more interested in Italian culinary history and practice. So I started diving into the science and the tradition of cooking, reading books such as <a href="https://www.amazon.it/scienza-carne-chimica-bistecca-dellarrosto/dp/8858016025">the science of meat</a> which combine chemistry and good taste, and I have now cooked enough lasagne to build a statistically significant sample.<br />
<br />
<i>Disclaimer: this post is full of meat references as that's culturally significant as a metaphor to transmit the concepts I have in mind. You may find this distasteful if you have chosen to follow a different path.</i><br />
<br />
So here's 5 ways in which software development and cooking are alike...<br />
<h3>
Feedback loops</h3>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9Hk0OcqVyUh5jnBUXgsgxaNhc8fq3YASSdj8I0p6Hn9VGI71rp_PTJDiMUZ2ohzKYsPEERfobjjCqDf96f9-vjkggih2FVHX2Kjzo2rcg5IPLUXqdc9o7KNnHVAkQBjJcLMBIWw/s1600/bender.jpeg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="375" data-original-width="500" height="150" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9Hk0OcqVyUh5jnBUXgsgxaNhc8fq3YASSdj8I0p6Hn9VGI71rp_PTJDiMUZ2ohzKYsPEERfobjjCqDf96f9-vjkggih2FVHX2Kjzo2rcg5IPLUXqdc9o7KNnHVAkQBjJcLMBIWw/s200/bender.jpeg" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">"to serve man"</td></tr>
</tbody></table>
There's a joke in a Futurama episode about Bender (being a robot) not having a sense of taste and hence playfully disgusting the humans in the team with too much salt. The joke works because in cooking you need a continuous feedback loop to conform to your taste, for example adding salt and pepper at the end of a preparation until it tastes right.<br />
<br />
We are no strangers to this process in software development: most of the <a href="https://www.giorgiosironi.com/search/label/continuous%20delivery">practices I preach about</a> lead to getting working software in front of someone that will use it as soon as possible, to better steer future development with the feedback.<br />
<br />
There are shorter, inner feedback loops than tasting: the speed with which meat is browning will make you adjust your source of heat for that phase of the preparation to avoid charring the exterior surface to a pitch black color. Not too different from your unit tests failing and informing you of an issue well before it gets to an actual customer that will send the steak back to the kitchen.<br />
<h3>
Quality is in the eye of the beholder </h3>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUNrPCVkkU1_t1E4vv6zv84uEFsfNoyKgy8rZk9tDxjOYgWXUMm4EMfCz4zKnVFTpNTJUuR8r0nqzeTrlfKC8wSLVZK0asUZypHOYvnPtv7XI5YwK4zEvpEoSw-IMPOTZl2uQ90g/s1600/cacio_e_pepe.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="900" data-original-width="1600" height="112" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUNrPCVkkU1_t1E4vv6zv84uEFsfNoyKgy8rZk9tDxjOYgWXUMm4EMfCz4zKnVFTpNTJUuR8r0nqzeTrlfKC8wSLVZK0asUZypHOYvnPtv7XI5YwK4zEvpEoSw-IMPOTZl2uQ90g/s200/cacio_e_pepe.jpg" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Cacio e pepe: simple&tasty or poor?</td></tr>
</tbody></table>
Taste has lots of different components, including not just what your tongue perceives but also smells, presentation, and expectations. But for most of these aspects, quality is in the eye of the beholder and we can't avoid coming to grips with the variety of people and cultures.<br />
<br />
Therefore, despite how good you think your burgers are, some people just don't like fat, mince meat. I appreciate Indian curries but I have some physical limits on the spiciness levels that make me almost always choose mild <a href="https://en.wikipedia.org/wiki/Korma">korma</a>. And imagine the cultural shock of discovering I have been wrongly putting lemon in tea all my life <a href="https://www.theguardian.com/science/brain-flapping/2014/oct/03/how-to-make-tea-science-milk-first">instead of milk as the only acceptable choice</a>.<br />
(I know, tea doesn't even grow in Europe, but I grew up with British tea as the standard.)<br />
<br />
We all have good intentions in thinking hard about what a user will enjoy or be productive with; but we have to recognize there is <a href="https://uxmastery.com/create-ux-personas/">a vast variety of users</a> and we have to design (or cook) for each of them.<br />
<h3>
Control the process, rather than micromanaging the material</h3>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgpXboVXlKX_-VoFiwojl8r8bKMYvRoZjZi0ygwX3hkJhwixSR8YwyeTqljnrqDYMOki7ezhhxJNLiJC9ZRa59T-LLj5TqjXMcH1XM6ZwzI3VzcaKc9F2RkH9O4FF_SwIpuWwZJ1A/s1600/angel_fluo.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="900" data-original-width="1600" height="112" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgpXboVXlKX_-VoFiwojl8r8bKMYvRoZjZi0ygwX3hkJhwixSR8YwyeTqljnrqDYMOki7ezhhxJNLiJC9ZRa59T-LLj5TqjXMcH1XM6ZwzI3VzcaKc9F2RkH9O4FF_SwIpuWwZJ1A/s200/angel_fluo.jpg" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">This cake was fluorescent on purpose</td></tr>
</tbody></table>
Convection ovens are a great example of a controlled process for cooking uniformly. In the context of large pieces of meat or fish, this mainly means getting them to a uniform, high-but-not-too-high temperature to avoid overcooking. The oven fan pushes hot air all around them, heating the surface evenly. Air transfers less heat than, for example, water; so there is time for the temperature to rise across your roasting chicken rather than overcooking the outside while leaving some parts dangerously raw.<br />
<br />
Thus for large cuts this is pretty much impossible to achieve in a pan, unless you literally cut everything into slices thin enough that they can cook quickly. The oven-based process is much more convenient as you literally abandon your tray in there, checking from time to time if it's ready with a thermometer.<br />
<br />
Generally speaking of enterprise applications and websites, I favor a process in which we catch bugs with multiple safety nets (up to user experimentation if possible) than overdesigning for every possible problem. While you can think of possible scenarios to test endlessly, bugs are always going to happen and it's more important to have a process in place for which they will be fixed and never regress because of new automated tests. That makes your software converge to a steady, stable state like a perfectly cooked chicken.<br />
<h3>
You need to measure</h3>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzMJyCMUN9WWf6r8fLTvlVwqBwczePefy4E75O-F8XCtL5kcPa1e50FX0LS1xSc7zm7jUIB_Ia6f7TukruAfPm8ypsVDuqnOnlGzHVZo7ihgHIVLBWXdfuEWqwBjVMW6HIjRXPUA/s1600/2019-02-03+17.16.20.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="900" data-original-width="1600" height="112" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzMJyCMUN9WWf6r8fLTvlVwqBwczePefy4E75O-F8XCtL5kcPa1e50FX0LS1xSc7zm7jUIB_Ia6f7TukruAfPm8ypsVDuqnOnlGzHVZo7ihgHIVLBWXdfuEWqwBjVMW6HIjRXPUA/s200/2019-02-03+17.16.20.jpg" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">That's definitely cold, not just for meat</td></tr>
</tbody></table>
If you want to consistently cook meat to your liking, there is no escape from using a thermometer to understand when it's ready - its center reaching a set temperature that corresponds to medium-rare (for a steak) or well-cooked (for poultry) or something else. Looking at the external color? No relation with the inside. Checking how hard it has become? Too subjective. Roast for a certain amount of time? Ignores the variability of both the ingredients and the heat source.<br />
<br />
When we perceive part of an application as slow, we need to use a profiler to find out what functions or methods are taking the most time to execute. Making as few assumptions as possible, we collect data to point us in the right direction. Opinions don't count: your browser timings and other metrics do (if collected correctly).<br />
<h3>
You can substitute ingredients, to a certain extent</h3>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjD0rxvsl8H8qIlIC0m4ZZoFUkOQ7HmpPLJzh0gr5iSARoCLdo2vLCX6_APFGaAVbZUyyH03ZgwoH9LGnh4hAUtxawDugiej0m_l2dQp890SN-QajXSpGq3w2r0GzfAH_PNX1RR0g/s1600/focaccia.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="900" data-original-width="1600" height="112" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjD0rxvsl8H8qIlIC0m4ZZoFUkOQ7HmpPLJzh0gr5iSARoCLdo2vLCX6_APFGaAVbZUyyH03ZgwoH9LGnh4hAUtxawDugiej0m_l2dQp890SN-QajXSpGq3w2r0GzfAH_PNX1RR0g/s200/focaccia.jpg" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Focaccia genovese</td></tr>
</tbody></table>
Cornstarch and flour are used in small quantities in many recipes, with the goal of thickening a liquid. This is due to their starch content, as this carbohydrate granules swell up with water creating enough friction to transform a liquid with the viscosity of water into something that feels like cream.<br />
<br />
If you try to use cornstarch to make bread however, you won't be able to get an elastic product as it lacks the proteins that would build gluten. Even if you use the wrong flour (cake flour as opposed to bread flour, to keep it simple), this will greatly affect the result due to the smaller <i>percentage</i> of proteins that it contains. Baking, both for sweet and savoury goods, require much more precision.<br />
<br />
In software development, we have grown up with Lego bricks as a metaphor and we continuously try to swap out pieces, hiding details behind a useful abstraction that sometimes leaks. Nowadays relational databases can be queried interchangeably if you stick to standard SQL queries. But the data types for columns can be pretty different in the range they support, especially if they are somewhat more exotic like JSON and XML fields rather than integer and strings. A wise decision is still required to understand when substituting components is possible, or where <a href="https://scholar.harvard.edu/waldo/publications/note-distributed-computing">some combinations will never work</a>.<br />
<br />
And here are 5 ways in which software development and cooking are very different...<br />
<h3>
Cooking is a repeatable process</h3>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgiinGEbJ4UfatskLztE1VK_Ft8KvJB0b3i26nUf1m4cPJd2NZP5m0aMmIpRnuIVT0l18XpZ8uUcKESsJpM4tkm6dK99OK_BUz-G4pSjcs02svyGLJ7jIGMOiijqru-xvhyASr7Mw/s1600/dutch_cheese.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="900" data-original-width="1600" height="112" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgiinGEbJ4UfatskLztE1VK_Ft8KvJB0b3i26nUf1m4cPJd2NZP5m0aMmIpRnuIVT0l18XpZ8uUcKESsJpM4tkm6dK99OK_BUz-G4pSjcs02svyGLJ7jIGMOiijqru-xvhyASr7Mw/s200/dutch_cheese.jpg" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Lots of Dutch cheese around Amsterdam</td></tr>
</tbody></table>
Recipes (at least the good ones) are literally the codification of a process that should be robust to external variations to get a consistent result. It's the mark of a good cook to be able to deal with variations in ingredients or tools, but unless you are up on a mountain water boils at pretty much the same temperature, and the physical transformation that your carrots undertake when they are heated is well established.<br />
<br />
In software, every new feature is a new design to make rather than the execution of a plan. Even porting software or reimplementing it bears surprises as the platform it is running on is now different. And no one understands how long it took to produce the original version, no matter give an estimation for the new one that is being created. We have processes for understanding what a feature should do, and safely implementing it and rolling it out; but there are always land mines waiting on the path.<br />
<h3>
Cooking has some <i>precise</i> physical quantities you can rely on </h3>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCoIMX0x8LP3xHFJM-y3cur_wcjic0yKrPhItnrw7IHZ6fT7-gsrwavYLG9VmDjUuhZGLbWK8Gce6TqVG-Cnh9GvKFa44BzNjuu_5uLlouCMN6VHHlYEEIumMOr1elffxbYZ-zHQ/s1600/pork_fillet.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1600" data-original-width="900" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCoIMX0x8LP3xHFJM-y3cur_wcjic0yKrPhItnrw7IHZ6fT7-gsrwavYLG9VmDjUuhZGLbWK8Gce6TqVG-Cnh9GvKFa44BzNjuu_5uLlouCMN6VHHlYEEIumMOr1elffxbYZ-zHQ/s200/pork_fillet.jpg" width="112" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Peppered pork fillet steak</td></tr>
</tbody></table>
Understood: measurement is needed in both fields. But as much as your oven oscillates around its target temperature, it is still much more precise than a developer's effort. Even without meetings and other time variables, how fast and precise we are in a certain day varies: humans aren't robots. Just how knowledgeable we are about a technology influences greatly the designing and testing phases. The <a href="https://en.wikipedia.org/wiki/The_Mythical_Man-Month">Mythical Man-Month</a> remains, well, mythical.<br />
<br />
In the food industry, the right tools can even measure the <a href="https://en.wikipedia.org/wiki/Wheat_flour#Flour_strength_%E2%80%93_W_index">strength of a flour</a>, to check whether it's good for the bread you want to obtain. If you look at a technology team, measuring how many tasks per week we have completed is probably as good as it gets. There's humans involved and applying social science to a very small group probably doesn't get you very far in terms of collecting data and drawing inferences.<br />
<br />
You can still measure other times objectively, like time to deploy: how long it takes for a commit on <i>master</i> to reach the production environment. We partly do this because it's important but also because it's feasible to measure. What most project managers would care about will be time from idea to complete implementation instead. But that requires estimating the length of a queue that changes all the time, and is just the first step of a creative development process with its own variations.<br />
<h3>
Determinism of the digital world</h3>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhMcxIGUlvQpsYCVETRP8IsGwukGS2oR2SJJfY3EHYLr1ZjFG9ACaW1_JDDb0to8BjO88RwvpeD61ohu-sZPygcrMluQQx0xA_u5lsGDXgiCRaLcMK_2IH4JcTunkZOL1QVD3zFWw/s1600/2017-08-12+17.12.46.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="900" data-original-width="1600" height="112" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhMcxIGUlvQpsYCVETRP8IsGwukGS2oR2SJJfY3EHYLr1ZjFG9ACaW1_JDDb0to8BjO88RwvpeD61ohu-sZPygcrMluQQx0xA_u5lsGDXgiCRaLcMK_2IH4JcTunkZOL1QVD3zFWw/s200/2017-08-12+17.12.46.jpg" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Blackberries from the garden</td></tr>
</tbody></table>
<div style="text-align: right;">
</div>
It's pretty difficult to get the same tomatoes, courgettes or grapes as last week, and pretty much impossible to get the same ones in-season and off-season. You can ship them in from South Africa or Australia but travel time and refrigeration can modify their contents, and thus their taste.<br />
<br />
If you look at a physical server, it's much more similar to laboratory equipment than to a living product: you can run programs and see them always taking a similar amount of time to complete, controlling the randomness of the operating system around it. This gets eroded a bit in the cloud, where performance may be affected by your neighbors due to <a href="https://link.medium.com/oXoQFtVDDT">co-tenancy</a>.<br />
<h3>
Timing in the kitchen</h3>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhSlD9NB4x4yU9G6b3imKg4aHL3OWf5uG9u0IpfozBBN_j2x89wCA4PHt8iQT6SapN_6ysuxapFwV6FGJVuEF9gz6xwLcKPWHcnOZS9QGv9CbmPhmawvLmGyOpjQ27rtf3vTLH2SA/s1600/crochembouche.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1600" data-original-width="900" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhSlD9NB4x4yU9G6b3imKg4aHL3OWf5uG9u0IpfozBBN_j2x89wCA4PHt8iQT6SapN_6ysuxapFwV6FGJVuEF9gz6xwLcKPWHcnOZS9QGv9CbmPhmawvLmGyOpjQ27rtf3vTLH2SA/s200/crochembouche.jpg" width="112" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Very unstable crochembouche</td></tr>
</tbody></table>
<div style="text-align: left;">
</div>
Whether it is simply changing the temperature of a meat joint, or a more complex transformation like baking a cake, timing should be one of the concerns if you want to obtain a good result. I formalize this concept by thinking that it's not possible to stop time, in many cases.<br />
<br />
Cooks know tricks like cooking <a href="https://www.seriouseats.com/2014/10/the-food-lab-how-to-poach-eggs-for-a-party.html">eggs</a> or rice to a certain degree, than cooling it down and finish the process later when the food has to be served; or simply reheat it if fully cooked. This works for various categories of products, but it's an ad-hoc process.<br />
<br />
Consider the power we have in a digital world: firing up a debugger literally stops execution at some point in the life of the program, allowing us to take a look at what we want in the right context. Since the state of the program is <a href="https://en.wikipedia.org/wiki/The_Matrix">the Matrix</a>, we can slow it down, speed it up, and change things causing a <a href="https://matrix.fandom.com/wiki/D%C3%A9j%C3%A0_Vu">déjà vu</a> to your objects.<br />
<br />
If you want to reproduce some computation, you have the tools available to build a Docker image containing all sorts of dependencies and store it for future usage. If you want to reproduce your perfect croissants, the only tools you have are a recipe and your own memories. Add the variation of ingredients and even temperature and humidity in your kitchen, and you can understand why scientific exploration needs a laboratory with its controlled conditions to be able to make progress.<br />
<h3>
Cooking equipment makes a difference </h3>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgU6oS5uNTRaepi9lrWhw5-0dwm3xJpp8q9sjviN0-tJ5n8-i9PM6o7ylwzTtSjsLSWmmmIV_em5zbGLpVLGmaZcIZmbqTPNwLPKYeLatwaj4rORxq7T76oQiWD-KC_WBZk7H4fOQ/s1600/2019-02-03+17.23.56.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1600" data-original-width="900" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgU6oS5uNTRaepi9lrWhw5-0dwm3xJpp8q9sjviN0-tJ5n8-i9PM6o7ylwzTtSjsLSWmmmIV_em5zbGLpVLGmaZcIZmbqTPNwLPKYeLatwaj4rORxq7T76oQiWD-KC_WBZk7H4fOQ/s200/2019-02-03+17.23.56.jpg" width="112" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Now, grate parmesan without this...</td></tr>
</tbody></table>
Besides basic tools like appropriately shaped knives, a pressure cooker would make you able to reach certain results that would take a long time with an ordinary pot of boiling water. A <a href="https://www.seriouseats.com/2016/10/food-lab-complete-guide-to-sous-vide-rack-of-lamb.html">temperature bath</a> (I don't own one of these) can help cooking meat evenly only to then finish the process with a 2-minute searing. Even a <i>scale</i> is just <i>necessary</i> for baking, as measuring ingredients like flour by volume has a 50% margin of error due to its compressibility.<br />
<br />
Consider how you can write code on your old laptop from the beach instead. You target an open source interpreter, and the end product will run on the same server that could accept strictly regulated banking software. As long as you can literally string bytes together, you can produce running software: everything else helps. The ephemeralization of software tools due to virtualization and the large availability of open source platforms make digital startups a reality, whereas opening a restaurant remains a capital-intensive operation.<br />
<br />
But there's more...<br />
<h3>
The power of metaphors </h3>
Metaphors can foster understanding of a new system, or lead us ashtray. They are powerfully transmitting a mental model, but that model has its limitations and may even be less precise than a more formal model like a math analogy. But especially in complicated fields like cryptography, terms such as <i>key</i> and <i>signature</i> have popularized concepts to generation of students that would have otherwise found them very hard to think about.<br />
<br />
I wrote this post for fun, but I stand behind most of the comparisons: that's all for now. You'll find me using an <a href="https://www.giorgiosironi.com/2019/01/practical-helm-in-5-minutes.html">Helm</a> to ship my containers...Giorgiohttp://www.blogger.com/profile/12689416577856305650noreply@blogger.com0tag:blogger.com,1999:blog-36547168.post-62055741537659528782019-01-10T19:59:00.000+01:002019-01-10T20:33:53.327+01:00Practical Helm in 5 minutes<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://helm.sh/" style="margin-left: auto; margin-right: auto;"><img alt="https://helm.sh/" border="0" data-original-height="256" data-original-width="256" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTtLSr4PM5XbdGj3lbx4H92mk4yPi55hvR1DiyqS7Od6UdrHuZ0j3MCI1iOw1TgIMxuf8pT5vnpSONiZwaO0x35gqEDUKscRAdp9xcIUP_PO56bM83_UhLrj9ssO-9Yu4Iutf4HQ/s1600/helm.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Yet another ship-themed name</td></tr>
</tbody></table>
<br />
Containerization is increasingly a powerful way to deploy applications on anonymous infrastructure, such as a set of many identical virtual machines run by some cloud provider. Since container images ship a full OS, there is no need to manage packages for the servers (a PHP or Python interpreter), but there are still other environment-specific choices that need to be provided to actually run the application: configuration files and environment variables, ports, hostnames, secrets.<br />
<br />
In an environment like Kubernetes, you would create all of this declaratively, writing YAML files describing each Pod, ConfigMap, Service and so on. Kubernetes will take these declarations and apply them to its state to reach what is desired.<br />
<br />
As soon as you move outside of a demo towards multiple environments, or towards updating one, you will start to see Kubernetes YAML resources not directly as code to be committed into a repository, but as an output of a generation process. There are many tweaks and customizations that need to be performed in each environment, from simple hostnames (<span style="font-family: "courier new" , "courier" , monospace;">staging--app.example.com</span> vs <span style="font-family: "courier new" , "courier" , monospace;">app.example.com</span>) to entire sections being present or not (persistence and replication of application instances).<br />
<br />
The problem you need to solve then is to generate Kubernetes resources from some sort of templates: you could choose any template engine for this task, and execute <span style="font-family: "courier new" , "courier" , monospace;">kubectl apply</span> on the result. To avoid reinventing the wheel, <a href="https://helm.sh/">Helm</a> and other competitors were created to provide an higher abstraction layer.<br />
<h3>
Enter Helm</h3>
Helm provides templating for Kubernetes .yaml file; as part of this process, it extracts the configuration values for Kubernetes resources into a single, hierarchical data source.<br />
<br />
Helm doesn't stop there however: it aims to be a package manager for Kubernetes, hence it won't just create resources such as a Deployment, but it will also:<br />
<ul>
<li>apply the new resources on the Kubernetes cluster</li>
<li>tag the Deployment with metadata and labels</li>
<li>list everything that is installed in terms of applications, rather then Deployments and ConfigMaps</li>
<li>find older versions of the Deployment to be replaced or removed</li>
</ul>
The set of templates, helpers, dependencies and default values Helm uses to deploy an application is called a <i>chart</i> whereas every instance of a chart created on a cluster is called a <i>release</i>. Therefore, Helm keeps track of objects in terms of releases and allows you to update a release and all its contents, or to remove it and replace it with a new one.<br />
<h3>
Folder structure</h3>
The minimal structure of an Helm chart is simply a folder on your filesystem, whose name must be the name of the chart. As an example, I'll use <i>green-widgets</i> as a name, a fictional web application for ordering green widgets online.<br />
<br />
This is what you'll see inside a chart:<br />
<ul>
<li><span style="font-family: "courier new" , "courier" , monospace;">Chart.yaml</span>: metadata about the chart such as name, description and version.</li>
<li><span style="font-family: "courier new" , "courier" , monospace;">values.yaml</span>: configuration values that may vary across releases. At a bare minimum the image name and tag will have defaults here, along with ports to expose.</li>
<li>the <span style="font-family: "courier new" , "courier" , monospace;">templates/</span> subfolder: contains various YAML templates that will be rendered as part of the process of creating a new release. There is more in this folder like a readme for the user and some helper functions for generating common snippets.</li>
</ul>
Apart from this minimal setup, there may also be a <span style="font-family: "courier new" , "courier" , monospace;">requirements.yaml</span> file and a <span style="font-family: "courier new" , "courier" , monospace;">charts/</span> subfolder to deal with other charts to use as dependencies; for example, to install a database through an official chart rather than setting up PostgreSQL replication on your own. These can be safely ignored until you need these features though.<br />
<br />
Once you have the <span style="font-family: "courier new" , "courier" , monospace;">helm</span> binary on your system, you can generate a new chart with <span style="font-family: "courier new" , "courier" , monospace;"><i>helm create green-widgets</i></span>.<br />
<h3>
Cheatsheet</h3>
You can download a <span style="font-family: "courier new" , "courier" , monospace;">helm</span> binary for your platform from the <a href="https://github.com/helm/helm/releases">project's releases page on Github</a>. The <span style="font-family: "courier new" , "courier" , monospace;">helm init</span> command will use your kubectl configuration (and authentication) to install <span style="font-family: "courier new" , "courier" , monospace;">tiller</span>, the server-side part of Helm, onto a cluster's system namespace.<br />
<br />
Once this is setup, you will be able to execute <span style="font-family: "courier new" , "courier" , monospace;">helm install</span> commands against the cluster, using charts on your local filesystem. For real applications, you can install official charts that are automatically discovered from the default Helm repositories.<br />
<br />
The command I prefer to use to work on a chart however is:<br />
<span style="font-family: "courier new" , "courier" , monospace;">helm upgrade --install --set key=value green-widgets--test green-widgets/</span><br />
<br />
The mix of <i>upgrade</i> and <i>install</i> means this command is idempotent and will work for the first installation as well as for updates. Normally you would issue a new release for a change to the chart, but this approach allows you to test out a chart while it's in development, using a 0.0.1 version.<br />
There is no constraint on the release name <span style="font-family: "courier new" , "courier" , monospace;">green-widgets--test</span>, and Helm can even generate random names for you. I like to use the application name and its environment name as a team convention, but you should come up with your own design choices.<br />
<br />
A final command to keep in mind is <span style="font-family: "courier new" , "courier" , monospace;">helm delete green-widgets--test</span> which will delete the release and all the resources created by your templates. This is enough to stop using CPU, memory and IP addresses, but it's not enough to completely remove all knowledge of the release from Tiller's archive. To do so (and free the release name allowing its re-creation) you should use add the <span style="font-family: "courier new" , "courier" , monospace;">--purge</span> flag.<br />
<h3>
Caveats</h3>
This 5-minute introduction makes it all seem plain and simple, but it should be clear that simply downloading Helm and installing it is not a production-ready setup. I myself have only rolled out this setup to testing environment at the time of writing.<br />
<br />
I can certainly see several directions to explore, that I either cut from the scope in order to get these environments up and running for code review; or investigated and used but not included in this post. For example:<br />
<ul>
<li><span style="font-family: "courier new" , "courier" , monospace;">requirements.yaml</span> allows to include other charts as dependencies. This is very powerful for off-the-shelf open source software such as databases, caches and queues; it needs careful choices for the configuration values being passed to these dependencies, and your mileage may vary with the quality of the chart you have chosen.</li>
<li>chart repositories are a good way to host stable chart versions rather than copying them onto a local filesystem. For example, you could <a href="https://github.com/hypnoglow/helm-s3">push tarballs to S3</a> and have a plugin regenerate the index.</li>
<li>the whole Helm and Tiller setup arguably needs to be part of a Infrastructure as Code apporach like the rest of the cluster. For example, I am <a href="https://github.com/elifesciences/terraform-eks">creating a EKS cluster using Terraform</a> and that would need to include also the installation and configuration of Tiller to provide a turnkey solution for new clusters.</li>
</ul>
Giorgiohttp://www.blogger.com/profile/12689416577856305650noreply@blogger.com0tag:blogger.com,1999:blog-36547168.post-10073094416950479822019-01-02T10:42:00.000+01:002019-01-10T20:32:53.695+01:00The path from custom VM to VM with containers<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File:Kanda_container.jpg" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img alt="https://commons.wikimedia.org/wiki/File:Kanda_container.jpg" border="0" data-original-height="853" data-original-width="1280" height="213" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_azNVdRFmTr0WJ7z7jct6L3uDWwyE7V-MfrbW17D6oZHQhRJP5L2hcwZyzsu8q5EoprDHpAyvNV9NnCdxnLGGzpOA7ag1vDCru86phZnspvKSYjMKfotTMlXDdPFKiocPjQrw6A/s320/1280px-Kanda_container.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Image of a single container being transported by <a href="https://commons.wikimedia.org/wiki/File:Kanda_container.jpg">OiMax</a></td></tr>
</tbody></table>
Before the transition to Docker containers started at eLife, a single service deployment pipeline would pick up the source code repository and deploy it to one or more virtual machines on AWS (EC2 instances booted from a standard AMI). As the pipeline went across the environments, it repeated the same steps over and over in testing, staging and production. This is the story of the journey from a pipeline based on source code for every stage, to a pipeline deploying an immutable container image; the goal pursued here being the time savings and the reduced failure rate.<br />
<br />
The end point is seen as an intermediate step before getting to containers deployed into an orchestrator, as our infrastructure wasn't ready to accept a Kubernetes cluster when we started the transition, nor Kubernetes itself was trusted yet for stateful, old-school workloads such as running a PHP applications that writes state on the filesystem. Achieving containers-over-EC2 allows developers to target Docker as the deployment platform, without realizing yet cost savings related to the bin packing of those containers onto anonymous VMs.<br />
<h3>
Starting state</h3>
A typical microservice for our team would consist of a Python or PHP codebase that can be deployed onto a usually tiny EC2 instance, or onto more than one if user-facing. Additional resources that are usually not really involved in the deployment process are created out of band (with <a href="https://www.martinfowler.com/bliki/InfrastructureAsCode.html">Infrastructure as Code</a>) for this service, like a relational database (outsourced to RDS), a load balancer, DNS entries and similar cloud resources.<br />
<br />
Every environment replicates this setup, whether it is a <i>ci</i> environment for testing the service in isolation, or an <i>end2end</i> one for more large-scale testing, or even a sandbox for exploratory, manual testing. All these environments try to mimic the <i>prod</i> one, especially <i>end2end</i> which is supposed to be a perfect copy on fewer resources.<br />
<br />
A deployment pipeline has to go through environments as a new release is promoted from <i>ci</i> to <i>end2end</i> and <i>prod</i>. The amount of work that has to be repeated to deploy from source on each of the instances is sizable however:<br />
<br />
<ul>
<li>ensure the PHP/Python interpreter is correctly setup and all extensions are installed</li>
<li>checkout the repository, which hopefully isn't too large</li>
<li>run scripts if some files need to be generated (from CSS to JS artifacts and anything similar)</li>
<li>installing or updating the build-time dependencies for these tasks, such as a headless browser to generate <a href="https://www.smashingmagazine.com/2015/08/understanding-critical-css/">critical CSS</a></li>
<li>run database migrations, if needed</li>
<li>import fixture data, if needed</li>
<li>run or update stub services to fill in dependencies, if needed (in testing environments)</li>
<li>run or update real <a href="https://docs.microsoft.com/en-us/azure/architecture/patterns/sidecar">sidecar</a> services such as a queue broker or a local database, if present</li>
</ul>
These ever-expanding sequence of operations for each stage can be optimized, but in the end the best choice is not to repeat work that only needs to be performed once per release.<br />
<br />
There is also a concern about the end result of a deploy being different across environments. This difference could be in state, such as a JS asset served to real users being different from what you tested; but also in outcome, as a process that can run perfectly in testing may run into a APT repository outage when in production, failing your deploy halfway through, only on one of the nodes. Not repeating operations leads not just to time savings but to a simpler system in which fewer operations can fail just because there are fewer of them in general.<br />
<h3>
Setting a vision</h3>
I've automated before <b>builds that generated a set of artifacts</b> from the source code repository and <b>then deploy that across environments</b>, for example zipping all the PHP or Python code into an archive or in some other sort of package. This approach works well in general, and it is what compiled languages naturally do since they can't get away with recompiling in every environment. However, artifacts do not take into account OS level dependencies like the Python or PHP version with their configuration, along with any other setup outside of the application folder: a tree of directories for the cache, users and groups, deb packages to install.<br />
<br />
Container images promise to ship a full operating system directory tree, which will run in any environment only sharing a kernel with its host machine. Seeing <i>docker build</i> as the natural evolution of <i>tar -cf ... | bzip2</i>, I set out to port the build processes of the VMs into portable container images per each service. We would then still be deploying these images as the only service on top an EC2 virtual machine, but each deployment stage should just be consisting of pulling one or more images and starting them with a docker-compose configuration. The stated goal was to reduce the time from commit to live, and the variety of failures that can happen along the way.<br />
<h3>
Image immutability and self-sufficiency</h3>
To really save on deployment time, the images being produced for a service must be the same across environments. There are some exceptions like a <i>ci</i> derivative image that adds testing tools to the base one, but all prod-like environment should get the same artifact; this is not just for reproducibility but primarily for performance.<br />
<br />
The approach we took was to also isolate services into their own containers, for example creating two separate <i>fpm</i> and <i>nginx</i> images (<i>wsgi</i> and <i>nginx</i> for Python); or to use a standard <i>nginx</i> image where possible. Other specialized testing images like our own <i>selenium</i> extended image can still be kept separate.<br />
<br />
The isolation of images doesn't just make them smaller than a monolith, but provides Docker specific advantages like leveraging independent caching of their layers. If you have a monolith image and you modify your composer.json or package.json file, you're in for a large rebuild. But segregating responsibilities leads instead to only one or two of the application images being rebuilt: never having to reinstall those packages for Selenium debugging. This can also be achieved by embedding various targets (<i>FROM ... AS ...</i>) into a single Dockerfile, and having docker-compose build one of them at a time with the <i>build.target</i> option.<br />
<br />
When everything that is common across the environments is bundled within them, what remains is configuration in the form of docker-compose.yml and other files:<br />
<ul>
<li>which container images should be running and exposing which ports</li>
<li>which commands and arguments the various images should be passed when they are started</li>
<li>environment variables to pass to the various containers</li>
<li>configuration files that can be mounted as volumes</li>
</ul>
Images would typically have a default configuration file in the right place, or be able to work without one. A docker-compose configuration can then override that default with a custom configuration file, as needed.<br />
<br />
One last responsibility of portable Docker images is their definition of a basic <i>HEALTHCHECK</i>. This means an image has to ship enough basic tooling to, for example, load a <i>/ping</i> path on its own API and verify a 200 OK response is coming out. In the case of classic containers like PHP FPM or a WSGI Python container, this implies some tooling will be embedded into the image to talk to the main process through that protocol rather than through HTTP.<br />
<br />
It's a pity to reinvent the lifecycle management of the container (being started, then healthy or unhealthy after a series of probes), whereas we can define a simple command that both docker-compose or actual orchestrators like Kubernetes can execute to detect the readiness of the new containers after deploy. I used to ship smoke tests with the configuration files to use, but these have largely been replaced by polling for an health status on the container itself.<br />
<h3>
Image size</h3>
Multi-stage builds are certainly the tool of choice to keep images small: perform expensive work in separate stages, and whenever possible only copy files into the final stage rather than executing commands that use the filesystem and bloat the image with their leftover files.<br />
<br />
A consolidated RUN command is also a common trick to bundle together different processes like <i>apt-get update</i> and <i>rm /var/lib/apt/lists/*</i> so that no intermediate layers are produced, and temporary files can be deleted before a snapshot is taken.<br />
<br />
To find out where this optimization is needed however, some introspection is needed. You can run <i>docker inspect</i> over a locally built image to check its <i>Size</i> field and then <i>docker history</i> to see the various layers. Large layers are hopefully being shared between one image and the next if you are deploying to the same server. Hence it pays to verify that if the image is big, most of its size should come from the ancestor layers and they should seldom change.<br />
<br />
A final warning about sizes is related to images with many small files, like node_modules/ contents. These images may exhaust the inodes of the host filesystem well before they fill up the available space. This doesn't happen when deploying source code to the host directly as files can be overwritten, but every new version of a Docker image being deployed can easily result in a full copy of folders with many small files. Docker's <a href="https://github.com/elifesciences/builder-base-formula/blob/master/elife/docker-scripts/docker-prune"><i>prune</i></a> commands often help by targeting various instance of containers, images and other leftovers, whereas <i>df -i</i> (as opposed to <i>df -h</i>) diagnoses inodes exhaustion.<br />
<h3>
Underlying nodes</h3>
Shipping most of the stack in a Docker image makes it easier to change it as it's part of an immutable artifact that can be completely replaced rather than a stateful filesystem that needs backward compatibility and careful evolution. For example, you can just switch to a new APT repository rather than transition from one to another by removing the old one; only install new packages rather than having to remove the older ones.<br />
<br />
The host VMs become leaner and lose responsibilities, becoming easier to test and less variable; you could almost say all they have to run is a Docker daemon and very generic system software like syslog, but nothing application-specific apart from container dependencies such as providing a folder for config files to live on. Whatever Infrastructure as Code recipes you have in place for building these VMs, they will become easier and faster to test, with the side-effect of also becoming easier to replace, scale out, or retire.<br />
<br />
An interesting side effect is that most of the first stages of projects pipelines lost the need for a specific CI instance where to deploy. In a staging environment, you actually need to replicate a configuration similar to production like using a real database; but in the first phases, where the project is tested in isolation, the test suite can effectively run on a generic Jenkins node that works for all projects. I wouldn't run multiple builds at the same time on such a node as they may have conflicts on host ports (everyone likes to listen on <i>localhost:8080</i>), but as long as the project cleans up after failure with <i>docker-compose down -v</i> or similar, a new build of a wholly different project can be run with practically no interaction.<br />
<h3>
Transition stages</h3>
After all this care in producing good images and cleaning up the underlying nodes, we can look at the stages in which a migration can be performed.<br />
<br />
A first rough breakdown of the complete migration of a service can be aligned on environment boundaries:<br />
<ol>
<li>use containers to run tests in CI (xUnit tools, Cucumber, static checking)</li>
<li>use containers to run locally (e.g. mounting volumes for direct feedback)</li>
<li>roll out to one or more staging environments</li>
<li>roll out to production</li>
</ol>
This is the path of least resistance, and correctly pushes risk first to less important environments (testing) and only later to staging and production; hence you are free to experiment and break things without fear, acquiring knowledge of the container stack for later on. I think it runs the risk of leaving some projects halfway, where the testing stages have been ported but production and staging still run with the host-checks-out-source-code approach.<br />
<br />
A different way to break this down is perform the environment split by considering the single processes involved. For example, consider an application with a server listening on some port, a cli interface and a long-running process such as a queue worker:<br />
<ol>
<li>start building an image and pulling it on each enviroment, from CI to production</li>
<li>try running CLI commands through the image rather than the host</li>
<li>run the queue worker to the image rather than the host</li>
<li>stop old queue worker</li>
<li>run the server, using a different port</li>
<li>switch the upper layer (nginx, a load balancer, ...) to use the new container-based server</li>
<li>stop old server</li>
<li>remove source code from the host</li>
</ol>
Each of these slices can go through all the environments as before. You will be hitting production sooner, which means Docker surprises will propagate there (it's still not as stable as Apache or nginx); but issues that can only be triggered in production will happen on a smaller part of your application, rather than as a big bang of the first production deploy of these container images. <br />
<br />
If you are using any dummy project, stub or simulator, they are also good candidates for being switched to a container-based approach first. They usually won't get to production however, as they will only be in use in CI and perhaps some of the other testing environments.<br />
<br />
You can also see how this piece-wise approach lets you run both versions of a component in parallel, move between one and the other via configuration and finally remove the older approach when you are confident you don't need to roll back. At the start using a Docker image doesn't seem like a huge change, but sometimes you end up with 50 modified files in your Infrastructure as Code repository, and 3-4 unexpected problems to get them through all the environments. This is essentially <a href="https://www.martinfowler.com/bliki/BranchByAbstraction.html">Branch by Abstraction</a> applied to Infrastructure as Code: a very good idea for incremental migrations applied to an area that normally needs to move at a slower pace than application code.Giorgiohttp://www.blogger.com/profile/12689416577856305650noreply@blogger.com0tag:blogger.com,1999:blog-36547168.post-54264701580136943332018-12-28T10:28:00.003+01:002018-12-28T10:30:00.545+01:00Delivery pipelines for CDNs<div class="separator" style="clear: both; text-align: center;">
<a href="https://www.fastly.com/network-map" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img alt="https://www.fastly.com/network-map" border="0" data-original-height="567" data-original-width="1140" height="159" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhB-pYFTNyrJ5Yq15nX9rycSG0EZ3-weMzVFSEjgp1dL-2JPOU-PexZ-HcVkSkkUoiKLVdS0ZZggFOAqUgWBm4OewwcZmUt0HVtmGY2RjvXnKdPbPlhEAM5XfDKgd3tCyFT8u9Jfw/s320/fastly-map.png" width="320" /></a></div>
In the last couple of years I have integrated Content Delivery Networks into various <a href="https://elifesciences.org/">eLife</a> applications, managing objects ranging from static files and images to dynamic HTML. These projects mainly consisted of:<br />
<ul>
<li>implementing <a href="https://www.thoughtworks.com/insights/blog/infrastructure-code-reason-smile">Infrastructure as Code</a> for these CDNs inside the Github repositories we already use for all other cloud resources (AWS and GCP)</li>
<li>effectively authorize HTTPS on the CDN side, which will be impersonating your origin servers</li>
<li>create instances of the same CDN services, first in testing and then in production environments, keeping them in parity with each other</li>
<li>expand end-to-end testing (the tip of the pyramid) to cover also the CDNs rather than just covering the applications involved</li>
<li>integrate logging in order to catch any problem happening between the user and the origin servers</li>
<li>finally phase in the new CDNs with new geotagged DNS entries</li>
</ul>
Our first implementation from 2016 was widely integrated into AWS and as such <a href="https://aws.amazon.com/cloudfront/">CloudFront</a> was the chosen solution. We subsequently switched to <a href="https://www.fastly.com/">Fastly</a> for all ordinary traffic, experiencing a general increase in features, customization and expenses. What follows is a comparison that isn't just meant to orient the reader between CloudFront and Fastly, but also against the third option of not using a CDN at all. In fact, there are many concerns that may be glossed upon but that you need to take into account seriously when you move your web presence from a few origin servers to a global network of shared, locked down servers managed by an external organization.<br />
<h3>
Infrastructure as Code</h3>
Our AWS-based setup is making a large use of <a href="https://aws.amazon.com/cloudformation/">CloudFormation</a>, the native service for declaratively specifying resources such as servers, load balancers and disks. The simple setup has been augmented over the years by a code generation layer for the CloudFormation templates; this Python code reduces duplication between the various templates by starting from standard EC2/ELB/EBS resources that can be customized in size and other parameters.<br />
If we start from a simple single-server setup for a microservice (this was before Docker containers got stable enough), we are looking at a template containing at least an EC2 instance and a DNS entry pointing to it. With multiple servers, we expand this with a load balancer that pulls in a TLS certificate provided to IAM by an administrator.<br />
To configure CloudFront via CloudFormation, an additional resource for the CDN distribution is introduced. All the configuration you need will be visible in this resource, a JSON dictionary or XML tag respecting a certain schema.<br />
Since CloudFormation can only manage AWS resources and nothing outside that tended garden, Fastly was the reason for introducing <a href="https://www.terraform.io/">Terraform</a> alongside it. Whereas almost anything AWS-specific still goes through CloudFormation, Terraform has opened up new roads such as Infrastructure as Code implementations for Google Cloud Platform (storage buckets and BigQuery tables).<br />
Applying changes in this context is not trivial as you may inadvertently reboot or destroy a server while believing you were only changing a minor setting. Yet Infrastructure as Code is about making the current state of infrastructure and <b>all changes visible, easy to review and safe to rollout across multiple environments</b>. It is imperative therefore to maintain testing environments created with the same tooling as production, and to use them to ultimately integration test all changes.<br />
The caveat of using multiple tools in lockstep for the same instance of a project (including servers, cloud resources and CDNs) is that they <b>can't declare dependencies between resources managed by different tools</b>. For example, since we manage DNS in CloudFormation and Fastly CDNs in Terraform, we can both at the same time but can't couple together the existence of a DNS and the CDN it points to, or impose a creation or update order that is different from the general order we run the tools in.<br />
The most glaring difference in updates rollout between the various options is that, to rollout a CDN configuration change, it takes:<br />
<ul>
<li>no deployment time if you don't use a CDN (obviously)</li>
<li>10s of seconds for Fastly</li>
<li>10s of minutes (up to 1 hour was common) for CloudFront</li>
</ul>
This means Fastly opens up the possibility for experimentation, even if with a slower feedback that your local TDD cycle. With CloudFront this is painful and haphazard as you decide on a change, start applying it and come back one hour later to check its effects, after having already switched to another task.<br />
Still, <b>minutes</b> of update and/or creation time make Fastly unavailable for inclusion in the CI environments where the tests of a single service are run. You could in theory create a Fastly service on the fly when the build of the service runs, but this will add minutes to your build _and_ also promote coupling to the CDN itself. Fast forward this a bit and <b>you'll see an application unable to be run locally anymore</b> for exploration because of the missing CDN layer. Therefore, like cloud services the CDN is treated like a long-lived resource, with its regression testing performed into a shared environment on every new application commit, but after merge.<br />
<h3>
Logging</h3>
Within a web service, you usually have some kind of access log being generated by nginx or Apache. These logs can sit on a single server or can be uploaded to some aggregation point, whether it is a local Logstash or an external platform that can index them.<br />
Even load balancing doesn't change this picture very much as the load balancer(s) logs should be identical to the ones of the application servers if everything is working well. But with a CDN, large-scale caching is introduced and so it's plausible that you will stop directly seeing a large percentage of your traffic. Statistics or monitoring based on access logs may get skewed; or worse, Japan may be cut off from your website for a while because the health checks from the CDN points of presence there have a timeout of a few milliseconds too low to get to your servers in us-east-1 (of course this never happened).<br />
Hence, to understand what's going on in those few hundred servers you have no access to, you need a way to stream them to some outsourced service; this can be storage as a service (S3 or GCS) or directly some log infrastructure provider. The latency with which logs can get in the right place is a key metric of the feedback loop from changes.<br />
Since we are striving for Infrastructure as Code, <b>all the logging configuration should be kept under version control</b> together with hostnames and caching policies. We got to a standard logging format (JSON Lines with certain fields) and frequency, along with GCS bucket where to put new entries, bucket names following conventions. This was later expanded into BigQuery tables providing queries over the same data, after the Terraform Fastly provided started supporting this delivery mechanism.<br />
The main difficulty in integration was <b>credentials management</b>: you aren't told much if credentials are not correct or not authorized to perform certain actions like writing to BigQuery. Moreover, you can't just commit a bunch of private keys for anyone to see, especially since Infrastructure as Code repositories tend to be made very visible to as many people as possible.<br />
We ended up putting GCP credentials and similar secrets in <a href="https://vault.io/">Vault</a>, running on the same server as the Salt master (same thing as Puppet master). The GCP Service Account itself and its permissions to write to the bucket needed some special permissions to set up (it's turtles all the way down) so couldn't put it directly into Infrastructure as Code but had an admin manually creating it instead. The ideal thing would be for Vault to generate credentials by itself, following the pattern of periodically rotating them. But then it would need to push these credentials somehow into the Fastly configuration and I'm here to provide efficient delivery pipelines, not make cloud giants wrestle.<br />
<h3>
Flexibility</h3>
Your own application is usually highly customizable, with a certain cost associated. You have to write some code in your favorite programming language, possibly following some framework conventions and calling your classes <i>Middleware</i> or <i>EventListener</i>.<br />
CDNs work on shared servers, so they have limits on what can be safely run in that sandboxed environment. Nevertheless, Fastly provides the possibility to customize the <a href="https://docs.fastly.com/vcl/">VCL</a> that runs each service with your own snippets and macros.<br />
This is very flexible, perhaps even too much: you can introduce headers with random values, write conditionals and implement loops by restarting requests. It feels similar to working in nginx configurations but with a more predictable language.<br />
The main problem with this form of customization is that <b>there is no way to run it or test it on your own</b>. The best feedback loop we found is the <a href="https://fiddle.fastlydemo.net/">Fastly Fiddle</a> (similar to JS Fiddle) where you test out bits of code, hit a save button and see it propagated to servers around the world for you to test.<br />
The fact that this even exists is impressive, but you can imagine how well it works for actual development. Once you get past experimenting, you can't integrate a Fiddle with your own Infrastructure as Code approach (e.g. Terraform templates) nor easily port code from one to another besides copying and pasting. You can run integration-only tests in some other window, but the feedback loop can't be shorter than the deployment time; unit tests are not a thing. You can't even use your IDE as much as you may love it. In the end, <a href="https://www.fastly.com/blog/benefits-using-varnish">Fastly's Varnish diverged from the open source one 4 major versions ago</a>; hence, this VCL is a proprietary language and you'll feel the same as writing stored procedures in Oracle's PL/SQL.<br />
I tend to see VCL and other intermediate declarative templates (such as Terraform .tf files) as <a href="https://github.com/elifesciences/builder/blob/master/src/buildercore/fastly.py">a generation target for Infrastructure as Code to compile to</a>. This lets you unit test that your tools generate a certain output for these templates; use dummy inputs in tests and check dummy expected outputs; all of this will still need to be integration tested with the application itself in a real environment, but some of the responsibilities can be developed in the tool itself and reused across many applications.<br />
<h3>
Integration testing</h3>
We have understood by now that to keep the ensemble of servers, code, cloud services and CDNs we <b>need some automated integration testing in place that touches all the different pieces</b>. We don't want many scenarios to be tested at this level because it's slow and brittle to do so, but we need a tracer bullet that goes through <i>everything</i>, if only to verify all configurations are correct.<br />
In the general context of outsourcing of responsibilities to a service or a library, you still own it as a dependency of your application and still need to verify the emergent behavior of custom code and borrowed architecture.<br />
Therefore, I always put at least a staging environment in place replicating production where <i>automated</i> tests can run. This doubles as the place where to try and roll out infrastructure updates that are risky (which are risky? If you have to ask, all of them; just roll out everything through staging).<br />
As we have seen, creating too many different, ad-hoc environments to test pull requests doesn't scale; this will reach death by feature branch as all of your Jenkins nodes are waiting for yet one more RDS node or CloudFront distribution to be created.<br />
A common example of a coupled, integration-related feature to test is the forwarding of <i>Host</i> and other headers; these go through so many layers: a couple of CDN servers, a load balancer, an nginx daemon and finally the application. Some headers don't just have to be forwarded, but have to be rewritten or renamed or added (<i>X-Forwarded-For</i>). All of this can in theory be specified for every single layer but testing the whole architecture probably makes for easier long-term maintenance.<br />
<h3>
Why?</h3>
In various projects you always have to ask yourself why you are doing something (especially complex things) and what value you want to get out of it. CDNs are one of the go-to solution for web performance, their killer feature being <b>huge caches for slow-changing HTML and assets across the world</b> so that even a casual Indian reader can load your homepage in one second. Moreover, if done right the <b>load on your origin servers will also be greatly reduced</b> with respect to not using caching layers.<br />
On the other hand, you can see the <b>complexity, observability and maintenance needs</b> that every additional layer introduces. When asking whether a CDN should do something or your application should do something, it's the same decision as for a database or a cloud service: how can you effectively store and update its configuration in multiple environments? Do you want to oursource that responsibility? How will you know when something's wrong? Do you feel comfortable writing stored procedures in a language you can't run on your laptop? All of these are architectural questions to go through when evaluating various CDNs, or no CDN.Giorgiohttp://www.blogger.com/profile/12689416577856305650noreply@blogger.com0tag:blogger.com,1999:blog-36547168.post-31729408828004620562018-12-06T17:01:00.000+01:002018-12-06T17:01:06.328+01:00Book review: The 5 Dysfunctions of a team<div class="separator" style="clear: both; text-align: left;">
<a href="https://en.wikipedia.org/wiki/The_Five_Dysfunctions_of_a_Team" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img alt="https://en.wikipedia.org/wiki/The_Five_Dysfunctions_of_a_Team" border="0" data-original-height="499" data-original-width="330" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBdy6OXuDAse9SlWn5LlJAw_tVMyQR66XgAxrqhhxHdv3sF6qjh5DsLKe_rb7qD_wYdfSM9QUx69OQHjp8vas-HzLDHQ2otRTvDSfzo2WZxFHzn3L8i1O6wipdWlp8l3fJkDI-Lg/s320/51QArmq8raL._SX328_BO1%252C204%252C203%252C200_.jpg" width="211" /></a></div>
This is a spur-of-the-moment review of <a href="https://en.wikipedia.org/wiki/The_Five_Dysfunctions_of_a_Team">The 5 Dysfunctions of a Team</a>, a business novel on team health that I've read today as part of the quarterly Professional Development Days I take as part of working at <a href="https://twitter.com/elife">eLife</a>.<br />
<br />
As a follow-up to my role evolution into <a href="https://www.giorgiosironi.com/2018/01/new-role-software-engineer-in-tools-and.html"><i>Software Engineer in Tools and Infrastructure</i></a>, I am looking again more into the people skills side of my job (as opposed to purely technical skills). I have done this cyclically during my career, as the coder hat becomes too restrictive and you have to pick up other tools to achieve improvement. In particular, I am working on eLife's Continuous Delivery platform and it is crucial to work with multiple product-oriented teams to have them adopt your latest Jenkins pipelines and Github reports.<br />
<h3>
Dysfunctions</h3>
Patrick Lencioni's model of team dysfunctions (or of blessed behaviors if you flip all definitions) is a pyramid where each dysfunction prevents the next level from being reached. It would be a disservice to how well and quickly this is got across in the novel to just try to list them here, but if I had to summarize this in a long paragraph it would look like:<br />
<blockquote class="tr_bq">
Building <b>trust</b> between team members allows <b>constructive conflicts</b>, which enable people to <b>commit</b> to action and hold each other <b>accountable</b> for what has been decided; all in the service of <b>results</b>. -- not really a quote</blockquote>
The dysfunctions are the flip side of these positive behaviors, for example <i>lack of trust</i> or <i>fear of conflict</i>. The definitions of some of these terms are more precise than what you find in many Agile and business coaching books; so don't dismiss <i>trust</i> as just a buzzword, for example.<br />
<h3>
Some context</h3>
The case being treated in the book, which comes from the author's management consulting firm, is that of a CEO turning around a team of executives. This makes for a somewhat more fascinating view of results as it's talking about a (fictional) company's IPO or eventual bankruptcy. Despite not being a clear parallel to a software development team, I do think this is applicable in every situation <b>where professionals are paid to work daily together</b>, with some caveats.<br />
In fact, I suspect the level of commitment to the job that you see in the book would be typical of either high stakes roles (executives) or a generally healthy organization that has already removed common dysfunctions at the individual level. If in your organization:<br />
<ul>
<li>people are <i>primarily</i> motivated by money</li>
<li>they look forward to 5 PM</li>
<li>they browse Facebook and Twitter for hours each day</li>
</ul>
then there are personal motives that have to be addressed before teams can start thinking about collective health. <br />
<h3>
Yes but, what can I do in practice?</h3>
After the narrative part, an addendum to the book contains a <b>self-administered test</b> to zoom in on which possible dysfunctions your team may exhibit at the moment. It continues with a series of <b>exercises and practices</b> that address these topics, with an estimation of their time commitments or how difficult they would be to run. I definitely look forward to anonymously try out the test with my technical team, out of curiosity for other views.<br />
<h3>
My conclusions</h3>
I think most of the dysfunctions are real patterns, that can only be exacerbated by the currently distorted market for software developers and CV-driven development. The last dysfunction, <i>Inattention to results</i>, is worth many books on its own on how to define those results as employees at all levels are known for optimizing around measurable goals to the detriment of, for example, long-term maintenance and quality.<br />
So don't start a crusade armed by this little book, but definitely keep this model in your toolbox and share it with your team to see if you can all identify areas for collective improvement; it is painfully obvious to say <b>you can't work on this alone!</b><br />
The author is certainly right when writing that groups of people truly working together can accomplish what any assembly of single individuals could never dream of doing.Giorgiohttp://www.blogger.com/profile/12689416577856305650noreply@blogger.com0tag:blogger.com,1999:blog-36547168.post-17495907890926029522018-09-16T10:59:00.000+02:002018-09-16T10:59:14.393+02:00Eris 0.11.0 is out<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj79ECIoIL2rLwVDnea5ylasGxOzmnvWLktbzsQeqKjQdtrNbxwrmk-Bx6jWRDrAtxTiVz7VuhDliBVyAzFk6gpIXwC29DXivNrB0ekpbbty6crXM2qayaK8a-_YKru3UVYbJFT4A/s1600/Artist%2527s_impression_dwarf_planet_Eris.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1066" data-original-width="1600" height="213" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj79ECIoIL2rLwVDnea5ylasGxOzmnvWLktbzsQeqKjQdtrNbxwrmk-Bx6jWRDrAtxTiVz7VuhDliBVyAzFk6gpIXwC29DXivNrB0ekpbbty6crXM2qayaK8a-_YKru3UVYbJFT4A/s320/Artist%2527s_impression_dwarf_planet_Eris.jpg" width="320" /></a></div>
<a href="http://www.giorgiosironi.com/search/label/eris">Eris</a> 0.11.0 has been freshly released, and I'll be listing here various contributions that the project has received that are included in this new version and in the previous one, 0.10.0, which didn't have an associated blog post.<br />
<br />
For a full list and links to the relevant pull requests and commits, see the <a href="https://github.com/giorgiosironi/eris/blob/master/ChangeLog.md">ChangeLog</a>.<br />
<h3>
0.10</h3>
<ul>
<li>The <i>Eris\Facade</i> class was introduced to allow usage outside of a PHPUnit context.</li>
<li>Official PHPUnit 7 support was introduced.</li>
<li>Fixed a corner case in <i>suchThat()</i></li>
</ul>
There are some small backward compatibility<i> </i>breaks with respect to 0.9; they regard unused features (or at least I thought) including <i>Generator::contains()</i>.<br />
<h3>
0.11</h3>
<ul>
<li><span style="font-weight: normal;">Official PHP 7.2 support</span></li>
<li><span style="font-weight: normal;">Annotations support for configuring behavior that is usually configured through methods: <i>@eris-method</i>, <i>@eris-shrink</i>, <i>@eris-ratio</i>, <i>@eris-repeat</i>, <i>@eris-duration</i> </span></li>
</ul>
<h3>
Some acknowledgements</h3>
Most of this work comes from <b><a href="https://github.com/giorgiosironi/eris/blob/master/CONTRIBUTORS.md">contributions</a></b>, not from me. I'd like to say a word of thanks to the people that have taken the time to use Eris in some of their projects but also to feed back a fix, an extension, or a substantial improvement.Giorgiohttp://www.blogger.com/profile/12689416577856305650noreply@blogger.com0tag:blogger.com,1999:blog-36547168.post-39566691947116443142018-01-21T13:41:00.005+01:002018-01-21T13:41:51.615+01:00Book review: Production-ready microservices<div class="separator" style="clear: both; text-align: center;">
<a href="https://www.amazon.co.uk/Production-Ready-Microservices-Standardized-Engineering-Organization/dp/1491965975" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img alt="https://www.amazon.co.uk/Production-Ready-Microservices-Standardized-Engineering-Organization/dp/1491965975" border="0" data-original-height="499" data-original-width="381" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNeP8P9duVGpbaTNu0FZWjKQSCX7y1n7ThKrOHXdULTygvtmrFgljtLCuay8eKdv51iBmbqgV0c62dllT2wALzVBN2yl33ihqgwiLERLfHjVCnCFxG5VZeqpsGJo5KEuIPje_Kdg/s320/production-ready-microservices.jpg" width="244" /></a></div>
<a href="https://www.amazon.co.uk/Production-Ready-Microservices-Standardized-Engineering-Organization/dp/1491965975">Production-Ready Microservices</a> is a short book about consistently practicing architecture and design over a fleet of microservices.<br />
In general, I think the principles described here apply very much to any service-oriented initiative, even more so if the services are coarse grained and hence require more maintenance than finely isolated ones.<br />
<h3>
Uber </h3>
The book extrapolates from the author's experience at Uber "standardizing over a thousand microservices". Given a few developers for each microservice team, that makes up 2000-3000 engineers from the total >10000 Uber employees (I wonder how many are lawyers). After WhatsApp's famous story of being acquired at 55 employees in total, that really highlights <b>the difficulty level of running a business and operations all over the physical world</b> (sending cars and drivers around in dozens of countries) with respect to a digital-only enterprise. We should remember this and many other directions of change the next time we hear a technology advocate saying how much the cost of his 2-people startup has been reduced by $technology.<br />
<br />
<h3>
The main message</h3>
You <b>should be <a href="https://martinfowler.com/bliki/MicroservicePrerequisites.html">this tall to use microservices</a></b>; this architecture doesn't necessarily fit every context; although integrating separate services of some size is becoming a standard after the API revolution (before that, it was <i>integrate through the database</i> which is arguable worse).<br />
You will encounter many different social and technical problems, such as:<br />
<ul>
<li><b>Inverse Conway's Law</b>, with the shape of the products defining the shape of the company. Although I found out this doesn't really apply at smaller scales as development teams can own more than one service and experience a successful decoupling between people and code.</li>
<li><b>Technical <a href="https://en.wikipedia.org/wiki/Urban_sprawl">sprawl</a></b>, where multiple languages, databases and other key choices spread without a consistent, central planning.</li>
<li><b>More ways to fail</b>: distributed and concurrent systems are more difficult to work with and to reason upon, plus the fact that there are more servers, containers or applications will simply multiply the failures you'll see.</li>
</ul>
There are lots of non-functional requirements like scaling each microservice and isolating it from the rest of the fleet; perhaps don't go too micro- if you don't have the resources to ensure an acceptable level in each service. Perhaps in your context the acceptable SLA for some particular service is low, because it's not change often or is only internally facing or is only used several times per day.<br />
One particular aspect of the <i>Consistency is important</i> lesson is that <b>the whole lifecycle of services should be considered</b>. Maintenance and even decomissioning are as important as producing new MVPs: but I've seen many times services being neglected, or being considered very easy to migrate away from one some new shiny substitute was available. In reality, it takes time and effort to keep services up and running, and to finally kill them when you have an alternative, as data and users are slowly migrated off from the old to the new platform.<br />
Lots of requirements are also overlooked but often turn out to be important as you increase your population of services: the scalability of a single endpoint, fault tolerance, even documentation (<a href="http://thinkrelevance.com/blog/2011/11/15/documenting-architecture-decisions">ADR</a> are the only form I trust very much right now in a fast-moving organization context.) Every single section of this book will make you think about it, but won't give much of an overview: you're better served by reading the <a href="http://www.giorgiosironi.com/2017/03/book-review-site-reliability-engineering.html">SRE book</a> for example.<br />
<h3>
Value for money</h3>
This book is a short read which gives you an overview of what microservices challenges you're likely to face down that rabbit hole; in particular, it focuses on a medium-to-large organization context. I'm <b>not sure this book is worth the price tag</b> however: 20 pounds for a Kindle edition of ~170 pages, where ~25 pages are glossary, index and lots of checklists.Giorgiohttp://www.blogger.com/profile/12689416577856305650noreply@blogger.com0tag:blogger.com,1999:blog-36547168.post-6681824340790826702018-01-03T12:53:00.000+01:002018-01-03T13:01:08.766+01:00Book review: Algorithms to live by<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8UyhPmTgx5ioWRV_dTb7NsrMpOtlD0OV-XC_bUsOENwn8GofekzHSdPRx57GdvslV9zyfO6p5JoTn7qN15ksOo0h3Lzho4lxCQ1pVEGdr2Pin1blEL-LEb9xUvazz6C7mZW-33A/s1600/algorithms.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img alt="https://www.amazon.com/Algorithms-Live-Computer-Science-Decisions/dp/1627790365" border="0" data-original-height="499" data-original-width="327" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8UyhPmTgx5ioWRV_dTb7NsrMpOtlD0OV-XC_bUsOENwn8GofekzHSdPRx57GdvslV9zyfO6p5JoTn7qN15ksOo0h3Lzho4lxCQ1pVEGdr2Pin1blEL-LEb9xUvazz6C7mZW-33A/s200/algorithms.jpg" width="130" /></a>
<a href="https://www.amazon.com/Algorithms-Live-Computer-Science-Decisions/dp/1627790365">Algorithms to Live By: The Computer Science of Human Decisions</a> is a book that puts together the domains of computer science and real life. The ensemble of topics being touched is wide. The book treats deterministic algorithms such as optimal sorting, but then moves on to more context-dependent strategies for caching and scheduling. The last chapter even get to model identification, (tractable and intractable-made-tractable) optimization problems, stochastic algorithms and game theory.<br />
<br />
All the while, computer science concepts are compared to conscious and unconscious human processes.
For example, caching and the memory hierarchy have <b>great parallels with </b>how<b> the human brain</b> recollects memories of recent events, and how we can augment our brain with external, slower supports like paper. Scheduling is useful not only to allocate processes on CPU cores, but also to make an explicit choice of strategy when prioritizing the tasks that you or your team face. Up to the more extreme examples of game theory and mechanism design, when the incentive system becomes more important than the individual agents (<a href="https://www.getbeyond.com/blog/manage-system-not-people/">manage the system, not the people</a> rings a bell?)<br />
<br />
If you like viewing the world throughout the lens of algorithms and see how the strategies of humans and computer compare with each other, I would strongly recommend this book as it will make for an entertaining read and some principles to take away for real life usage (I hope sorting socks will be easier now). Skip it if you have a very wide knowledge of computer science, operations research, Nash equilibria... but even if I was familiar with the technical part, I was missing <b>the connection to different domains or everyday, real world problems.</b>
I listened to the audiobook version, which lasts about 12 hours. You may find it easier to skim through some chapters if you are more (or less) interested in some topics. The problem with audiobooks is that I can't easily take notes, while highlighting on an e-book reader is quick and lets me recollect all important gotchas later into a text file.
Giorgiohttp://www.blogger.com/profile/12689416577856305650noreply@blogger.com0tag:blogger.com,1999:blog-36547168.post-28357828567243057632018-01-03T09:36:00.001+01:002018-12-06T17:01:27.317+01:00New role: Software Engineer in Tools and InfrastructureAfter working on eLife's testing and deployment infrastructure in 2016, in the last year my responsibilities in the technical team have shifted towards the domain of engineering productivity. Testing is one phase of the development process that is often a bottleneck, but there are many more areas like code reviews, monitoring and infrastructure itself (being it servers or services):<br />
<blockquote class="tr_bq">
In summary, the work done by the SETs naturally progressed from
supporting only product testing efforts to include supporting product
development efforts as well. Their role now encompassed a much broader
Engineering Productivity agenda. -- <i><a href="https://testing.googleblog.com/2016/03/from-qa-to-engineering-productivity.html">Ari Shamash on the Google Testing Blog</a></i></blockquote>
Moreover, the team starts from a high level of coverage and design on many projects, to the point that my focus has always been on the provisioning and automation of testing environments, and on large-scale end2end testing.<br />
<br />
What seems just a letter on a job title (from SET to SETI) is in fact an alignment of responsibilities so that I am not accidentally mistaken for "the QA guy" but always seen as a problem solver instead.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://en.wikipedia.org/wiki/Pulp_Fiction" style="margin-left: auto; margin-right: auto;"><img alt="https://en.wikipedia.org/wiki/Pulp_Fiction" border="0" data-original-height="400" data-original-width="400" height="320" src="https://memegenerator.net/img/instances/500x/81016744/im-winston-wolf-your-seti.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Solving problems and propagating the solution, so that you don't have to solve them over and over again</td></tr>
</tbody></table>
Roles are always an approximation in a team of <a href="https://blog.codinghorror.com/swiss-army-knife-or-generalizing-specialist/">generalizing specialist</a> that also distributes and collaborate on some roles such as that of architecture. But it's helpful in a cross-functional team to have someone dedicated to the task of productivity, whether it is reached through automation, tooling, or continuous improvement.Giorgiohttp://www.blogger.com/profile/12689416577856305650noreply@blogger.com0tag:blogger.com,1999:blog-36547168.post-74566281235194590372017-11-30T22:33:00.003+01:002017-12-01T11:25:36.687+01:00Book review: Building Microservices<div class="separator" style="clear: both; text-align: center;">
<a href="http://shop.oreilly.com/product/0636920033158.do" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img alt="http://shop.oreilly.com/product/0636920033158.do" border="0" data-original-height="656" data-original-width="500" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIkAorOCSsJaJdwmWbB60-OUVuemf1d7X3mO5lc2lCipISAVbvZW2GVRESkIq_5wajHPK9-VBKTfefQUCuqjz2QDHGgtZbzvLIbIsZ2Nz5VB611Ex9kA9BldwkZbgUw4HVYAmvqQ/s320/microservices.jpg" width="243" /></a></div>
<a href="http://shop.oreilly.com/product/0636920033158.do">Building Microservices: Designing Fine-Grained Systems</a> by Sam Newman is the seminal book on microservices as a concept. It was published at the start of 2015 (that's a long time ago in tech... or is it?), it's focused on high-level topics rather than implementation and hence has aged well.<br />
<br />
There are in fact several concepts, both at the methodology and the technical level, that the books does justice to. Here's what turned on the light bulb over my head.<br />
<h3>
Modeling</h3>
Modeling is important and a whiteboard discussion can save weeks of implementation down the line. Both me and Sam Newman are not the first to say this. Modelling in microservices, like in Domain-Driven Design, is all built around business capabilities and the shape of your organization (yes, Conway).<br />
<br />
<h3>
Styles of integration</h3>
Like for other design choices, especially at the architectural level, it's important to explicitly choose whether to go for a shared database (please don't), synchronous or asynchronous communication; orchestration through a Facade or coreography distributing responsiblities between services; explicit versioning the kinds of backward compatibility; pushing or pulling data from one physical location to another, and with which granularity of time and entity.<br />
There are also styles of isolation, not just of integration: code reuse is maturely described on a trade-off scale with decoupling. It feels like a pattern book in which these options are given a standard name for further discussion, and evaluated with respect to the contexts in which they work well.<br />
<br />
<h3>
Deployment</h3>
Should you go for virtual machines or containers? How do you map services to physical or virtual machines? The book couldn't possibly keep up with the rise of container orchestrators in the last couple of years, so it won't be a complete guide but could give you a sense of the problems that virtual machines create and that we are going to solve in this next generation. What will be the problems that containers create, and most of all how to solve them, is not in the scope of this book instead.<br />
<h3>
Testing</h3>
After the basics like a testing pyramid, I don't find myself in complete agreement with large scale testing strategies proposed here like consumer-driven contract testing. Yes, it works well enough if you can specify a formal contract that a service should adhere to, and test it in isolation in the implementer service. But the overhead of doing so, in a context in which we are supposed to create dozens if not hundreds of service, is very significant.<br />
At <a href="https://elifesciences.org/">eLife</a> we have relied on a <a href="https://github.com/elifesciences/api-raml">wide RESTful API specification</a>, each group of endpoints implemented by a different service. As such, the overhead is limited, and this is just a description for validating requests and responses rather than a full contract, as most of these services are read-only.<br />
All in all, <b>I find myself relying on the end2end beast</b> to get heterogeneous services, written in multiple languages, in different times by different people, to talk together reliably. I would spend a lot of energy in trying to square the circle of contracts, but I suppose they work well at an higher scale of traffic or on selected services.<br />
The <a href="https://github.com/elifesciences/elife-spectrum/">end2end tests</a> we use are limited in their scope, constrained by being at the top of the pyramid; they do not necessarily cover a full end2end scenario but rather a data path involving more than on service, often skipping the user interface. It helps that we have no Selenium-based testing <i>in the end2end layer</i>, as the user interface is fully accessible to a HTML parser and requires no Javascript.<br />
The problems that we encounter daily happen in production all the same: timeouts, dirty data, the automation challenge of turning on and off new nodes reliably, the race conditions that come from distributed executions. I'd rather not hide these problems but solve them, and I'm looking at containers instead to try to shrink the big picture and have a simpler end2end environment, easier to spin up and down, or to provide with clean databases.<br />
<h3>
Hidden gems</h3>
There are several hidden gems that would let you pick the brain of the author on a common problem cited in a chapter. For example, we have such a problem in integrating with <a href="https://civicrm.org/">a CRM that doesn't even support PHP 7</a> (what is it with CRMs and always being sources of technical debt?) There are some example patterns that you could apply in that situation, like hiding the CRM behind a specialized service that cleans up its API. Nothing miraculous, but a glimmer of hope for these desperate situations.<br />
<h3>
Conclusions</h3>
If you are going to work with microservices, or milliservices, or small enough services, this book is worth a read. If you are only having troubles with a particular area, such as testing or security, going through a single chapter will give you a big picture before you go in depth with further sources.<br />
Remember that this book is starting to become a bit dated, so you cannot take highly technical lessons from it (and I doubt books are a great tool for those in general in this fast-moving environment). Think about your context, learn the theory, and fill in the parts in which the map is blank (or erased) with what you are learning from the web in 2017 (soon to be 2018 - attempt to make this review valid for one more year).<br />
<br />Giorgiohttp://www.blogger.com/profile/12689416577856305650noreply@blogger.com2tag:blogger.com,1999:blog-36547168.post-36743350319825547622017-04-03T23:22:00.001+02:002017-04-03T23:27:46.721+02:00Pipeline Conf 2017<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAO3Z-zNGOzxRY2d9U7Ln0-3xa0Tg9vpMRZMRF9qNDVIbXqPoiGgbopfJ2gkL6KNQ-3iVPhpZKZlFpgAJ1GXibTELnBDkDAZ9fKW4YIy8v07EAM8xg7l3l3Ci-r3EUIWQg08-oUA/s1600/pipeline.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAO3Z-zNGOzxRY2d9U7Ln0-3xa0Tg9vpMRZMRF9qNDVIbXqPoiGgbopfJ2gkL6KNQ-3iVPhpZKZlFpgAJ1GXibTELnBDkDAZ9fKW4YIy8v07EAM8xg7l3l3Ci-r3EUIWQg08-oUA/s200/pipeline.png" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Not liquorice</td></tr>
</tbody></table>
<i>Post originally shared on <a href="https://elifesciences.org/">eLife</a>'s internal blog, but in the spirit of (green) <a href="https://en.wikipedia.org/wiki/Open_access">open access</a> here it is.</i><br />
<br />
Last month I have attended the <a href="https://web.pipelineconf.info/">PIPELINE conference</a> in London, 2017 edition. This event is a not-for-profit day dedicated to <a href="https://continuousdelivery.com/">Continuous Delivery</a>, the ability to get software changes into the hands of users, and to do so safely, quickly, and in a sustainable way. It is run by practitioners for practitioners, everyone on different sides of the spectrum like development, operations, testing, project management, or coaching. <br />
<br />
The day is run with parallel tracks, divided into time slots of 40-minute talks and breaks for discussions and, of course, some sponsor pitches. I have been picking talks from the various tracks depending on their utility to eLife's testing and deployment platform, since our tech team has been developing every new project with this approach for a good part of 2016.<br />
<br />
The conceptual model of Continuous Delivery and of eLife's implementation of it is not dissimilar to the scientific publishing process:<br />
<ul>
<li>there is some work performed into an isolated environment, such as a laboratory, but also someone's laptop;</li>
<li>which leads to a transferable piece of knowledge, such as a manuscript, but also a series of commits, roughly speaking some lines of code;</li>
<li>which is then submitted and peer reviewed. We do so through pull requests, which perform a series of automated tests to aid human reviewers inside the team; part of the review is also running the code to reproduce the same results on a machine which is not that original laptop.</li>
<li>after zero or more rounds of revisions, this work gets accepted and published...</li>
<li>which means integrating it with the rest of human knowledge, typesetting it, organizing citations and lots of metadata about the newly published paper. In software, the code has to be transformed into an efficient representation, or virtual machines have to be configured to work with it.</li>
<li>until, finally, this new knowledge (or feature) is in the hands of a real person, who can read a paper or enjoy the new search functionalities</li>
</ul>
Forgive me for the raw description of scientific work.<br />
<br />
In software, Continuous Delivery tries to automate and simplify this process to be able to perform it on microchanges multiple times per day. It aims for speed to be able to bring a new feature live in tens of minutes; it aims for safety to avoid breaking the users work on new changes; and to do all of this in a sustainable way, not to sacrifice tomorrow's ability to evolve for a quick gain today.<br />
<br />
Even without the last mile of real user traffic, the 2.0 software services have been running in production or production-like servers from the first weeks of their development. A common anti-pattern in software development is to say "It works on my machine" (imagine some saying "It reproduces the results, but only with my microscope"); what we strive for is "It works on multiple machines, that can be reliably created; if we break a feature we know within minutes and can go back the latest version known to work."<br />
<h3>
Dan North: opening keynote</h3>
Dan North started to experiment with Continuous Delivery in 2004, at a time when builds were taking 2 days and a half to run in a testing environment contended by multiple teams. He spoke about several concepts underpinning Continuous Delivery:<br />
<ul>
<li>conceptual consistency: the ability of different people to make similar decisions without coordination. It's an holy grail for scaling the efforts of an organization to more and more members and teams.</li>
<li>supportability: championing Mean Time To Repair over Mean Time Between Failures. The three important questions for facing a problem as what happened? Who is impacted? How do we fix it?</li>
<li>operability: what does it feel like to build your software? To deploy it? Test it? Releasing it? Monitor it? Support it? Essentially, developer experience in additio to user experience.</li>
</ul>
Operability is a challenge we have to face ourselves more and more as we move from running our own platform to provide open source software for other people to use. Not only reading an article has to be a beautiful experience, but publishing one should be.<br />
<h3>
John Clapham: team design for Continuous Delivery</h3>
This talk was more people-oriented, I agree with the speaker that engagement of workers is what really drive profits (or value in case of non-profits).<br />
Practically speaking:<br />
<ul>
<li>reward the right behaviors to promote the process you want;</li>
<li>ignore your job title as everyone's job is to deliver value together; </li>
<li>think small: it's easier to do 100 things 1% better than to do 1 thing 100% better (aka <a href="http://jamesclear.com/marginal-gains">aggregation of marginal gains</a>)</li>
</ul>
<h3>
Abraham Marin: architectural patterns for a more efficient pipeline</h3>
The target for a build is for it to take less than 10 minutes. The speaker promotes the fastest builds as the one you don't have to run, introducing a series of patterns (and the related architectural refactorings) to be executed, safely, to simplify your software components:<br />
<ul>
<li>decoupling an api from implementation: extracting an interface package to reduce the dependencies to a component to a dependency to an interface;</li>
<li>dividing responsibiliteis vertically or horizontally trying to isolate the most frequent changes and minimizing cross-cutting requirements;</li>
<li>transform a library into a service;</li>
<li>transform configuration into a service.</li>
</ul>
Some of these lessons are somewhat oriented to compiled languages, but not limited to them. My feeling is that even if you reduce compile times, you still have to test some components in integration, which is a large source of delay.<br />
<h3>
Steve Smith: measuring Continuous Delivery</h3>
How do you know whether a Continuous Delivery effort is going well? Or more pragmatically, which of your projects is in trouble?<br />
The abstract parameters to measure in pipelines are speed (throughput, cycle time) and stability. Each declines differently depending on the context.<br />
In deployment pipelines that go from a commit to a new version release in production, lead time and the interval of new deployments can be measured. But also failure rate (how many runs fail) and failure recovery time are interesting. In more general builds or test suites, execution time is a key parameter but a more holistic view includes interval (how frequent are builds executed). <br />
I liked some of these metrics so much that they are now in my OKRs for the new quarter. Simplistic quote: <a href="http://www.druckerinstitute.com/2013/07/measurement-myopia/">you can't manage what you can't measure</a>.<br />
<h3>
Alastair Smith: Test-driving your database</h3>
To continuously deploy new software versions, you need an iterative approach to evolve your database and the data within it. When you evolve, you also have to test every new schema change. Even in the context of stored procedures for maximum efficiency (and lock-in), Alastair showed how to write tests that can reliably run on multiple environments.<br />
<h3>
Rachel Laycock: closing keynote, Continuous Delivery at Scale</h3>
Rachel Laycock is the Head of technology for North America at Thoughtworks, the main sponsor of the conference. The keynote however had nothing to do with sales pitches. Here are some anti-patterns:<br />
<ul>
<li><i>"We have a DevOps team"</i> is an oxymoron, as that kind of team doesn't exist; what often happens is that the Ops team gets renamed.<i>"</i></li>
<li><i>Do we chose Kubernetes or Mesos?"</i> as in getting excited about the technology before you understand the problem to solve.</li>
</ul>
The "at scale" in the title pushes for seeing automation as a way to build a self-serving platform, where infrastructure people are not bottlenecks but enablers for the developers to build their own services.<br />
The best quote however really was <i>"yesterday's best practice becomes tomorrow's anti-pattern"</i>. What we look for is not to be the first to market but to have an adaptable advantage, a product that can evolve to meet new demands rather than being a dead end. Giorgiohttp://www.blogger.com/profile/12689416577856305650noreply@blogger.com0tag:blogger.com,1999:blog-36547168.post-20012945763074952952017-03-28T22:59:00.002+02:002017-03-28T22:59:51.344+02:00Book review: Site Reliability Engineering<div class="separator" style="clear: both; text-align: left;">
<a href="http://shop.oreilly.com/product/0636920041528.do" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img alt="http://shop.oreilly.com/product/0636920041528.do" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnKJDIuQdHSRO1m4OMp7u0Zcryb9OnfEwU8-9F-IDMgZMPtKqQMrargpVcyuxElo-H3iM-h6g7eP0cj34cCOT6MEKc9Daw1MkLBO4ULemYUe1HI3qb7IN_3J1mbedsPw_y54RtmA/s1600/sre.gif" /></a></div>
<br />
<blockquote class="tr_bq">
<i>The overwhelming majority of a software system’s lifespan is spent in use, not in design or implementation. So, why does conventional wisdom insist that software engineers focus primarily on the design and development of large-scale computing systems?</i> -- the book tagline</blockquote>
<a href="http://shop.oreilly.com/product/0636920041528.do">Site Reliability Engineering - How Google runs production systems</a> is a 2016 book about the ops side of Google services, and the set of principles and practices that underlie it.<br /><br />A Site Reliability Engineer is a software engineer (in the developer sense) that designs and implement systems that automated what would otherwise be done manually by system administrators. As such, SREs have a directive for employing a <b>minimum 50% of their time in development</b> rather than firefighting and maintenance of existing servers (named <i>toil</i> in the book).<br /><br />The book really is a collection of chapters, so you don't have to be scared by its size as you don't necessarily need to read it cover to cover. You can instead zoom in on the interesting chapters, being them monitoring, alerting, outage tracking or even management practices.<br />
<h3>
Principles, not just tools</h3>
I like books that reify and give names to concepts and principles, so that we can talk about those concepts and refer to them. This book gives precise definitions for the Google version of measures such as availability, Service Level Objectives and error budgets.<br />
This is abstraction at work: even if the examples being used show Google specific tools like Borgmon or Outalator, the solutions are described at an higher abstraction level that makes them reusable. When load balancing practices are made generic enough to satisfy all of Google services, you can bet that they are reusable enough to be somewhat applicable to your situation.<br />
<h3>
Caveat emptor</h3>
Chances are that <b>you're not Google</b>: the size of your software development effort is order of magnitudes smaller, and even when it is of a comparable size it's not easy to turn an established organization into Google (and probably not desirable nor necessary.)<br />However, you can understand that Google and the other giants like Facebook and Amazon <i>are</i> shaping our industry through their economically irresistible software and services. Angular vs React is actually Google vs Facebook; containers vs serverless is actually Kubernetes vs Lambda which is actually Google vs Amazon. The NoSQL revolution was probably started by the BigTable and Dynamo papers... and so on: when you deployed your first Docker container, Google engineers had already been using similar technology for 10 years; as such, they can teach you a lot at a relatively little cost through the pages of a book. And it's better to be informed on what may come to a cloud provider near you in the next years.<br />
<h3>
Conclusions</h3>
It took some time to get through this book, but it gives a realistic picture of running systems that undergo a large scale of traffic and changes at the same time. Besides the lessons you can directly get from it, I would recommend it to many system administrators and "devops guys" as a way to think more clearly about which forces and solutions are at play in their datacenters (or more likely, virtual private clouds).Giorgiohttp://www.blogger.com/profile/12689416577856305650noreply@blogger.com0tag:blogger.com,1999:blog-36547168.post-4952991671041539052017-03-12T20:13:00.002+01:002017-03-12T20:15:54.721+01:00Eris 0.9.0 is out<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiapjmw0yj8uAE8sdvprU9qmPRle6ao8Ql_7DuU13MZQPDNnAXUizoA0KHk2VHjLIORl5lqP3aZJMa3FYqe0zrUSIHnPKOpjVQf_u2YI9vJPeCnhDP7U34lUWKB1hEhnk24O4Na1A/s1600/Artist%2527s_impression_dwarf_planet_Eris.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="213" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiapjmw0yj8uAE8sdvprU9qmPRle6ao8Ql_7DuU13MZQPDNnAXUizoA0KHk2VHjLIORl5lqP3aZJMa3FYqe0zrUSIHnPKOpjVQf_u2YI9vJPeCnhDP7U34lUWKB1hEhnk24O4Na1A/s320/Artist%2527s_impression_dwarf_planet_Eris.jpg" width="320" /></a></div>
In 2016 <a href="http://www.giorgiosironi.com/2016/04/next-stop-cambridge.html">I moved to another country</a> and as a result of this change I didn't had much time to develop <a href="https://github.com/giorgiosironi/eris">Eris</a> further. Thankfully Eris 0.8 was already pretty much stable, and in this last period I could pick up development again.<br />
<h3>
What's new?</h3>
The <a href="https://github.com/giorgiosironi/eris/blob/master/ChangeLog.md">ChangeLog for 0.9</a> contains one big new feature, <b>multiple shrinking</b>. While minimization of failing test cases is usually performed with a single linear search, multiple shrinking features a series of different options for shrinking a value.<br />
For example, the integer <i>1234</i> was usually shrunk to <i>1233, 1232, 1231</i> and so on. With multiple shrinking, there are a series of options to explore that make the search logarithmic, such as <i>617, 925, 1080, 1157, 1195, 1214, 1224, 1129, and 1231</i>. If the simplest failing values is below 617 for example, at least (1234-617) runs of the test will be skipped by this optimization, just in the first step.<br />
This feature is the equivalent of QuickCheck's (and other property-based testing libraries') <a href="https://hackage.haskell.org/package/QuickCheck-2.9.2/docs/Test-QuickCheck-Property.html#g:4">Rose Trees</a>, but implemented here with an object-oriented approach that makes use of `GeneratedValueSingle` and `GeneratedValueOptions` as part of a Composite pattern.<br />
<br />
This release also features support for the latest versions of basic dependencies:<br />
<ul>
<li><b>PHPUnit 6.x</b> is now supported</li>
<li><b>PHP 7.1</b> is officially supported (I expect there were mostly no issues in previous releases, but not the test suite fully passes.)</li>
</ul>
Several small bugs were fixed as part of feedback from projects using Eris:<br />
<ul>
<li>the pos() and neg() generators should not shrink to 0.</li>
<li>float generation should never divide by 0.</li>
<li>shrinking of dates fell into a case of wrong operator precedence.</li>
<li>reproducible PHPUnit commands were not escaped correctly in presence of namespaced classes.</li>
</ul>
A few backward compatibility fixes were necessary to make room for new features:<br />
<ul>
<li><i>minimumEvaluationRatio</i> is now a method to be called, not a private field.</li>
<li><i>GeneratedValue</i> is now an interface and not a class. This is supposed to be an internal value: project code should never depend on it and it should build custom generators with <a href="http://eris.readthedocs.io/en/latest/generators/composite.html#map">map() and other composite generators</a> rather than implementing the <i>Generator</i> interface, which is much more complex.</li>
<li>the <i>Listener::endPropertyVerification()</i> method now takes the additional parameters <i>$iterations</i> and the optional <i>$exception</i>. When creating listeners should always subclass <a href="https://github.com/giorgiosironi/eris/blob/master/src/Listener/EmptyListener.php">EmptyListener</a> in order not to have to modify the not implemented methods, which will be inherited.</li>
</ul>
<h3>
What's next?</h3>
My Trello board says:<br />
<ul>
<li>still <b>decoupling from PHPUnit</b>, for usage in scripts, mainly as a programmable source of randomness.</li>
<li><b>more advanced Generators</b> for finite state machines and in general a more <b>stateful</b> approach, for testing stateful systems.</li>
<li><b>faster feedback for developers</b>, like having the option to run fewer test cases in a development environment but the full set in Continuous Integration.</li>
</ul>
I'm considering opening up the Trello board for public read-only visibility, as there's nothing sensible in there, but potential value in transparency and feedback from the random people encountering the project for the first time.<br />
<br />
As always, if you feel there is a glaring feature missing in Eris, feel free to request it on the <a href="https://github.com/giorgiosironi/eris/issues">Github's project issues</a>.Giorgiohttp://www.blogger.com/profile/12689416577856305650noreply@blogger.com0tag:blogger.com,1999:blog-36547168.post-54908042100237580902017-02-12T17:36:00.001+01:002017-02-12T17:36:32.318+01:00Book review: Fifty quick ideas to improve your tests<div class="separator tr_bq" style="clear: both; text-align: left;">
<a href="https://leanpub.com/50quickideas-tests" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img alt="https://leanpub.com/50quickideas-tests" border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKShBl9k5VTHQxKnCt6cT5Ymu0qg4Yy2Dmwtczrs1JUDoGYUNeRCUgdIeaIseXa3HVBPAXqJd13Gv5p-qSUiHsThGJ7bGxLdYRljaNMAszEDnvzHxgwLXkO0-O_QTStMZnB6C6Jw/s320/hero.jpeg" width="219" /></a></div>
<div class="tr_bq">
<a href="https://leanpub.com/50quickideas-tests">Fifty quick ideas to improve your tests</a> is, well, a series of fifty quick ideas that you can implement on some of your automated test suites to improve their value or lower their creation or maintenance costs.</div>
<br />
These ideas are pattern-like in which they are mostly self-contained and often independent from each other. They are distilled from real world scenarios that the authors (David Evans, Tom Roden and Gojko Adzic) have encountered in their work.<br />
<br />
This format helps a lot readability, as ideas are organized into themes, giving you the ability to focus on the area you want to improve and to quickly skip the ideas that do not make sense in your context or you find impractical or not worth the effort. For the same reasons I enjoyed the book this one is a sequel to, <a href="https://gist.github.com/giorgiosironi/84fe5d62acb7c0740891">Fifty quick ideas to improve your user stories</a>. Moreover, both were published on Leanpub so you have the ability to adjust the price to the value you think you'll get out of them; and despite Leanpub's large collection of unfinished books, this one is 100% complete and ready to read without the hassle of having to update to a new version later (who really does that?)<br />
<br />
Some selected quotes follow, highlighted phrases mine.<br />
<blockquote>
Something one person considers critical might not even register on the scale of importance for someone from a different group.
</blockquote>
<blockquote>
Drawing parallels between the different <a href="https://en.wikipedia.org/wiki/Maslow's_hierarchy_of_needs">levels of needs</a>, we can create <b>a pyramid of software quality levels</b>: Does it work at all? What are the key features, key technical qualities? Does it work well? What are the key performance, security, scalability aspects? Is it usable? What are the key usability scenarios? Is it useful? What production metrics will show that it is used in real work? Is it successful?
</blockquote>
<blockquote>
In order to paint the big picture quickly, we often kick things off with a ten-minute session on identifying things that should always happen or that should never be allowed. This helps to set the stage for more interesting questions quickly, because absolute statements such as ‘should always’ and ‘should never’ urge people to come up with exceptions.
</blockquote>
<blockquote class="tr_bq">
Finally, <b>when an aspect of quality is quantified</b>, teams can better evaluate the cost and difficulty of measuring. For example, we quantified a key usability scenario for MindMup as ‘Novice users will be able to create and share simple mind maps in under five minutes’. Once the definition was that clear, it turned out not to be so impossible or expensive to measure it.</blockquote>
<blockquote>
Avoid checklists that are used to tick items off as people work (Gawande calls those Read-Do lists). Instead, aim to create lists that allow people to work, then pause and review to see if they missed anything (‘Do-Confirm’ in Gawande’s terminology).
</blockquote>
<blockquote class="tr_bq">
A major problem causing overly complex examples is the <b>misunderstanding that testing can somehow be completely replaced by a set of carefully chosen examples</b>. For most situations we’ve seen, this is a false premise. Checking examples can be a good start, but there are still plenty of other types of tests that are useful to do. Don’t aim to fully replace testing with examples in user stories – aim to create a good shared understanding, and give people the context to do a good job.
</blockquote>
<blockquote class="tr_bq">
Waiting for an event instead of waiting for a period of time is the preferred way of testing asynchronous systems
</blockquote>
<blockquote class="tr_bq">
The sequence is important: ‘Given’ comes before ‘When’, and ‘When’ comes before ‘Then’. Those clauses should not be mixed. All parameters should be specified with ‘Given’ clauses, the action under test should be specified with the ‘When’ clause, and all expected outcomes should be listed with ‘Then’ clauses. Each scenario should ideally have only one ‘When’ clause that clearly points to the purpose of the test. </blockquote>
<blockquote class="tr_bq">
<b>Difficult testing is a symptom, not a problem</b>. When it is difficult for a team to know if they have a complete picture during testing, then it will also be difficult for it to know if they have a complete picture during development, or during a discussion on requirements. It’s unfortunate that this complexity sometimes clearly shows for the first time during testing, but the cause of the problem is somewhere else.
</blockquote>
<blockquote class="tr_bq">
Although it’s intuitive to think about writing documents from top to bottom, with tests it is actually better to start from the bottom. Write the outputs, the assertions and the checks first. Then try to explain how to get to those outputs. [...] Starting from the outputs makes it highly unlikely that a test will try to check many different things at once,
</blockquote>
<blockquote class="tr_bq">
Technical testing normally requires the use of technical concepts, such as nested structures, recursive pointers and unique identifiers. Such things can be easily described in programming languages, but are not easy to put into the kind of form that non-technical testing tools require.
</blockquote>
<blockquote class="tr_bq">
For each test, <b>ask who needs to resolve a potential failure in the future</b>. A failing test might signal a bug (test is right, implementation is wrong), or it might be an unforeseen impact (implementation is right, test is no longer right). If all the people who need to be make the decision work with programming language tools, the test goes in the technical group. If it would not be a technical but a business domain decision, it goes into the business group.
</blockquote>
<blockquote class="tr_bq">
Manual tests suffer from the problem of capacity. Compared to a machine, a person can do very little in the same amount of time. This is why manual tests tend to optimise human time [...] Since automated tests are designed for unattended execution, it’s critically important that failures can be investigated quickly. [...] To save time in execution, it’s common for a single manual test to check lots of different things or to address several risks. </blockquote>
<blockquote class="tr_bq">
Whenever a test needs to access external resources, in particular if they are created asynchronously or transferred across networks, ensure that the resources are hidden until they are fully complete.
</blockquote>
<blockquote class="tr_bq">
Time-based waiting <i>[sleep() instead of polling or waiting for an event in tests]</i> is the equivalent of going out on the street and <b>waiting for half an hour in the rain</b> upon receiving a thirty-minute delivery estimate for pizza, only to discover that the pizza guy came along a different road and dropped it off 10 minutes ago.
</blockquote>
<blockquote class="tr_bq">
Instead of just accepting that something is difficult to test and ignoring it, investigate whether you can measure it in the production environment.
</blockquote>
<blockquote class="tr_bq">
Test coverage is a negative metric: it measures how bad something is, not how good it is.
</blockquote>
<blockquote class="tr_bq">
There are just two moments when an automated test provides useful information: the first time it passes and when it subsequently fails.
</blockquote>
<blockquote class="tr_bq">
It’s far better to <b>optimise tests for reading than for writing</b>. Spending half an hour more writing a test will save days of investigation later on. [...] In business-oriented tests, if you need to compromise either ease of maintenance or readability, keep readability.
</blockquote>
Giorgiohttp://www.blogger.com/profile/12689416577856305650noreply@blogger.com0tag:blogger.com,1999:blog-36547168.post-81495005549314032642016-12-29T17:44:00.001+01:002016-12-29T17:45:48.987+01:00Book review: The Power of Habit<a href="http://charlesduhigg.com/the-power-of-habit/" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img alt="http://charlesduhigg.com/the-power-of-habit/" border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwQxU7gFEc00-iZH0RKGBcvczwQ6dPeGG8hrPwhiZF48mru0jbNDLvPlIddDm0xKS-1_wgqKU-NNEFrmmG34SHCRphwYRUEXxAU72Nbkkjr4wXMwftL-IjG3_bKMxQzznSl8Rq2w/s320/Power-of-Habit-TP_nospine1.jpg" width="249" /></a>Charles Duhigg, a New York Times reporter, <a href="http://charlesduhigg.com/the-power-of-habit/">collects stories of building and breaking habits</a>, supporting the thesis that habits form an important part of our lives and that they can make a big difference for better or worse. Both in the case of positive training or learning habits, or in the case of addictions, repeated behavior influences our energy levels, our free time and in the end many of our long-term results at work and in life (we will all go the gym in the new year, right?)<br />
<br />
The <b>storytelling</b> style of the book may smell like <b>anecdotal evidence</b>, but it keeps the reader intruigued and entertained long enough the get its message across, without delving into fictional stories. Take the science expressed here with a grain of salt (like you would with Malcom Gladwell): all experiments are real but they may have been cherry-picked to prove a point.<br />
<br />
The key take away for me was to think about our habits, try to influence them to stop or reinforce them depending on our <b>second-order desires</b>; for example with the <b>cue-routine-reward</b> framework proposed in the book but ultimately with whatever works for you as habit building and destroying must be very context-specific. Another concept that we find reasonable is <a href="https://en.wikipedia.org/wiki/Ego_depletion">ego depletion</a> (willpower as a finite resource that must be renewed), but the jury is still out on whether it is a confirmed and sizable effect, as meta-analysis of hundreds of studies do not agree yet (for good reasons).<br />
<br />
From exercising to learning, or from quitting smoking to a Facebook addiction, this self-reflection can have a large impact over our lives. Maybe it should be an habit?<br />
<br />
Selected quotes from the book follow:<br />
<blockquote>
This process within our brains is a three-step loop. First, there is a cue, a trigger that tells your brain to go into automatic mode and which habit to use. Then there is the routine, which can be physical or mental or emotional. Finally, there is a reward, which helps your brain figure out if this particular loop is worth remembering for the future [...] Every McDonald’s, for instance, looks the same—the company deliberately tries to standardize stores’ architecture and what employees say to customers, so everything is a consistent cue to trigger eating routines.</blockquote>
<blockquote>
“Even if you give people better habits, it doesn’t repair why they started drinking in the first place. Eventually they’ll have a bad day, and no new routine is going to make everything seem okay. What can make a difference is believing that they can cope with that stress without alcohol.”</blockquote>
<blockquote>
Where should a would-be habit master start? Understanding keystone habits holds the answer to that question: The habits that matter most are the ones that, when they start to shift, dislodge and remake other patterns.</blockquote>
<blockquote>
“Small wins are a steady application of a small advantage,” one Cornell professor wrote in 1984. “Once a small win has been accomplished, forces are set in motion that favor another small win.” Small wins fuel transformative changes by leveraging tiny advantages into patterns that convince people that bigger achievements are within reach.</blockquote>
<blockquote>
“Sometimes it looks like people with great self-control aren’t working hard—but that’s because they’ve made it automatic”</blockquote>
<blockquote>
“By making people use a little bit of their willpower to ignore cookies, we had put them into a state where they were willing to quit much faster,” Muraven told me. “There’s been more than two hundred studies on this idea since then, and they’ve all found the same thing. Willpower isn’t just a skill. It’s a muscle, like the muscles in your arms or legs, and it gets tired as it works harder, so there’s less power left over for other things.”</blockquote>
<blockquote>
As people strengthened their willpower muscles in one part of their lives—in the gym, or a money management program—that strength spilled over into what they ate or how hard they worked. Once willpower became stronger, it touched everything.</blockquote>
Giorgiohttp://www.blogger.com/profile/12689416577856305650noreply@blogger.com0tag:blogger.com,1999:blog-36547168.post-76516417331873938952016-11-14T00:12:00.000+01:002016-11-14T00:12:11.258+01:00How to run Continuous Integration on EC2 without breaking the bankI live in a world increasingly made up of services (sometimes micro-), that collaborate to produce some visible behavior. This decomposition of software into (hopefully) loosely coupled components has implications for production infrastructure concern such as how many servers or containers you need to run. <br />
Consequently, it impacts also testing infrastructure as it tries to be an environment as close as possible to production, in order to reliably discover bugs and replicate them in a controlled environment without impacting real users.<br />
In particular I like to have a <b>ci</b> testing environment where each service can be tested in its own virtual machine (or container for some); and each node is totally isolated from the other services. In addition to that, I also like to have and <b>end-to-end</b> testing environment where services talk to each other as they would in production, and where we can run long and complex acceptance tests. <br />
This end-to-end environment usually is a perfect copy of production with respect to the technologies being used (e.g. load balancers like HAProx or AWS ELB are in place, in the same way of production, even if there is no test that directly targets their existence); the number of nodes per service is however reduced from N to 2, as in computing there are only 0, 1 and N equivalence categories.<br />
<br />
In the likely case that you're using cloud computing infrastructure to manage this 2x or 3x volume of servers with respect to the production infrastructure, your costs are also by default going to double or triple. One option to try and optimize this is to throw away everything and start deploying containers, as they could share the same underlying virtual machines as production while preserving isolation and reproducibility. On an existing architecture made up of AWS EC2 nodes however, optimization can takes us far without requesting to rewrite all the DevOps(TM) work of the last two years.<br />
<h3>
Phase 1: expand</h3>
As I've been explaining, EC2 instances replicating production environments can expand until they bring the total number of EC2 nodes to three times the original number. Some project just have an single EC2 node, while others have multiple nodes that have to be at least 2 in the latest testing environment before production. Moreover, the time the tests take to run on these instances is inversely correlated with how powerful they are in CPU and I/O terms, so you pay good money for every speed improvement you want to get on those 20- or 60-minute suites.<br />
In my current role at <a href="https://elifesciences.org/">eLife</a>, we initially got to more than 20 EC2 instances for the testing environments. This was beneficial from a quality and correctness point of view, as we could then run tests on all mainline branches before they go to production, but also on pull requests, giving timely feedback on the proposed changes without requiring developers to run everything on their machines (that should be an option, not an imperative.)<br />
<h3>
Phase 2: optimize</h3>
The AWS EC2 pricing model works by putting nodes into a <i>running</i> state when you launch them, and by pre-billing one hour of usage every 60 minutes. Therefore, booting existing instances or creating from scratch is going to incur at least a 1-hour cost for each of these events:<br />
<ul>
<li>at boot, 1 hour is billed</li>
<li>at 60:00 from boot a new hour is billed,</li>
<li>at 120:00 from boot a new hour is billed, and so on</li>
</ul>
All the EC2 nodes that we have however, use EBS disks for their root volumes. EBS is the block remote storage provided by AWS, and while some generations of instances use local instance storage for their / partition, EBS makes a lot of sense for that partition as it gives you the ability to <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-lifecycle.html">starting and stopping</a> instances without losing state in between; essentially, it gives you the ability to shutdown and reboot instances when you need without paying EC2 bills for the time in which they are stopped. The only billing being performed is for the EBS storage space, which means AWS has some hard disks in its data centers that has to preserve your instances files, but it allocates new CPU and RAM resources as a virtual machine only when you start the EC2 instance. Therefore, an EBS-backed EC2 instance on your AWS console does not always correspond to a physical place, but it's really virtual as it can be stopped and started many times, moving between racks in the same availability zone while keeping the same data (persisted to multiple disks in other racks) and even private IP address.<br />
Since the process of allocating a new virtual machine and reconfiguring networks to connect everything together has some overhead, this reflects in the boot time necessary to start an EC2 instance, which can be several seconds to be added to the standard boot time for the operating system. This is also reflected in the pricing model, which makes it a <b>bad idea</b> to launch a new EC2 instance for every test suite you need to run: as long as your test suite takes less than 1 hour, you are already paying for a full hour of resources and you would be throwing away. Running 6 builds in an hour would make you pay for 6 hours, which is not what you want.<br />
<h3>
Phase 2.1: stop them manually</h3>
A first optimization that can be performed manually is to stop and start these instances from the console. You would usually stop them at some hour of the day or evening and then start them again as the first thing in the morning.<br />
Of course, there is a good potential for automating this as AWS provides many ways to access EC2 with its API and all the SDKs that build on top of it, available for many different programming languages. You can easily build some commands that query the EC2 API looking for an instance basing on its tags, and then issue commands for starting and stopping it. In general, this is almost transparent for the CloudFormation templates that you are surely using for launching these instances.<br />
The first time you start and stop an instance, there are a few problems thay may come up.<br />
The first problem is that of <b>ephemeral storage</b>: as I wrote before, you have to make sure the root volume of the instance and any data you want to persist are EBS-backed and not local instance storage.<br />
The second problem is that of <b>public IP addresses</b>. While private IP addresses inside a VPC stay the same after a stop and start commands, public IP addresses are a scarce resource and are only allocated to it when the instance is running. Therefore, if you had a DNS pointing to it, it has to be updated after the boot, whether it was manually created or part of the CloudFormation template. Default DNS entries have the form <i>ec2-public-ip-address.compute-1.amazonaws.com</i> which depends on the public ip address and hence does not provide a good indirection.<br />
The third problem is that of <b>long-running processes</b> managed by SysV/Upstart/Systemd: the daemons of servers like Apache, Nginx or MySQL are usually configured to restart upon boot, but if you have written your own deamons or Python/PHP long running processes and are starting them through /etc/init or /etc/init.d configuration, it pays to check everything is in its place again after boot.<br />
The last problem I have found at this level (manual restarts) is about <b>files in /run and /var/run</b>, which are temporary directory used by deamons to place locks and other transient files like a pidfile indicating an instance of that program is running. If you have folders in /run or /var/run, those folder will have to be recreated. Systemd provides the <i>tmpfiles.d</i> option which automatically creates a hierarchy of files, but it's usually just easier (and portable) to have the daemons create their folders (php-fpm does that) or if they are not able to do that, not placing them in /var/run/some_folder_that_will_stop_existing but in /var/run or even /tmp without subfolders.<br />
<h3>
Phase 2.2: start them on demand</h3>
Instead of manually starting EC2 instance or to automate their stopping and starting as a periodical task, you can also start them on-demand as needed by the various builds that need to be run. So whenever project <i>x</i> needs to build a new commit on master or a pull request, you will start the <i>x--ci</i> EC2 instance.<br />
In this case, however, there is a larger potential for race conditions as you may try to run a deploy or any command on an instance before it's actually ready to be used. Therefore, we wrote <a href="https://github.com/elifesciences/builder/blob/master/src/buildercore/lifecycle.py">some automation code</a> that waits for several events before letting a build proceed:<br />
<ol>
<li>the instance must have gone from the <i>pending</i> to the <i>running</i> state on the EC2 API. This hopefully means AWS has found a CPU and other resources to assign to it.</li>
<li>the instance must be accessible through <b>SSH</b>.</li>
<li>through SSH, we monitor that the file <b>/var/lib/cloud/instance/boot-finished</b> has appeared. This file will appear at each boot when all daemons have been started, as art of the standard cloud-init package.</li>
</ol>
Once the instance has gone through all these phases, you can SSH into it and run whatever you want.<br />
<br />
<h3>
Phase 2.3: stop them when it's more efficient</h3>
Once you have transitioned from starting instances in the last responsible moment, you can do the same for stopping them instead of just wait for the end of the day to shutdown everything. <br />
We now have a periodical job, running every 2 minutes, that takes a list of servers to stop. In parallel, it performs <a href="https://github.com/elifesciences/builder/blob/master/src/buildercore/lifecycle.py#L55">the following routine</a> for each of the EC2 instances:<br />
<ul>
<li>checks if the server has been running for an amount of time between <i>h:55:00</i> and <i>h:59:59</i> minutes, where <i>h</i> is some number of hours.</li>
<li>if the condition is true, stop the instance before we incur in a new hour being billed.</li>
<li>otherwise, leave the instance running: you already paid for this hour so it makes no harm to do so, as the instance can be used to run new builds at no cost.</li>
</ul>
Therefore, when developers open a dozen pull requests on the same project, only the first starts the necessary instances to run the tests; the other ones are queued behind that and will get access to the same instance, one at a time.<br />
<h3>
Bonus: Jenkins locks</h3>
Starting and stopping instances periodically would be otherwise dangerous if there wasn't a mechanism for mutual exclusion between builds and lifecycle operations like starting and stopping. Not only you don't want to run builds for the same project on the same instance if they interfere with each other, but you definitely don't want an instance to be shutdown while a build is still running.<br />
Therefore, we wrap both these lifecycle operations and builds in locks for resource, using <a href="https://wiki.jenkins-ci.org/display/JENKINS/Lockable+Resources+Plugin">Jenkins Lockable Resources</a> plugin. If the periodical stopping task tries to stop an instance where the build is running, it will have to wait to acquire the <a href="https://github.com/elifesciences/elife-jenkins-workflow-libs/blob/master/vars/builderStopAll.groovy#L6">lock</a>. This ensures that machines that see many builds do not get easily stopped, while other ones that are idle will be stopped at the end of their already paid hour.<br />
<h3>
Conclusions</h3>
Cloud computing is meant to improve the efficiency with which we allocate resources from someone else's data centers: you pay for what you use. Therefore, with a little persistence provided by EBS volumes you can efficiently pay for the hours that your builds require, and not for keeping idle EC2 instances running every day of the year. Of course, you'll need some tweaking of your automation tools to easily start and stop instances; but it is a surefire investment that can usually save more than half of the cost of your testing infrastructure by putting it at rest in weekend and non-working hours.Giorgiohttp://www.blogger.com/profile/12689416577856305650noreply@blogger.com0tag:blogger.com,1999:blog-36547168.post-35701350340482853692016-10-02T19:20:00.000+02:002016-10-02T19:23:39.846+02:00Deep learning: an introduction for the layperson<div class="separator" style="clear: both; text-align: center;">
<a href="https://giorgiosironi.github.io/talks/deep_learning/#/"><img alt="https://giorgiosironi.github.io/talks/deep_learning/#/" border="0" height="213" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEizR-7YZtCpjG_THrB3wU1DSn8FRi1tmtZLFfe3x7Sz2xWiwXw78eOO8wwLrNYyqHcQazQMSC-Te8qJ3vU8AJvDJiOgBOc384SRLsFhhzOBa1vIYC8iYheuf7tiS6YcJ0vzwXbRmg/s320/alphago.jpg" width="320" /></a></div>
Deep learning is one of the buzzwords of 2016, promising to revolutionize the world possibly without <a href="https://en.wikipedia.org/wiki/Skynet_(Terminator)">Skynet</a> gaining self-awareness.<br />
<br />
Last week I held an internal talk at <a href="https://elifesciences.org/">eLife</a><a href="https://www.blogger.com/null"> Sciences</a> to introduce colleagues from both the software and scientific backgrounds to the concept.<br />
<br />
Here are the <a href="https://giorgiosironi.github.io/talks/deep_learning/#/">slides</a>, complete with notes (accessible through a popup by pressing S) that explain what the diagrams and other figures mean.<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br />Giorgiohttp://www.blogger.com/profile/12689416577856305650noreply@blogger.com0tag:blogger.com,1999:blog-36547168.post-56667371206466149902016-05-22T21:35:00.002+02:002017-03-12T20:15:09.868+01:00Eris 0.8.0 is out<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhMTVULNyCwTQ4-c-IvF8o4DHFhyphenhyphenHMt3JEOVua1CO6R3gRDmPS8sirOn0zG6_GpfAxlIjnGTS6lSi_7F-Ah16CCD6RJbVZTXXxnMlt-WSGEAHzbzu_J7jk5eDzC9KjDN3yUlVOSrg/s1600/Artist%2527s_impression_dwarf_planet_Eris.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="213" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhMTVULNyCwTQ4-c-IvF8o4DHFhyphenhyphenHMt3JEOVua1CO6R3gRDmPS8sirOn0zG6_GpfAxlIjnGTS6lSi_7F-Ah16CCD6RJbVZTXXxnMlt-WSGEAHzbzu_J7jk5eDzC9KjDN3yUlVOSrg/s320/Artist%2527s_impression_dwarf_planet_Eris.jpg" width="320" /></a></div>
In the period before my move to Cambridge I got some time to work on <a href="https://packagist.org/packages/giorgiosironi/eris">Eris</a>, and to use it to <a href="http://www.giorgiosironi.com/2016/03/on-property-based-testing-highly.html">test the guts of Onebip's infrastructure</a>. Lots of new features are now incorporated in the 0.8.0 version, along with a modernization of PHP standards compliance carried out by <a href="https://twitter.com/localheinz">@localheinz</a>.<br />
<h3>
What's new?</h3>
Here's the most important news, a selection from the <a href="https://github.com/giorgiosironi/eris/blob/master/ChangeLog.md">ChangeLog</a>:<br />
<ul>
<li>The<i> bind</i> Generator allows to use the random output of a Generator to build another Generator.</li>
<li>Optionally logging generations with `hook(Listener\log($filename))`.</li>
<li><i>disableShrinking()</i> option.</li>
<li><i>limitTo()</i> accepts a DateInterval to stop tests at a predefined maximum time.</li>
<li>Configurability of randomness: choice between rand, mt_rand, and a pure PHP Mersenne Twister.</li>
<li>The <i>suchThat</i> Generator accepts PHPUnit constraints like `when()`.</li>
</ul>
Some bugs and annoyances were fixed: <br />
<ul>
<li>No warnings on PHP 7 anymore.</li>
<li>Fixed bug of size not being fully explored due to slow growth.</li>
<li>Switched to PSR-2 coding standards and PSR-4 autoloading.</li>
</ul>
And there were some backward compatibility breaks (we are in 0.x after all):<br />
<ul>
<li>The <i>frequency</i> generator only accepts variadics args, not an array anymore.</li>
<li>Removed <i>strictlyPos</i> and <i>strictlyNeg</i> Generators as duplicated of <i>pos</i> and <i>neg</i> ones. </li>
<li>Removed <i>andAlso</i>, <i>theCondition</i>, <i>andTheCondition</i>, <i>implies</i>, <i>imply</i> aliases which expand the surface area of the API for no good reason. Added <i>and</i> for multiple preconditions. </li>
</ul>
Eris is now quite extensible with custom Generators for new types of data; custom Listeners to know what's going on; and even different sources of randomness to tune repeatability and performance.<br />
I believe what's very important about this release is the <b>release of technical <a href="http://eris.readthedocs.io/en/latest/">documentation</a></b>. This is not a list of APIs generated by parsing the code, but is a full manual of Eris features, which will be kept up-to-date religiously <a href="https://github.com/giorgiosironi/eris/tree/master/docs">in the repository itself</a> and rebuilt automatically at each commit.<br />
<h3>
What's next?</h3>
My Trello board says:<br />
<ul>
<li><b>decoupling</b> from PHPUnit: it should be possible to run Eris also with PHPSpec (already possible but not as robustly as it can be) or in scripts.</li>
<li>Multiple possibilities for <b>shrinking</b>, borrowing from test.check rose trees. This feature may speed up the shrinking process and make it totally deterministic.</li>
<li>A few more advanced <b>Generators</b>: for example testing Finite State Machines.</li>
</ul>
If you are using Eris and wanna give feedback, feel free to <a href="https://github.com/giorgiosironi/eris/issues">open a Github issue</a> to discuss. Giorgiohttp://www.blogger.com/profile/12689416577856305650noreply@blogger.com0tag:blogger.com,1999:blog-36547168.post-27919428857711979952016-04-19T12:03:00.003+02:002016-04-19T12:04:05.980+02:00Next stop: Cambridge<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijr1lSmXvNpNUxLN3uCpk-TkqaEYiBTkZ3W7xRwAmCVgiJ-hP7PUwxgWUkCRAp7j1nGGJi3GrhwhUpTVPlMMRzWwFSd7RaxEHZq5bQKABKtqb5P5O0baRX2hD5ipCVuPvbBCTlIw/s1600/Onebiplogo.gif" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="53" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijr1lSmXvNpNUxLN3uCpk-TkqaEYiBTkZ3W7xRwAmCVgiJ-hP7PUwxgWUkCRAp7j1nGGJi3GrhwhUpTVPlMMRzWwFSd7RaxEHZq5bQKABKtqb5P5O0baRX2hD5ipCVuPvbBCTlIw/s200/Onebiplogo.gif" width="200" /></a></div>
Last Friday has been my last working day in <b><a href="http://corporate.onebip.com/">Onebip</a></b>, the carrier billing payment platform headquartered in Milan. I leave the best technical team I have ever worked with, who has tackled endless challenges from transitioning to a microservice architecture, to adopting CQRS and Event Sourcing, and testing a large product depending on the integration with 400 mobile carriers.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrtA6kifUDYg5qZtJEoD5IeSt8X4PS5YxU_zskzkhQ7PTfrhF6-GfmhKy9CyZKUhAvw4qr4ePO2Cs2llwKQtmWsSlXH7KFvqiMmRWDm2qvrbXXfkCoySKX7wrQRAfYKXLljbz6Pg/s1600/elife-full-color-horizontal.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="101" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrtA6kifUDYg5qZtJEoD5IeSt8X4PS5YxU_zskzkhQ7PTfrhF6-GfmhKy9CyZKUhAvw4qr4ePO2Cs2llwKQtmWsSlXH7KFvqiMmRWDm2qvrbXXfkCoySKX7wrQRAfYKXLljbz6Pg/s200/elife-full-color-horizontal.png" width="200" /></a></div>
In May, I will start a new adventure as a Software Engineer in Test at <a href="http://elifesciences.org/">eLife</a>. Located in Cambridge, <b>eLife</b> is an open access journal that publishes scientific articles in the fields of biology and medicine, with the goal of improving the peer review process and accelerating science. As a non-profit organization, it's quite a different context with respect to selling product and services, but indeed a potentially large and positive impact on the world.<br />
<br />
Cambridge is a city of research and technology, and welcomes students and scientists, but also software developers like me. Moreover, it's small and peaceful (you can cycle around anywhere), while showing peaks of high technical level. It's the first place where I have been to a <a href="http://www.meetup.com/Cambridge-Programmers-Study-Group/">study group on the book Structure and Interpretation of Computer Programs</a>, or to a quite good <a href="https://blog.cambridgecoding.com/2016/01/18/free-evening-tech-talk-an-introduction-to-machine-learning/">introductions to machine learning</a> talk (and not to <i>Transpile typed ECMAScript without left-pad nor using arrays because you would need a polyfill for that</i> or some other hipster hallucination).<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8juyHIi4efqnSsYXOJVU2nWPZTXOPEqWc_Lv7oo5BF0WxFjk6eyGnSm3IuaCIBlMUUQknwtX3gfwBAV1dsqIE-F3lynvhSXN2Aa2ldnkxXtkqa0qw-A6Zu2nc_qBj5mT6c5DZpg/s1600/2016-01-24+09.13.22.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8juyHIi4efqnSsYXOJVU2nWPZTXOPEqWc_Lv7oo5BF0WxFjk6eyGnSm3IuaCIBlMUUQknwtX3gfwBAV1dsqIE-F3lynvhSXN2Aa2ldnkxXtkqa0qw-A6Zu2nc_qBj5mT6c5DZpg/s640/2016-01-24+09.13.22.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">See you on the other side of the Channel...</td></tr>
</tbody></table>
<br />Giorgiohttp://www.blogger.com/profile/12689416577856305650noreply@blogger.com2