image of me

Paul Drummond

On Continuous Delivery

I've been using continuously delivery for over 4 years now. Here's what I've learned along the way.

Debunking some myths right off the bat

  • This isn't about making it easy for anyone to push bad code into master without a care in the world.
  • This isn't about going to production without testing your code.
  • This isn't incompatible with testing practices such as TDD.
  • This isn't about promoting a haphazard approach with an emphasis on rushing to get changes out without caring about quality.

What is Continuous Delivery?

As Martin Fowler puts it:

Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time.

And here is how the folks over at AWS define it:

Continuous delivery is a software development practice where code changes are automatically prepared for a release to production. A pillar of modern application development, continuous delivery expands upon continuous integration by deploying all code changes to a testing environment and/or a production environment after the build stage. When properly implemented, developers will always have a deployment-ready build artifact that has passed through a standardized test process.

Testing new code vs testing existing code

In a Continuous Delivery environment every change you make goes through a Deployment Pipeline, which builds the software using a CI system such as CircleCI and runs all the unit tests, integrated tests and acceptance tests in order to determine if your change breaks anything. Therefore, if your change is lucky enough to reach the end of the Deployment Pipeline without breaking anything, you can be confident it can be deployed to production at any time.

The key thing to remember here is that your change itself does NOT need to be tested before it goes to production, unless it affects end users. If your change is a smaller part of a bigger feature that won't be released to end users for a while yet, you can push it to production during development behind a feature flag with the confidence end users won't be affected. When in the early stages of development, the focus of the Deployment Pipeline is to ensure your change doesn't break existing functionality in production. There isn't really much value in testing the new code, certainly not on day one when there is barely anything to test. As development continues, you work alongside your team to increase the test coverage as more functionality is added. During this time, the purpose of the Deployment Pipeline gradually shifts more and more towards providing confidence the new functionality works correctly as well as checking for regressions.

In other words, testing isn't a barrier for going to production because you will be going to production from the very moment you start working on a new feature. Testing is just as important as ever but now, you write your tests while continuously going to production instead of trying to get everything perfect up front before going to production. It's a very different approach. It requires a mindset shift and in some cases unlearning habits you may have held close for years!

It's a huge mindset shift for developers

In order for Continuous Delivery to work, developers must become accustomed to committing changes to the main branch "at least once a day" as a general guideline (this is actually a requirement of Continuous Integration which is a prerequisite for Continuous Delivery). It's a guideline only, however - a way to encourage developers to become familiar with working this way. But after a while, most developers I work with couldn't tell you whether they commit once a day or not - it's usually more often actually. As long as the PRs are small and deployable, it doesn't matter.

I've found getting started with this is by far the most difficult part of switching to CD though. Developers just hate working this way initially - I should know, it happened to me! Most developers prefer to create a branch and work on it for days at a time in isolation, with the freedom to contemplate all the variants and consider all the use-cases. But with Continuous Delivery, often even a simple task must be broken down further and when you create a PR, sometimes it only contains a small part of the overall solution. Rather than working on functionality in isolation then delivering the entire piece at the end, you must become comfortable continuously delivering small parts of the overall solution and potentially exposing code that is often a work-in-progress or an experiment that may have to be rethought later on. It takes quite a while to get used to this, but eventually it becomes second nature and all is forgiven once you start noticing the benefits of this way of working.

DEPLOY TO PROD != DONE && DEPLOY TO PROD != RELEASE

With Continuous Delivery the definition of "deploy to prod" changes dramatically and that can be a very difficult thing to adapt to. We used to have this so ingrained into our culture and process, we didn't even realise it was a thing! We would say "deploy to prod" all day long and it would be implied to mean "it's done, it's ready for release". Suddenly, as we started to gradually switch some of our services to CD, "deploy to prod" meant something completely different! This was an extremely difficult issue for us to resolve and it took time. It's often not enough to just communicate it - it has to be learned gradually over time and through incremental process changes and ongoing discussions with all teams involved.

This is also complicated by the fact that tickets often also have a "definition of done", common when using agile frameworks such as SCRUM. Even when talking about a specific ticket though, "deploying to production" doesn't necessarily mean the ticket is done. A single ticket may require several PRs and each one will be deployed to production separately. A single ticket is considered "done" when all the functionality in the ticket is in production and tested.

I cover more on why deploying to production is not the same as making a release below as this is where Feature Flags come in.

Learning to embrace Feature Flags

Feature flags are a pretty simple idea, but in practice they can be quite controversial. They are very easy to abuse and if they aren't managed correctly, they can make your codebase overly complex and difficult to reason about.

But when utilized properly, they are a valuable way to allow developers to deliver "under development" functionality to production on a daily basis. While your end users are working away using your software, they will be oblivious to the fact that you are delivering a new "work-in-progress" feature before their very eyes. The feature is there, they just can't see it because it's behind a feature flag.

Using specialized software such as Launch Darkly, it's possible to control feature flags from a central admin interface and even turn flags on and off for individual users or groups of users. This is ideal for testing/demo purposes when you want to show off a new feature prior to release. And when it's finally time to release the feature to everyone, you simply turn the flag on and it just appears to end users as if it was always there all along (because it was! It was just hidden from them).

This all sounds like hard work - why bother?

Here are just a few reasons:

  • Deployment to production becomes routine and frictionless.
  • It enables us to deliver high quality functionality to our customers quicker with less hassle.
  • We are able to deliver small increments of end user value, so it's easy to adapt to change as we learn more about the needs of our customers.
  • Feature Flags allow us to gradually roll out features to select groups of users to try them out before the full launch.
  • We never worry about deploying on a Friday!
  • There is no lengthy release procedure - it's as simple as clicking a button.
  • Our PRs are small and simple, which allows us to do in-depth reviews without them taking too much time.
  • We rarely have to deal with merge conflicts and if we do they are small and it's easy to identify what the different changes represent and how best to resolve the conflicts.
  • When something goes wrong it's much easier to identify the problem because the changes are small.
  • We don't put off production issues until right at the end, so things like production configuration, scalability concerns, issues affecting end users, and so on are all dealt with up front and continuously considered during development.
  • We can deploy bug fixes to production extremely quickly, sometimes within an hour of them being reported, almost always within the same day.
  • We can demo new features to stakeholders in production during the various stages of development (usually at the end of each sprint) without affecting end-users.
  • We don't need complex branching strategies. We don't even need branches really. It's perfectly fine to merge directly to master. We only branch so we can get the benefit of peer reviews through GitHub Pull Requests but even then, our branches are usually small and short lived.

For more information

To find out more about Continuous Delivery check out: