DevOps Zone is brought to you in partnership with:

Having worked as both a developer and an operations manager, I’ve gotten all too familiar with the ever widening gap between what developers and customers consider “done”. In order to help narrow this, I’d like to share some of my ideas and experiences concerning the software development processes with a vision towards actually releasing what customers need. Daniel is a DZone MVB and is not an employee of DZone and has posted 27 posts at DZone. You can read more from them at their website. View Full User Profile

Prerequisites for Continuous Deployment

05.21.2011
| 7719 views |
  • submit to reddit
Although we’ve skirted around the edges of Continuous Deployment on this blog, we haven’t really gone into any details. The main reason for this is simply that neither Matthias nor myself have ever continuously deployed to our production environments.

By cottagetocastlechic

How hard could it be? Well, like with most engineering endeavors, it’s about 99% preparation and 1% implementation. Unlike Continuous Integration, however, continuous deployment (CD) involves business as well as technical considerations. Let’s outline them and see how close (or far) we are to the holy grail of devops.

Continuous Deployment is founded upon Continuous Integration

CD takes your CI process to it’s ultimate conclusion. After a successful build and automated tests, the changes are automatically pushed to production. Your CD process will only be as good as your CI process.

A good CI means you’ve got your technical ducks in a row. Sufficient test coverage combined with test environments that could easily be used in production in an emergency are indicative of considerable engineering effort on both the development and sysadmin teams.

My use of “sufficient” is no accident. I’m not going to give some magic number here, because it depends on your business. I know our current code coverage of 35% is enough for us to avoid full regression tests before making releases (which we do many times a week).

Business at the Speed of Thought

Not only was this an excellent book written by Bill Gates in 1999, but continuous deployment is the process that enables it to happen. The problem is that most businesses (like a lot of people) tend to speak before they think.

“Why is ‘widget A’ already live? I didn’t clear that for release.”
“We planned that story together for this sprint and I finished it this morning.”
“Wow – but, I didn’t clear this idea with (marketing|management|shareholders)…you have to disable it ASAP!”

This has happened to me more times than I’d care to count. Unfortunately, folks are so trained in waterfall-style thinking, they usually reserve resources for their idea months in advance in the hopes that it might be implemented before the deadline.

Until your product owners are used to your CD’s quick turn-around times, use configuration flags for enabling and disabling of “risky” features (i.e. new or majorly refactored advertisement placements). I recommend some “alphaConfiguration” file to store these temporary flags. You’ll be able to keep a better overview of all the alpha features and, of course, can remove them once they’ve finally been approved for release.

These flags give you the all-important integration step in a tightly controlled fashion. Individual commits are small changes, and the sooner you get these changes live, the sooner you detect problems. This lean methodology

Continuous Deployment is the Holy Grail of DevOps

Isn’t this a bit over the top? No, not at all. CD means that developers are making production changes with every commit (albeit via build server proxy). Operations will have 100% confidence in their infrastructure including all the necessary monitoring and logging before they hand over this amount of control.

There are countless success stories out there: Etsy, Heyo, IMVU & Atlassian all do CD.

How close am I to CD? I’m really only missing the “alphaConfiguration” file to effectively hide half-baked ideas from public consumption. How about you? What are your major obstacles to deploying continuously to your live servers?

References
Published at DZone with permission of Daniel Ackerson, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Loren Kratzke replied on Mon, 2011/05/23 - 11:46am

I fully agree with your 35% number. It's a very sane number. I have a serious attitude about unit tests.

But I'm not so sure about developers pushing directly to production. The ability to do so is good, but not so much in practice. I think ejection seats are good in jet fighters but in the ideal world they would never be used. I think QA needs a role here unless the automated tests cover absolutely everything which is only possible on a very basic site. A site with hundreds or thousands of use cases will likely not have that kind of automated test coverage.

So I guess I am asking, does this puppy scale all the way up or is CD just for small/medium sites? And what about integration? Developer-A pushes to prod and then developer-B pushes and breaks developer A's fix (or other race conditions). At what point do you need to "cook" the release in a QA environment?

Daniel Ackerson replied on Mon, 2011/07/25 - 1:41pm in response to: Loren Kratzke

I think you can scale CD up as far as your test coverage goes. As you begin pushing the envelope and encounter bugs, you have to ensure automated tests cover the bugfixes. It wouldn't surprise me that Google does CD and they're about as big as you get. And, yes, they do have bugs (and sometimes, spectacular ones) - nevertheless, their CI systems capture most of the bad guys.

Emma Watson replied on Fri, 2012/03/30 - 6:12am

Yes there’s a lot to learn from Continuous Deployment, about streamlining and simplifying release and deployment, and reducing risk by breaking work down into smaller and smaller pieces and tieing all of this together with ops monitoring and metrics. But it’s not the “Holy Grail of Devops” or at least it shouldn’t be.

Swing

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.