DevOps Zone is brought to you in partnership with:

Mitch Pronschinske is a Senior Content Analyst at DZone. That means he writes and searches for the finest developer content in the land so that you don't have to. He often eats peanut butter and bananas, likes to make his own ringtones, enjoys card and board games, and is married to an underwear model. Mitch is a DZone Zone Leader and has posted 2573 posts at DZone. You can read more from them at their website. View Full User Profile

Why You Shouldn't Have to Deploy Overnight

10.17.2011
| 8825 views |
  • submit to reddit

Are you still doing deployments at 3:00am?  If you are, you should have taken a look at Brian Crescimanno's post: "Why are you still deploying overnight?"

Whatever you call the process of turning your development codebase into a live, production application, I sincerely hope you’re not living in the Stone Age and doing it in the middle of the night under the guise of avoiding customer impact. Unfortunately, if my past experiences, and the experiences of many I’ve spoken to, are the norm, you very likely are.  If your strategy to avoid customer interruption is based solely on trying to avoid your customers, you’re setting yourself up for even more headaches and long-term failure. --Brian Crescimanno

 



Brian asserts that if you're doing these overnight deployments to avoid your customers during the initial release, then it's probably indicates that one or more things are broken in your process.  Here were the problems he listed with overnight deploys:

  • Problem 1: You presume there will be problems that impact availability.  
  • Problem 2: You’ve got a complicated process and you’re sending over-tired, over-worked people to deal with it.
  • Problem 3: You have no means of doing a phased rollout or a quick rollback.

So the moral of the story, if you buy in to his assessment, is that you should be getting some sleep at night instead of deploying.

Source - http://briancrescimanno.com/2011/09/29/why-are-you-still-deploying-overnight/
Image Source - http://www.flickr.com/photos/zenera/28726053/






Comments

Wojciech Kudla replied on Tue, 2011/10/18 - 5:25am

Problem 1: You presume there will be problems that impact availability.

I would call assuming there will be no problems at all a sign of ignorance.

Problem 2: You’ve got a complicated process and you’re sending over-tired, over-worked people to deal with it.

They don't have to be over-worked people, just people. If you don't account for people making mistakes, your doing your job wrong.

Problem 3: You have no means of doing a phased rollout or a quick rollback.

 

It is sometimes impossible to do a quick rollback. If someone thinks every deployment can be handled quicky and easily, they apparently have no experience in managing serious, big infrastructures.

Walter Bogaardt replied on Tue, 2011/10/18 - 4:14pm

Rolling out deployment to some servers in your production environment, and then rolling it out to the rest of the servers in the cluster. A common problem is that when a team spec'd out what they want website 'A' to do, not all the requirements were captured. Example:"When I talked to my friend mentioned above about his 3:00 am deployment, he told me they had to do it to end a contest that their website had been running. I’m sorry; that’s just not a reason to have to push new code to production. Feature switches targeting alternate code paths could have solved that problem. " A lot of times deployments are not practiced in a staging environment (between QA and Production) due to resource constraints, or the project has slipped so deployment process is often cut out over coding and then QA.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.