Recently I discovered the power of Vagrant and Puppet. They allow me to automate all the steps I used to manually make before. Here I test drive the process of automatically configuring a Hadoop cluster in virtual machines for a fully distributed mode.
Today, there are two broad categories of provisioning and deployment automation. The first are convergent tools such as Puppet and Chef. The second are directed automation tools.
In this write up I'll be writing some of the techniques I'm using to model environment-specific differences between dev, staging, testing, and production.
A few months ago I was struggling (again) with logging configuration in JBoss 7.x. I was so depressed that I wrote to the Java EE 7 expert group : Logs. Should we finally do something? As a developer I’ve used all the possible logging Java frameworks for the last 12 years and I still struggle with logging in 2012.
Most of what you read and hear about devops is in online startups – about getting to market faster and building tight feedback loops with Continuous Delivery and Continuous Deployment. But devops is even more important in keeping systems running – in maintenance and sustaining engineering and support.
By using Jenkins, it’s pretty easy to get a Continuous Integration server set up with an Android project. But before you dive into setting up the software itself, it’s very helpful to have some basic concepts on a few different types of software that you will run into.
One of the things that is pushed here a lot is the necessity to make all new applications really production-ready before being deployed as a service. There is of course the adage “If it’s not monitored, it doesn’t exist” but beyond that there are a lot of other little fiddly details just to make something run.
Exceptions should be easily visible to the support and development teams and your development process should look to address all exceptions in forthcoming sprints.
How often you should analyze your log data really depends on the reason why you are carrying out the task in the first place, i.e., why are you analyzing your logs, and what exactly are you interested in finding? Below I list some of the common use cases for analyzing logs and give some pointers on how often it makes sense to do so in these situations.
Just like mainstream programming languages, it is possible (and good practice) to test your Puppet manifests so that you have higher confidence in them working when it comes time to actually run them.
While few of us want to live at the extremes of many production deployments of an app per day, many of us want to detect production problems quickly and be able to respond accordingly. Copy the infrastructure of good tests, piecemeal automated deployments, and good monitoring and apply them to your more reasonable goals.
I couldn’t find any good, brief, practical introduction into Puppet that gives you basic working knowledge in minimal time, so here it is. You will learn how to do the elementary things with Puppet – install packages, copy files, start services, execute commands. I won’t go into Puppet installation, nodes, etc. as this introduction focuses on the users of Vagrant, which comes with Puppet pre-installed and working in the serverless configuration.
Once we know something, we find it hard to imagine what it was like not to know it. Our knowledge has “cursed” us. And it becomes difficult for us to share out knowledge with others, because can’t readily re-create our listeners’ state of mind.
Mainly because my production environment is usually a cloud, and changing the code is a mess. What can we do? The solution that I like for these kind of problems is to use Apache’s environ variables.
With all of these tools and examples around , there should be no excuses anymore for Drupal developers to hack on their own machine and tell the systems people "It works on my machine" (let alone to hack in production).
Learn some of the neat tricks and reasons that you would want to use the open source CM tool, Chef.
For those that have to deal with release management, release train is a well-understood term. It refers to a software development schedule where multiple products are released as a part of a single ‘train’ on a regular, pre-planned schedule.
Four of the principals and laws that my company cites most frequently can help reinforce this direction and provide some needed checks as you begin transforming towards an organization whose path from idea to value (the software development lifecycle or SDLC in stodgy terms) needs to be more DevOps friendly.
In 1966 Abraham Maslow said, “I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.” Maslow gave us all too much credit. When we have a hammer and know how great it is, we not only treat everything as a nail, we actually perceive everything to be a nail.
At XebiaLabs, many of the questions we get about our enterprise deployment automation solution Deployit are from users looking for automated deployment as a prerequisite for Continuous Delivery. Often, this is the result of initiatives to extend existing Continuous Integration tooling to support application deployments.
Tom Limoncelli has his own version of The Joel Test – except his one is for sysadmins. I was only vaguely aware of Joel Spolsky’s test and only just read through it and rated my current team, and I’m glad to say we are just about at twelve for twelve.
Another great Europe-based DevOps Day is on it's way for anyone who missed the earlier ones. It's on October 5th and 6th and it's going to be in Rome, Italy.
One of the host powerful filters in logstash is the grok filter. It takes a grok pattern and parses out information contained in the text into fields that can be more easily used by outputs. This post serves hopefully as both an explanation of why and an example of how you might do that.
My alma-mater may be better known for its football team, but the engineering fraternity Theta Tau hosts a pretty wicked egg drop competition.
The Rubygem Builds have changed along with the internal #monitoringsucks repository. All of these Vagrant projects are basically my test setups to play with those new tools.