SparkBuild: A Free Build Avoidance Tool
Just a few weeks ago, Electric Cloud launched the first GA version of SparkBuild, a free GNU Make (gmake) and NMAKE-based build tool that offers true build avoidance. Windows and Open Source Linux developers writing software in C/C++ will find that SprarkBuild can add significant agility to their build process, says Scott Castle, the Product Manager for the ElectricAccelerator product line. DZone spoke with Castle about what SparkBuild can do and where it came from. "We're really directing ourselves at individual developers who want to improve how they write software," said Castle. SparkBuild uses build avoidance technology for C/C++, which is similar to what Ant does for Java.
SparkBuild is a package containing two components: SparkBuild emake for build automation and SparkBuild Insight for build analysis. The package contains several benefits offered in Electric Cloud's commercial tools, such as annotation and build analysis It even offers some techniques that haven't been implemented yet in Electric Cloud's proprietary software, such as subbuild compiling. SparkBuild emake and Insight are derived from ElectricAccelerator and ElectricInsight - two Electric Cloud commercial tools.
Electric Cloud's parallel build tool, ElectricAccelerator, was the inspiration for SparkBuild, said Castle. "Over the years, we had to solve a number of build problems for ourselves. We had implemented agile just like many companies had and we needed to get faster builds for developers." Castle said they had the fast, 'full build' problem solved with ElectricAccelerator, but the Electric Cloud developers, like many programmers, do small incremental changes up to fifty times a day just running "make all" and recycling the build over and over. "So we came up with some interesting technology to make that faster and more reliable, and we decided to productize that, make it free, and provide it to the community," said Castle.
True Build Avoidance
The key feature of Spark build is build avoidance. Build avoidance has been around for some time as a way to reduce build times by rebuilding only the pieces that need to be rebuilt, and not the whole code base. Castle says it's lighter, and usually smarater, than the brute force approach, which involves throwing more hardware at the problem. Castle says build avoidance solves the daily dilemma of whether a developer should build software slow and safe, rebuilding the whole source code frequently, or build fast and dangerous by skipping parts of the build. Tools like ccache and ClearCase winkins have co-opted the term build avoidance, but they are actually doing object reuse. Castle said object reuse is when you use objects that other people have compiled in order to skip compiling those yourself. Object reuse only works in narrow scenarios he says, and it can be a headache to manage. Castle also mentioned a more traditional build avoidance tactic - an incremental build from the top of the build tree. "It basically means, 'look at my source base, find all the sources that have been modified since the last time I built the source code, and just recompile those,'" said Castle. That works pretty well for simple builds, he said, but for more complex codebases in Apache and Mozilla projects, an incremental build is slow, wasteful, and unreliable.
To build only what needs rebuilding, SparkBuild uses subbuilds. Using subbuilds allows developers to skip entire swaths of the build tree by building only the parts of the tree that are relevant to what they're working on at the moment. The build avoidance method makes the edit-build-test cycle a lot faster, and developers don't have to deal with incremental builds that can easily break.
The dependency graph below shows the agent component (util, xml, http libraries, and the agent application code) as solid. SparkBuild can recognize that only this component needs to be rebuilt.
"What SparkBuild does, is it can calculate exactly what's relevant to just the piece that you're trying to build," said Castle. For example, Castle says if you want to build just the MySQL back end database and not the front end user interface or utilities, SparkBuild can rebuild just that part. This can make the build run up to ten times faster with the same quality as any other build, and with more accurate runtimes.
SparkBuild emake can also emit XML-annotated build logs or annotation. The additional information includes:
- The environment and working directory used to run each command.
- The exact, expanded command-line of each command — even those that were run in “silent” mode with the @ prefix.
- The runtime of each command.
- The relationship between submakes in the build.
- The build dependency graph.
Developers who didn't get this information sometimes had to put date commands into makefiles for performance measurement. SparkBuild provides sophisticated build analysis using standard XML processing tools.
Spark Build Insight fulfills most developers' build analysis needs. The analysis tools calculate the amount of time spent on different types of work, including compiling and links. Developers can see the specific targets associated with each type of work and drill down into targets to see the commands executed and the output from those commands. SparkBuild Insight also searches for targets using numerous criteria like runtime, commands, name, and type of work.
To communicate what SparkBuild can do and to show people how to use it, Electric Cloud built a community website where forum moderators are especially responsive. The site's been up for only a month and it includes detailed blogs and videos telling you how to best leverage SparkBuild. Castle said the SparkBuild project is not part of any product strategy, but instead, it's about giving some of the benefits of Electric Cloud's innovations to individual developers. He says SparkBuild is aimed at individual programmers who are trying to embrace a lighter, more agile development strategy.
There are more features and integrations planned for SparkBuild. "We're heading in two main directions," said Castle. "Because SparkBuild is based on technology for our commercial products, we're dragging more of that commercial technology into the free product." Alongside that, they're also considering support for Ant or SCons, but Castle says feedback from the SparkBuild community will be a key factor for the direction of the product.
Here is a 10-minute 'self-pilot' for SparkBuild.