Debugging broken builds with SparkBuild Insight: Simple Problems

Writing code is great - good money, free soda, much more comfortable than yard work - until that code suddenly won't compile, and you're stuck trying to troubleshoot why. SparkBuild Insight owes its roots to solving build problems: in the beginning at Electric Cloud we had to run other peoples' builds, and (while being watched over our shoulder) quickly diagnose and fix failures. So we built a visualization tool which would display the results of a software build, and let the user drill into the results of specific rule jobs. This was much faster and more elegant than running grep -i 'ERROR' across a multi-megabyte stdout log, and as our team used it we added tons more functionality to rapidly (and gracefully) diagnose all kinds of problems.

This article will show you an example of troubleshooting real-world build problems with SparkBuild Insight, and give you a reason never to scan through a build log again.

Case Study: Simple problems

(Grab this annotation file if you'd like to look at this example in your own SparkBuild Insight: error1.anno.xml.tar.gz)

Let's pretend that you've written some code, typed emake all, and are pleasantly minding your own business when, out of nowhere, tail -f spits up this hairball:

emacs.c:51: error: parse error before "if"
emacs.c: At top level:
emacs.c:58: error: parse error before "else"
make[3]: *** [emacs.o] Error 1
make[3]: Leaving directory `c:/example/mysql-4.1.22/cmd-line-utils/libedit'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `c:/example/mysql-4.1.22/cmd-line-utils'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `c:/example/mysql-4.1.22'
make: *** [all] Error 2


Should you start scanning the build log from the bottom-up to find where the build failed? Try opening the anno file in sbinsight instead:

You can see immediately in the upper-left overview display that this was a failed build. Click the 'search for failed jobs' button to display a list of all the rule jobs which reported an error.

You'll get the following display:

This list shows all the jobs which finished with a non-zero return code (you can see the search string used: type:rule exitcode:!=0) Because our sort order (top-right pulldown) is set to serial order, the jobs are displayed in the order they are started. The first three failed jobs are rules which call a submake - you can check this by double-clicking any of them to pull up the job details - and their return codes are non-zero because the submake each invoked returned an error code. The last entry is the interesting one - emacs.o - since it's a rule which is actually running a compile. Double-click it and you'll see the details of that job:

Our build error is right here:


emacs.c: In function `em_delete_or_list':
emacs.c:51: error: `trivial_source_error' undeclared (first use in this function)


A garden-variety syntax error. This would have been pretty easy to find just by looking through the build log: but look at the data SparkBuild Insight is able to give you:the working directory (which will tell you which emacs.c, not always obvious); the makefile which defines the compile rule, good for checking compiler options and include paths; and the environment stack for the job, great for checking configurations. Most importantly: since SparkBuild Insight shows you just the output related to the failed command, it is easy to separate the relevant error from the cascade of 'me too' error reporting in the build. (This is the simplest possible example of root cause analysis.)

You've seen how to debug a straightforward build break with SparkBuild Insight. Now give it a try for yourself - and discover how much easier it is to troubleshoot with this kind of build analysis.