Debugging broken builds with SparkBuild Insight: Malformed Makefiles

In the previous 'Debugging broken builds' article we looked at how to troubleshoot a build that failed from a source code error. In part II, we'll look at a more difficult problem: makefile-based failures.

Case Study: Malformed Makefiles

(Grab this annotation file if you'd like to look at this example in your own SparkBuild Insight: error2.anno.xml.tar.gz)

Let's look at a little bit more complicated example. This time, tail -f has blessed us with this snippet:


chared.c:751: error: dereferencing pointer to incomplete type
chared.c:752: error: dereferencing pointer to incomplete type
chared.c:755: error: dereferencing pointer to incomplete type
make[3]: *** [chared.o] Error 1
make[3]: Leaving directory `c:/example/mysql-4.1.22/cmd-line-utils/libedit'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `c:/example/mysql-4.1.22/cmd-line-utils'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `c:/example/mysql-4.1.22'
make: *** [all] Error 2


Since we could have used grep pretty easily to solve the problem in case 1 (a nice, localized syntax error in an otherwise working build is trivial to find), let's just give it a spin here:
anno2log error2.anno.xml | grep 'error'

449 errors! This build is broken all over the place! Let's take a look with SparkBuild Insight:

Displaying all the failed jobs gives us this:

Already, SparkBuild Insight has made our work simpler - there's just one failed rule we need to pay attention to in this build. Double-clicking on the chared.o rule gives us more information:

That's a long list of errors! The best way to get started is to begin at the top: chared.c:35:20: config.h: No such file or directory

The lack of this header file has broken the build. We could open up chared.c, and try to find out exactly which config.h gcc is complaining about - it's not always clear, with system includes, relative paths, etc - but let's also take into account the subsequent errors:

...
In file included from chared.c:41:
el.h:131: error: parse error before "el_history_t"
...
chared.c:73: error: syntax error before "void"
chared.c: In function `cv_yank':
chared.c:76: error: dereferencing pointer to incomplete type
...


It's unlikely that a syntax error or simple coding mistake in the source file could cause this cascade. These kinds of errors, in this proliferation, are a pretty good hint at the problem here. Another is the original call to gcc, it's a little suspicious:

if gcc -MT chared.o -MD -MP -MF ".deps/chared.Tpo" -c -o chared.o chared.c; \
then mv -f ".deps/chared.Tpo" ".deps/chared.Po"; else rm -f ".deps/chared.Tpo"; exit 1; fi


There are no -I include paths in the compiler call, (and our first error is a missing header). This is the source of the compiler error - but why was the command-line malformed? To investigate further, we'll need to look at the makefile.Trying to identify the line of a makefile which defines a relevant rule is a pain in the console if all you've got is the stdout log - many rules will be patterns and searching for source file names or command-line arguements is an exercise in manual variable expansion. SparkBuild Insight makes it easy on us, however. If you have a local copy of the code, you could simply click the 'Makefile:0,446-447' link to have SparkBuild Insight open up the relevant lines. (Instead, for your convenience, here's the rule at line 446:)

.c.o:    if $(COMPILE) -MT $@ -MD -MP -MF "$(DEPDIR)/$*.Tpo" -c -o $@ $<; \
    then mv -f "$(DEPDIR)/$*.Tpo" "$(DEPDIR)/$*.Po"; else rm -f "$(DEPDIR)/$*.Tpo"; exit 1; fi

(download cmd-line-utils/libedit/Makefile)

We need to look at the value of $(COMPILE):

COMPILE = $(CC) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) $(AM_CPPFLAGS)\    $(CPPFLAGS) $(AM_CFLAGS) $(CFLAGS)

which all looks fine, until you take a look at the very next line in the makefile:

COMPILE = $(CC)

Here's the root cause of our problems: the makefile is malformed, and re-sets the compile command to lose some critical command-line arguments. We could have spent hours just trying to pick the error to trace, and tracking down which pattern rule is biting us; instead SparkBuild Insight lets us solve the problem in minutes.

At this point you've seen how to troubleshoot problems based on coding errors and on makefile errors using SparkBuild Insight. These are two of the most common failures you'll encounter when building software.