thesecretdogproject

OK, so that title is a lie.

This article won’t teach you how to debug, because the simple answer to the question “how can I debug this code?” is this: use your intuition. Intuition comes from years of practice and directed learning, not from a blog post.  But where intuition fails, science prevails, and that is what this post is really about: a scientific approach to debugging software. This approach may be slow, and it may be tedious, but it does work.

Step 1 : look, but don’t touch

When you first hit a bug, you’ve got a golden window of opportunity to make observations. Don’t waste it! Forget debuggers and text editors, your primary tools at this stage are a pen and a notebook. Put one of each in your hands and leave the keyboard alone while you look at what’s going on, and write everything down. I’m not even kidding! Write it down, on paper. This will force you to slow down and actually look at the system, as well as providing a record for later that you can use to sanity check your hypothesizes about what’s causing the problem. The most important thing to do at this stage, besides careful observation, is to keep an open mind. Don’t even think about what the cause of the bug might be. If you do you’ll subconsciously ignore information that doesn’t correspond to your initial analysis, and you can’t afford to miss details at this stage.

The first step in the process of observation is to note down everything you can observe without interacting with the system. Did the code core dump? Did it output anything on stdout? Does it appear to have locked up? What happened just before the event? What state is the rest of the system in? Are there unusual peripherals attached? Are you running a special build? The details will depend on your system, of course, but you need to get it all down on paper.

Once you’ve captured all you can without touching the system, you can move on to making unobtrusive probes of system state. By this I mean you can run tools to show you what the CPU is doing or which processes are using memory. You can look at logfiles of other applications to attempt to capture any IPC interactions leading up to the bug. You can look at working files in the filesystem, examine timestamps, check open file descriptors, and so on. Use your imagination and your domain knowledge to capture more information without directly changing the state of the system. Just as before, make sure you write everything down!

Finally, when you’ve captured absolutely everything you can extract without changing anything, you can start probing more invasively. At this point, you can try attaching to the process with a debugger, or analysing the coredump. You can try sending signals to applications that appear to have locked up. You can start removing lockfiles, unplugging hardware, and poking buttons. In short, the gloves are now off and you can do what you like. By the end of this the system state will be completely destroyed, and your notebook will be full of observations.

Step 2 : come up with a hypothesis

I hope you enjoyed that time at the keyboard, because now you’re going to walk away from the computer for a bit. Take your notebook and pen, find a quiet place, and sit down with a coffee for a quarter of an hour to review your notes. Maybe get a colleague in to bounce ideas off. Your goal now is to come up with an idea about what’s going on with your system. Once again, try to keep an open mind! Don’t shortcut the process by running at the first idea you have: instead, take the time to fully think out the whole of the domain and figure out what could possibly account for the observations you’ve made. You should be able to come up with loads of ideas. I want everything from the obvious (“I think we overran our string buffer”) to the outlandish (“I think the kernel’s corrupted the VM page table”). Get them all written down in your notebook.

When you’ve finished brainstorming all these ideas, it’s time to wander around the lab for a bit to decide which seems most likely. At this point you’re engaging your intuition, hopefully for the first time in this process. Use your experience to filter your previous suggestions into three categories, ranging from highly likely to highly unlikely. Then pick your favorite from the pool of highly likely causes. This idea is now your hypothesis, so write it in your book under the heading “Hypothesis 1″.

Step 3 : design an experiment to validate your hypothesis

Now you’ve figured out your hypothesis, please refrain from running back to your computer to hack the code. You’re not ready to code yet! Instead, go back to the quiet place you used to brainstorm possible causes, and turn to a fresh page in your notebook. What you need to do now is design an experiment. If this sounds a lot like a high-school physics lesson, you’re probably getting the hang of this ;-)

What you’re looking for from your experiment is a set of output data which will tell you something about your hypothesis. Ideally they should either prove or disprove it, so think about what you’d need to do to conclusively demonstrate your hypothesis to be true or to be false.  Write these things down. Think also about what sort of quantity of data you need to give you confidence in your results. At this point you don’t know whether the bug happens regularly or is a freak occurrence, so you’ll need to gather a “control” dataset to compare your experimental results against. Write down exactly what data you need to gather, and in what quantity. Write down how you’re going to capture the data and how you’ll store it. Make notes on any special equipment you’ll need, and what versions of software you propose to test.

Step 4 : run the experiment, and capture the results

Happy days: it’s your chance to go back to your computer! But don’t get giddy at this point. All you need to do is run the experiment you just designed. Make sure you capture the data you need, and make sure you note any extra observations you make during the course of the experiment. But don’t do anything more. In particular, don’t start trying to test something else while you’re at it. Stick to the script: run the experiment.

Step 5 : analyze the results

Here’s the fun part. Take a look at the data you’ve gathered, and figure out whether they prove or disprove your hypothesis. If the results are conclusive, then you’re done! You can code up a fix, check it in (along with your experimental results!), and move on. But even if the results are not conclusive, that’s still a valid and useful outcome. You should now know more about the problem than you did previously, and you should be able to design a further experiment to glean yet more. Whatever your analysis, it will probably not surprise you to learn that I want you to write it down in your notebook. Read it back to yourself and check it makes sense. Remember you’re doing science here, and a key part of good science is peer review. You should be able to present your notebook to your most esteemed colleagues with pride and confidence, so make those notes good!

Step 6 : rinse and repeat

By this stage in the process, you may have solved your problem. But if you have not, you will have at least uncovered something more about the nature of the problem. In this case, your next step is to retreat from the keyboard once more to review your previous list of hypothesizes in light of your new knowledge. You may be able to strike some from the list, or add new ones. You’ll almost certainly be able to move some between the three categories of likelihood that I proposed earlier.

While you’re reviewing your progress, try your best to get out of the mindset of investigating your original hypothesis.  Instead, return to the open and objective outlook you had before when you were doing your brainstorming.  With this open mind, decide what hypothesis to explore next, and then head on back to Step 3 for the next stage of the investigation.

Step 7 : there is no step 7

There is no step 7 because steps 1 to 6, applied enough times, will solve your problem.  It may take you a long time — but eventually you’ll get there.  That’s the power of science!

Despite the surety of this debugging method, I bet you a mixed bag of sweets that you don’t know anyone who actually debugs like this. This is for one of two reasons. Either your colleagues are just running in circles and debugging by suspicion (“back that last change out, and see if that fixes it!”), or they’ve done enough debugging work to be balancing intuition and science in their heads all the time as they work. This latter possibility is some trick to pull off, but it’s basically what all software engineers do. The trouble with this is that it’s really easy to lose time through failed intuition or biased thinking. In my experience, it’s rare to avoid these pitfalls entirely. Some wizards can do it, but mere mortals may struggle.

With that in mind, I recommend giving this method a try the next time you’re faced with a difficult bug. At the worst, it’ll take longer than a purely intuitive approach. But at best it’ll yield incremental progress as opposed to intuition’s random stabs in the dark. I know which one I prefer.

When I was a lad, I used to devour guitar magazines. My favourite articles in these magazines were the gear reviews. I’d read page after page about guitars, amplifiers and effects pedals that I’d never be able to afford, weighing the merits of the kit. It was complete fantasy: my guitar setup in the real world consisted of a Squire Stratocaster and a Peavey practise amp. But the fantasy was fun all the same.

One of the side-effects of this imaginary shopping was that I got quite a good feel for what instruments cost. For a time, I would have been able to give you a good benchmark price for pretty much any mainstream guitar or amplifier from 1960 onwards. All this knowledge was entirely theoretical, though. I didn’t really have any feeling for the value of any of this gear. My dad bought this home to me one day when he observed that I was talking like a man who knew the cost of everything, but the value of nothing.

I was reminded of this pithy phrase recently when working on some code with a colleague. We were working on his PC, which for a coder is rather like driving someone elses car: nothing is quite where you expect it. I wanted to do a bit of editing, and my colleague suggested I use his SlickEdit install. When I demurred, wanting to use vim instead, he was a little put out. “You’ve 300 quid’s worth of SlickEdit sat there, and you want to use vim?”

But I did want to use vim. The funny thing was that it was actually more effective. We were trying to track down the definition of a function in the codebase. I minimised vim in the shell using CTRL-z, then ran ctags to generate a database of the code. Returning to vim, I used vim’s ctags bindings to quickly navigate to the definition of the function. My colleague missed this, however, and took over from me to open SlickEdit. “This is what SlickEdit’s good at”, he said, before proceeding to perform exactly the operations I’d done, but in about twice the time.

We’ll work at my PC next time.

I recently wanted to implement a simple build farm for our build servers at work. Currently we have a pool of three machines whose job it is to watch the repository for checkins on various projects, and build each update to ensure that the build is still in working order.

Our present implementation has a per-project script which runs on each machine as a cronjob. This script is responsible for checking the SCM for updates, checking out updated code, running the build, and emailing the results out to the project team. So far, it works well, but it has a number of limitations. Firstly, a lot of the code for checking the SCM, checking out code, and emailing people is common across the per-project scripts. While this code doesn’t change much, it’s a shame to duplicate it unnecessarily. Worse than that, however, is that each machine in the pool runs only one project build. If one project has a busy day, then that machine may be swamped while all the others are idling. We could get a greater build coverage by sharing the load between all the build machines.

How best to do this? I decided early on that a master-slave configuration would be easiest to implement and administer. All the configuration for the builds could live on the build master, while the slaves could be relatively “dumb” machines easily added to (or removed from) the pool. I also decided that the nature of the managing and running the build processes was sequential in nature. You check the SCM, you check out some code, you run a build, you email people.

Beyond this basic characterisation of the task at hand, I also had some restrictions which limited my implementation choices. Since I had a small time budget, I needed to use a language which would let me implement the build farm quickly.  Furthermore, since the build servers themselves are typically powerful machines I wasn’t too worried about the runtime overheads of the language.  These considerations naturally lead me to consider using a scripting language.  Looking beyond the initial implementation, I also wanted a language that could be easily maintained by the whole team. With these restrictions in mind, I decided shell script fit the bill. This wasn’t such a bad idea: more or less every Linux box has BASH, and it’s easy to use BASH to leverage other common Linux command line tools.  My experience with shell also suggests that you can achieve a lot so long as you keep things relatively simple.

Having decided on an implementation language, I came up with the basic design of the system. The master machine would be configured via. an ini file which would specify the projects to be built, and the machines in the build pool. The master would be an event driven system centered around an input queue. Firstly, a “scan” request on the input queue would trigger the master to examine the project SCMs for code changes. If any were changes were detected, the master would generate a build script for that project (based on shared library code combined with a small piece of project-specific code) and send the script to a job runner machine for processing. Once the build had completed on the runner machine would generate a “job complete” event on the master’s input queue, which would trigger report emails from the master to the project team. The inter-machine communications would be managed using ssh and scp.

The tricky bit turned out to be implementing the queues, which are quite an essential element of the system. The canonical shell IPC mechanism is the named pipe (fifo). The trouble with fifos, insofar as my design goes, is that it is necessary to have a process listening on a pipe in order for a blocking write into the pipe to return. Try it for yourself!

# Create a fifo, then write some data into it
# This will block until you read from the fifo
tom@gibbon:~$ mkfifo /tmp/myfifo
tom@gibbon:~$ echo "Hello world" > /tmp/myfifo

# Now open a new terminal and read from the fifo
tom@gibbon:~$ cat /tmp/myfifo
Hello world

# Your original echo command will now return in your first terminal

This wouldn’t work too well in my scheme because I was looking to push events around the system using ssh. If the job runner wanted to report the results of a build to the master and had to wait around until the master was ready to read the result, that wouldn’t be ideal.

If fifos were out, then, perhaps a regular file would do. If the queue was a plain old file, it would be easy enough to append data to the end of the file, and also easy enough to read data from the front of the file. Sounds good so far. The trouble with a regular file, however, is two-fold. Firstly, you can’t tell when something has been written to the file without periodically polling it, something like this:

# Create a regular file, and wait for it to grow
tom@gibbon:~$ touch /tmp/myqueue.txt
tom@gibbon:~$ while true; do test $(stat -c %s /tmp/myqueue.txt) -gt 0 && break; done

# Now open a new terminal and echo into the file
tom@gibbon:~$ echo "Hello world" > /tmp/myqueue.txt

# Your while loop will now return in your first terminal
# Note your CPU use has rocketed because of the spinning while loop!

This works reasonably enough, especially if you add a short sleep in the while loop to prevent hammering the CPU. Polling is a bit ugly, though. It would be much nicer if you could somehow block on the file changing. Happily, there is a solution, in the form of the excellent inotify-tools, specifically inotify-wait:

# Create a regular file, and wait for it to be modified
tom@gibbon:~$ touch /tmp/myqueue.txt
tom@gibbon:~$ inotifywait -e modify /tmp/myqueue.txt

# Now open a new terminal and echo into the file
tom@gibbon:~$ echo "Hello world" > /tmp/myqueue.txt

# inotifywait will now return.  Note your CPU hasn't been
# working overtime :-)

The second part of the two-fold trouble with a regular file is that two processes can modify the same file at the same time with unpredictable results. We need some kind of locking infrastructure. The following approach works well, making use of the shell’s “noclobber” mode to reduce races between checking the file exists and writing to it:

# $1 -- filename
lock() {
    while ! ( set -o noclobber; echo "$$" > "${1}.lock" ) 2>/dev/null
    do
        sleep 1
    done
}

# $1 -- filename
unlock() {
    rm -f "${1}.lock"
}

Note this is a “busy” lock — that is, we’re periodically doing something while waiting for the lock to become free. In this case, checking to see whether the lockfile exists. This can be improved upon by leveraging inotifywait once more:

# $1 -- filename
lock() {
    while ! ( set -o noclobber; echo "$$" > "${1}.lock" ) 2>/dev/null
    do
        inotifywait -e delete_self "${1}.lock" &> /dev/null
    done
}

# $1 -- filename
unlock() {
    rm -f "${1}.lock"
}

Now the locking process simply sleeps until the lockfile has been deleted by the process holding it. Insofar as interprocess locking goes, this is pretty good, but there is one remaining gotchya in that it is possible to create deadlocks should the process holding the lock unexpectedly exit before releasing the lock. We can solve that with a trap:

# $1 -- filename
lock() {
    while ! ( set -o noclobber; echo "$$" > "${1}.lock" ) 2>/dev/null
    do
        inotifywait -e delete_self "${1}.lock" &> /dev/null
    done
    # We now hold the lock
    trap "rm -f ${1}.lock" EXIT
}

# $1 -- filename
unlock() {
    rm -f "${1}.lock"
}

The downside of this, of course, is that it rather rudely overrides any existing trap which is set for SIGEXIT. Is it possible to save the previous trap and restore it later? I leave this as an exercise for the reader.

Putting all of this together, then, we can implement a fairly nice queue interface for Bash:

# Lock the queue to prevent access from another process
# $1 -- queue file
queue_lock() {
    while ! ( set -o noclobber; echo "$$" > "${1}.lock" ) 2>/dev/null
    do
        inotifywait -e delete_self "${1}.lock" &> /dev/null
    done
}

# Unlock the queue to enable writing again
# $1 -- queue file
queue_unlock() {
    rm -f "${1}.lock"
}

# Wait on the queue being modified
# $1 -- queue file
queue_wait() {
    test -f "$1" && inotifywait -e modify "${1}" &>/dev/null
}

# Add an entry to the queue
# $1 -- queue file
# $2 -- entry
queue_push() {
    echo "$2" >> "$1"
}

# Remove an entry from the queue
# $1 -- queue file
queue_pop() {
    local e=$(head -n1 "$1")
    awk 'NR != 1 { print }' $1 > ${1}.new && mv ${1}.new ${1}
    echo "$e"
}

# Get current length of the queue
# $1 -- queue file
queue_length() {
    test -f "$1" || echo 0
    wc -l $1 | cut -d " " -f1
}

This test script demonstrates the queue in action:

#!/bin/bash

QUEUE=/tmp/queue

# include the queue interface
. $(dirname $0)/libqueue.sh

#
# Some wrapper functions for queue addition/removal
#

# $1 -- n items
add_to_queue() {
    local i=0
    for ((i=0;i<$1;i++))
    do
        queue_lock $QUEUE
        queue_push $QUEUE "Item $i"
        queue_unlock $QUEUE
        echo "> pushed : Item $i"
    done
}

# $1 -- n items
remove_from_queue() {
    local i=0
    local err=
    local ent=
    for ((i=0;i<$1;i++))
    do
        test 0 -eq $(queue_length $QUEUE) && queue_wait $QUEUE
        queue_lock $QUEUE
        ent="$(queue_pop $QUEUE)"
        queue_unlock $QUEUE
        test "$ent" = "Item $i" && err="" || err="ERROR"
        echo "< popped : $ent $err"
    done
}

# Spawn subshell processes to add to/remove from the queue
( add_to_queue 1000 ) &
( remove_from_queue 1000 ) &
wait $!

Any there you go! Multiprocess, block-free queues in Bash. Huzzah!

As a part of my day-job, I was tasked earlier this year with improving the stability of our software stack. This was a wide ranging brief, involving tracking down and fixing bugs at every level, from the kernel up to the UI. It often required poking into the dusty corners that other programmers preferred to ignore, unearthing weird behaviours we never had the time to understand during development. On the whole, it was a task which demanded a cast-iron cynicism, a well-developed sense of suspicion, and not a little patience!

Over the course of my time on stability-watch, I noticed that I was increasingly guiding my suspicion by a barely-conscious mental checklist, a list of signs that a particular area of the code might be worth a further look. Like Joel On Software’s 12-point team checklist I was developing my own secretdogproject checklist of code quality.

So, with no further ado, here’s my checklist of bad signs to look out for in code. These aren’t language-specific issues, since most coders are pretty good at picking these out. Rather, they’re more holistic signs of quality in a code base which might help signpost areas of poor quality for review.

Does the code live in a sensible place and have a sensible name?

Each component of a codebase should have a name that communicates something about what that component does.

When combined, components should make use of a coherent directory hierarchy in order to communicate something about how the components fit together as a whole.

Does the code have any documentation?

As any true hacker knows, the best approach when in doubt about what some code does is to use the source. But sometimes, a paragraph of top-level design information, written in a high-level human language like English, can save hours of grubbing around in the code.

At a bare minimum, you should have a readme. Bonus points for autogenerated API docs.

Does the code build cleanly?

Compiler warnings are a hint that your code is wrong, and you ignore such hints at your peril. Good code should compile without warnings.

Bonus points if you’ve configured the build to treat warnings as errors to prevent future warnings creeping into the build.

Does the code have a test strategy?

Code can be tested in many different ways.  Some components lend themselves to unit testing.  Others are better tested in-situ using test harnesses or other scaffolding. Whatever the best approach is for your component, you should have a developer test strategy in place, and it should be obvious how to use it.

Bonus points for including links to archived test results in the component documentation.

Does the code look nice?

If people spend time making a thing look nice, that probably means that they care about that thing. This applies as much to code as it does to a house or a garden. A consistent aesthetic in code means that the implementers took the time to make it look good, which may imply that they also took the time to consider corner cases, double check API call documentation, and prototype the implementation to chase out any unforeseen snags.

It doesn’t matter, by the way, what your coding convention is — so long as one is evident, and it is consistent throughout.

Do the comments make sense?

Code comments should be hints to the reader about what is going on. This isn’t the place for top-level design information. Comments should act as signposts to document why we’re currently doing what we’re doing. Since code is typically read many more times than it is written, handy implementer’s hints can be invaluable for future maintenance.

Comments are a good indicator of code quality. If a comment is out-of-date, or looks like it may have been copied-and-pasted from somewhere else (e.g. the comment is out of context), that may be a hint that someone wasn’t paying full attention when they made their changes.

Does the code have a good heritage?

One of the joys of modern source code management systems is that the history of each file in the code base is tracked. A developer can look back at how the code has changed over time, and this can be a valuable tool for estimating the quality of the code.

Bad signs include lots of change (which may indicate the design of the code is broken or fragile), poor changeset granularity (“big-bang” changes are easier to mess up and more difficult to test) and poor checking commenting (indicating the coder didn’t care enough or didn’t have time to write a decent comment).

Bonus points for documentation of why the change was made, the level of testing carried out, and references to bug/issue tracking tickets further describing the change.

Is the code doing anything ugly or stupid?

Every programmer knows that, every so often, one has to hack things a little to get them going. Whether you’re programming around a weird library bug, breaking an abstraction in order to get at some data you need, or adding an expeditious but aesthetically troubling function which you’ll clean up after this release is out the door (honest guv), sometimes a bit of outright hackery is the best way to get the job done.

The trouble with these little hacks is that, over time, they accumulate in a codebase to form technical debt. As with financial debt, you need to keep on top of it lest interest payments become prohibitive. This is partially what code refactoring is about: going back over the old ground to make good the things you missed or had to ignore the first time around.

If a code base contains a lot of hacks, it’s a good sign that the authors haven’t been able to carry out any refactoring work. This in turn suggests the code may be fragile, bug-prone, and difficult to extend.

Is there any evidence of code review?

As every competent coder should know, defects are cheaper to fix the earlier they’re discovered in the development cycle. In an ideal world, careful attention during system specification and design should minimise defects throughout the project. But in the real world, designs are not perfect. Even worse, programmers are only human and may introduce new errors of their own as they’re coding!

One well-understood method of catching errors at the coding stage is code review: formal inspections of work designed to catch programming faults. It is an effective way of improving code quality.

If there is evidence that a component has been subject to a code review process, it is far less likely to contain errors than an unreviewed component.

Debugging with GDB and Python — LJ article now freely available!

The article I had published in the June edition of Linux Journal has just become freely available on their website. You can read it here

I recently installed Ubuntu on an old PC I wanted to act as a temporary server for a website. Seeing as this box has to live in my home office, I wanted it to be as quiet as possible.

In the past I’ve built PCs for music recording purposes and designed them to be as quiet as possible by using large case fans, passive CPU coolers, and super-efficient power supplies. In this case, I had the PC and whatever was in my spares box at my disposal. Here’s how I quieted it down.

Remove surplus components

This PC used to be a graphics workstation, so it had a beefy graphics card as well as a pair of hard drives in a RAID array. I don’t need any of that for a server, so I removed all the extraneous components I could from the case. Fewer components means less power draw, hence less heat to remove from the case. Fewer cables connecting components to the motherboard means fewer obstructions to smooth airflow.

Ensure fans are using speed control

Most modern motherboards offer a speed control pin for fans. These can be used to slow down case and CPU fans when the box isn’t so hot. Slower fans mean less noise.

Install lm-sensors and fancontrol

lm-sensors provides hardware monitoring support in Linux, and fancontrol is a script which works with the sensor data to control the speed of fans in the system.

Ubuntu packages both, so you can install them with aptitude:

aptitude install lm-sensors fancontrol

Once the packages are installed, you can set them up for use with sensors-detect. This script works out which fans are attached to which PWM controllers by modifying PWM output settings while you listen for changes in fan speed. It then goes on to figure out the maximum and minimum PWM settings for the fans in your box, and allows you to configure temperature/fanspeed relationships.

Test it out!

With fancontol running, my Ubuntu server is no noisier than the road outside my house. This is an acceptable noise level for me.

However, I wanted to be sure that the fan controller was doing what it said on the tin. As such, I whipped up a simple shell script to log CPU and case temperature against fan speed:


#!/bin/bash
#
# Log temperature information from sensors
#

readonly SAMPLE_INTERVAL=30 # seconds
readonly HEADER_INTERVAL=10 # samples
readonly LOGFILE=/root/templog.csv

while true
do
if test $count -eq 0
then
do_header
count=$HEADER_INTERVAL
fi
do_templog
sleep $SAMPLE_INTERVAL
count=$((count - 1))
done >> $LOGFILE


I tried a quick test of system response by subjecting the CPU to some computational stress:

dd if=/dev/urandom | gzip -c -9 > /dev/null

With a few of these pipelines running in parallel, the CPU stayed more-or-less 100% busy.

Here’s a graph of the system response:

As you can see, the system is behaving as it should. As the CPU is loaded, core temperature increases and fan speed ramps up in response. Removing the CPU load allows the core to cool, and the fan slows down again. Perfect.

Just for fun I left my script running in the background for the next few days. Here’s a graph of the data:

Aside from a few outliers, the fan speed looks pretty much stable until the CPU temperature gets to around 35 degrees, which is the trigger temperature for a speed increase. At that point there is quite a lot of fluctuation in fan speed. It would be interesting to see whether hysteresis could be added to the fancontrol script to compensate for that.

In conclusion

With a little bit of effort you can reduce the noise generated by a standard PC, without necessarily buying any specialist parts. If you’ve got to share a room with that PC, the results can be more than worth it!