One of the side-effects of this imaginary shopping was that I got quite a good feel for what instruments cost. For a time, I would have been able to give you a good benchmark price for pretty much any mainstream guitar or amplifier from 1960 onwards. All this knowledge was entirely theoretical, though. I didn’t really have any feeling for the value of any of this gear. My dad bought this home to me one day when he observed that I was talking like a man who knew the cost of everything, but the value of nothing.

I was reminded of this pithy phrase recently when working on some code with a colleague. We were working on his PC, which for a coder is rather like driving someone elses car: nothing is quite where you expect it. I wanted to do a bit of editing, and my colleague suggested I use his SlickEdit install. When I demurred, wanting to use vim instead, he was a little put out. “You’ve 300 quid’s worth of SlickEdit sat there, and you want to use vim?”

But I did want to use vim. The funny thing was that it was actually more effective. We were trying to track down the definition of a function in the codebase. I minimised vim in the shell using CTRL-z, then ran ctags to generate a database of the code. Returning to vim, I used vim’s ctags bindings to quickly navigate to the definition of the function. My colleague missed this, however, and took over from me to open SlickEdit. “This is what SlickEdit’s good at”, he said, before proceeding to perform exactly the operations I’d done, but in about twice the time.

We’ll work at my PC next time.

Our present implementation has a per-project script which runs on each machine as a cronjob. This script is responsible for checking the SCM for updates, checking out updated code, running the build, and emailing the results out to the project team. So far, it works well, but it has a number of limitations. Firstly, a lot of the code for checking the SCM, checking out code, and emailing people is common across the per-project scripts. While this code doesn’t change much, it’s a shame to duplicate it unnecessarily. Worse than that, however, is that each machine in the pool runs only one project build. If one project has a busy day, then that machine may be swamped while all the others are idling. We could get a greater build coverage by sharing the load between all the build machines.

How best to do this? I decided early on that a master-slave configuration would be easiest to implement and administer. All the configuration for the builds could live on the build master, while the slaves could be relatively “dumb” machines easily added to (or removed from) the pool. I also decided that the nature of the managing and running the build processes was sequential in nature. You check the SCM, you check out some code, you run a build, you email people.

Beyond this basic characterisation of the task at hand, I also had some restrictions which limited my implementation choices. I needed to use a language which would let me implement the build farm, meaning that a scripting language was going to be the best choice. I also wanted something that could be easily maintained by the whole team. These two restrictions combined to suggest shell script as the most sensible choice of language. This wasn’t such a bad idea: more or less every Linux box has BASH, and it’s easy to use BASH to leverage other common Linux command line tools.

With these restrictions in mind, I came up with the basic design of the system. The master machine would be configured via. an ini file which would specify the projects to be built, and the machines in the build pool. The master would be an event driven system centred around an input queue. Firstly, a “scan” request on the input queue would trigger the master to examine the project SCMs for code changes. If any were changes were detected, the master would generate a build script for that project (based on shared library code combined with a small piece of project-specific code) and send the script to a job runner machine for processing. Once the build had completed on the runner machine would generate a “job complete” event on the master’s input queue, which would trigger report emails from the master to the project team. The inter-machine communications would be managed using ssh and scp.

The tricky bit turned out to be implementing the queues, which are quite an essential element of the system. The canonical shell IPC mechanism is the named pipe (fifo). The trouble with fifos, insofar as my design goes, is that it is necessary to have a process listening on a pipe in order for a blocking write into the pipe to return. Try it for yourself!

# Create a fifo, then write some data into it
# This will block until you read from the fifo
tom@gibbon:~$ mkfifo /tmp/myfifo
tom@gibbon:~$ echo "Hello world" > /tmp/myfifo

# Now open a new terminal and read from the fifo
tom@gibbon:~$ cat /tmp/myfifo
Hello world

# Your original echo command will now return in your first terminal

This wouldn’t work too well in my scheme because I was looking to push events around the system using ssh. If the job runner wanted to report the results of a build to the master and had to wait around until the master was ready to read the result, that wouldn’t be ideal.

If fifos were out, then, perhaps a regular file would do. If the queue was a plain old file, it would be easy enough to append data to the end of the file, and also easy enough to read data from the front of the file. Sounds good so far. The trouble with a regular file, however, is two-fold. Firstly, you can’t tell when something has been written to the file without periodically polling it, something like this:

# Create a regular file, and wait for it to grow
tom@gibbon:~$ touch /tmp/myqueue.txt
tom@gibbon:~$ while true; do test $(stat -c %s /tmp/myqueue.txt) -gt 0 && break; done

# Now open a new terminal and echo into the file
tom@gibbon:~$ echo "Hello world" > /tmp/myqueue.txt

# Your while loop will now return in your first terminal
# Note your CPU use has rocketed because of the spinning while loop!

This works reasonably enough, especially if you add a short sleep in the while loop to prevent hammering the CPU. Polling is a bit ugly, though. It would be much nicer if you could somehow block on the file changing. Happily, there is a solution, in the form of the excellent inotify-tools, specifically inotify-wait:

# Create a regular file, and wait for it to be modified
tom@gibbon:~$ touch /tmp/myqueue.txt
tom@gibbon:~$ inotifywait -e modify /tmp/myqueue.txt

# Now open a new terminal and echo into the file
tom@gibbon:~$ echo "Hello world" > /tmp/myqueue.txt

# inotifywait will now return.  Note your CPU hasn't been
# working overtime :-)

The second part of the two-fold trouble with a regular file is that two processes can modify the same file at the same time with unpredictable results. We need some kind of locking infrastructure. The following approach works well, making use of the shell’s “noclobber” mode to reduce races between checking the file exists and writing to it:

# $1 -- filename
lock() {
    while ! ( set -o noclobber; echo "$$" > "${1}.lock" ) 2>/dev/null
    do
        sleep 1
    done
}

# $1 -- filename
unlock() {
    rm -f "${1}.lock"
}

Note this is a “busy” lock — that is, we’re periodically doing something while waiting for the lock to become free. In this case, checking to see whether the lockfile exists. This can be improved upon by leveraging inotifywait once more:

# $1 -- filename
lock() {
    while ! ( set -o noclobber; echo "$$" > "${1}.lock" ) 2>/dev/null
    do
        inotifywait -e delete_self "${1}.lock" &> /dev/null
    done
}

# $1 -- filename
unlock() {
    rm -f "${1}.lock"
}

Now the locking process simply sleeps until the lockfile has been deleted by the process holding it. Insofar as interprocess locking goes, this is pretty good, but there is one remaining gotchya in that it is possible to create deadlocks should the process holding the lock unexpectedly exit before releasing the lock. We can solve that with a trap:

# $1 -- filename
lock() {
    while ! ( set -o noclobber; echo "$$" > "${1}.lock" ) 2>/dev/null
    do
        inotifywait -e delete_self "${1}.lock" &> /dev/null
    done
    # We now hold the lock
    trap "rm -f ${1}.lock" EXIT
}

# $1 -- filename
unlock() {
    rm -f "${1}.lock"
}

The downside of this, of course, is that it rather rudely overrides any existing trap which is set for SIGEXIT. Is it possible to save the previous trap and restore it later? I leave this as an exercise for the reader.

Putting all of this together, then, we can implement a fairly nice queue interface for Bash:

# Lock the queue to prevent access from another process
# $1 -- queue file
queue_lock() {
    while ! ( set -o noclobber; echo "$$" > "${1}.lock" ) 2>/dev/null
    do
        inotifywait -e delete_self "${1}.lock" &> /dev/null
    done
}

# Unlock the queue to enable writing again
# $1 -- queue file
queue_unlock() {
    rm -f "${1}.lock"
}

# Wait on the queue being modified
# $1 -- queue file
queue_wait() {
    test -f "$1" && inotifywait -e modify "${1}" &>/dev/null
}

# Add an entry to the queue
# $1 -- queue file
# $2 -- entry
queue_push() {
    echo "$2" >> "$1"
}

# Remove an entry from the queue
# $1 -- queue file
queue_pop() {
    local e=$(head -n1 "$1")
    awk 'NR != 1 { print }' $1 > ${1}.new && mv ${1}.new ${1}
    echo "$e"
}

# Get current length of the queue
# $1 -- queue file
queue_length() {
    test -f "$1" || echo 0
    wc -l $1 | cut -d " " -f1
}

This test script demonstrates the queue in action:

#!/bin/bash

QUEUE=/tmp/queue

# include the queue interface
. $(dirname $0)/libqueue.sh

#
# Some wrapper functions for queue addition/removal
#

# $1 -- n items
add_to_queue() {
    local i=0
    for ((i=0;i<$1;i++))
    do
        queue_lock $QUEUE
        queue_push $QUEUE "Item $i"
        queue_unlock $QUEUE
        echo "> pushed : Item $i"
    done
}

# $1 -- n items
remove_from_queue() {
    local i=0
    local err=
    local ent=
    for ((i=0;i<$1;i++))
    do
        test 0 -eq $(queue_length $QUEUE) && queue_wait $QUEUE
        queue_lock $QUEUE
        ent="$(queue_pop $QUEUE)"
        queue_unlock $QUEUE
        test "$ent" = "Item $i" && err="" || err="ERROR"
        echo "< popped : $ent $err"
    done
}

# Spawn subshell processes to add to/remove from the queue
( add_to_queue 1000 ) &
( remove_from_queue 1000 ) &
wait $!

Any there you go! Multiprocess, block-free queues in Bash. Huzzah!

As a part of my day-job, I was tasked earlier this year with improving the stability of our software stack. This was a wide ranging brief, involving tracking down and fixing bugs at every level, from the kernel up to the UI. It often required poking into the dusty corners that other programmers preferred to ignore, unearthing weird behaviours we never had the time to understand during development. On the whole, it was a task which demanded a cast-iron cynicism, a well-developed sense of suspicion, and not a little patience!

Over the course of my time on stability-watch, I noticed that I was increasingly guiding my suspicion by a barely-conscious mental checklist, a list of signs that a particular area of the code might be worth a further look. Like Joel On Software’s 12-point team checklist I was developing my own secretdogproject checklist of code quality.

So, with no further ado, here’s my checklist of bad signs to look out for in code. These aren’t language-specific issues, since most coders are pretty good at picking these out. Rather, they’re more holistic signs of quality in a code base which might help signpost areas of poor quality for review.

Does the code live in a sensible place and have a sensible name?

Each component of a codebase should have a name that communicates something about what that component does.

When combined, components should make use of a coherent directory hierarchy in order to communicate something about how the components fit together as a whole.

Does the code have any documentation?

As any true hacker knows, the best approach when in doubt about what some code does is to use the source. But sometimes, a paragraph of top-level design information, written in a high-level human language like English, can save hours of grubbing around in the code.

At a bare minimum, you should have a readme. Bonus points for autogenerated API docs.

Does the code build cleanly?

Compiler warnings are a hint that your code is wrong, and you ignore such hints at your peril. Good code should compile without warnings.

Bonus points if you’ve configured the build to treat warnings as errors to prevent future warnings creeping into the build.

Does the code have a test strategy?

Code can be tested in many different ways.  Some components lend themselves to unit testing.  Others are better tested in-situ using test harnesses or other scaffolding. Whatever the best approach is for your component, you should have a developer test strategy in place, and it should be obvious how to use it.

Bonus points for including links to archived test results in the component documentation.

Does the code look nice?

If people spend time making a thing look nice, that probably means that they care about that thing. This applies as much to code as it does to a house or a garden. A consistent aesthetic in code means that the implementers took the time to make it look good, which may imply that they also took the time to consider corner cases, double check API call documentation, and prototype the implementation to chase out any unforeseen snags.

It doesn’t matter, by the way, what your coding convention is — so long as one is evident, and it is consistent throughout.

Do the comments make sense?

Code comments should be hints to the reader about what is going on. This isn’t the place for top-level design information. Comments should act as signposts to document why we’re currently doing what we’re doing. Since code is typically read many more times than it is written, handy implementer’s hints can be invaluable for future maintenance.

Comments are a good indicator of code quality. If a comment is out-of-date, or looks like it may have been copied-and-pasted from somewhere else (e.g. the comment is out of context), that may be a hint that someone wasn’t paying full attention when they made their changes.

Does the code have a good heritage?

One of the joys of modern source code management systems is that the history of each file in the code base is tracked. A developer can look back at how the code has changed over time, and this can be a valuable tool for estimating the quality of the code.

Bad signs include lots of change (which may indicate the design of the code is broken or fragile), poor changeset granularity (“big-bang” changes are easier to mess up and more difficult to test) and poor checking commenting (indicating the coder didn’t care enough or didn’t have time to write a decent comment).

Bonus points for documentation of why the change was made, the level of testing carried out, and references to bug/issue tracking tickets further describing the change.

Is the code doing anything ugly or stupid?

Every programmer knows that, every so often, one has to hack things a little to get them going. Whether you’re programming around a weird library bug, breaking an abstraction in order to get at some data you need, or adding an expeditious but aesthetically troubling function which you’ll clean up after this release is out the door (honest guv), sometimes a bit of outright hackery is the best way to get the job done.

The trouble with these little hacks is that, over time, they accumulate in a codebase to form technical debt. As with financial debt, you need to keep on top of it lest interest payments become prohibitive. This is partially what code refactoring is about: going back over the old ground to make good the things you missed or had to ignore the first time around.

If a code base contains a lot of hacks, it’s a good sign that the authors haven’t been able to carry out any refactoring work. This in turn suggests the code may be fragile, bug-prone, and difficult to extend.

Is there any evidence of code review?

As every competent coder should know, defects are cheaper to fix the earlier they’re discovered in the development cycle. In an ideal world, careful attention during system specification and design should minimise defects throughout the project. But in the real world, designs are not perfect. Even worse, programmers are only human and may introduce new errors of their own as they’re coding!

One well-understood method of catching errors at the coding stage is code review: formal inspections of work designed to catch programming faults. It is an effective way of improving code quality.

If there is evidence that a component has been subject to a code review process, it is far less likely to contain errors than an unreviewed component.

It’s been an age since I finished this book, but I haven’t been able to write a review of it yet. There are many excuses for this: I’ve been starting a business, building walls in my front garden, enjoying the summer. Beyond that I suspect this is just a difficult book (strictly: set of books) to review. It’s a big book, both in the physical sense and the metaphysical sense.

Where No Country For Old Men and The Road are terse, The Border Trilogy is sprawling and lyrical. Sentences are long, free-flowing; reminiscent of Kerouac at times in their loping narrative. The landscapes of Mexico and Texas and the desert loom large, as does the physicality of the animals and people inhabiting these empty places.

Throughout all three books, there is an elegiac sense of mourning for something lost. Each novel has loss at its core, whether of a lover, a home, a family or a means of living. And this loss is amplified by a temporal dislocation of the narrative: it is difficult to place any of the books in a particular period of time, although references to wars and some technologies serve to give vague hints. The overall sense is one of a timelessness about to come to an end. At the end of the trilogy, when Billy Parham, old and drifting, marvels at seeing the rare sight of a cup hung up by a spring for travellers to drink from, you share his sense of bewilderment at something gone without a chance to mark its passing.

But The Border Trilogy isn’t just about nostalgia for a better, or at least more certain, time. In among the dense, beautiful pages there is much more to be found. Quite apart from the arresting and off-kilter central narratives there are many small stories and vignettes sewn into the fabric of the novels. Itinerant priests muse on the nature of dedication; hobos ask deep questions about the nature of reality; circus performers tell stories of love; prison-bound gangland bosses explore the ethics of suffering. Where The Road is purposefully grey and empty, The Border Trilogy, for all its enduring love of sparse desert landscapes, is full of the horrific strangeness of life. These aren’t easy reading books, but they are books you should read.

The article I had published in the June edition of Linux Journal has just become freely available on their website. You can read it here

In the past I’ve built PCs for music recording purposes and designed them to be as quiet as possible by using large case fans, passive CPU coolers, and super-efficient power supplies. In this case, I had the PC and whatever was in my spares box at my disposal. Here’s how I quieted it down.

Remove surplus components

This PC used to be a graphics workstation, so it had a beefy graphics card as well as a pair of hard drives in a RAID array. I don’t need any of that for a server, so I removed all the extraneous components I could from the case. Fewer components means less power draw, hence less heat to remove from the case. Fewer cables connecting components to the motherboard means fewer obstructions to smooth airflow.

Ensure fans are using speed control

Most modern motherboards offer a speed control pin for fans. These can be used to slow down case and CPU fans when the box isn’t so hot. Slower fans mean less noise.

Install lm-sensors and fancontrol

lm-sensors provides hardware monitoring support in Linux, and fancontrol is a script which works with the sensor data to control the speed of fans in the system.

Ubuntu packages both, so you can install them with aptitude:

aptitude install lm-sensors fancontrol

Once the packages are installed, you can set them up for use with sensors-detect. This script works out which fans are attached to which PWM controllers by modifying PWM output settings while you listen for changes in fan speed. It then goes on to figure out the maximum and minimum PWM settings for the fans in your box, and allows you to configure temperature/fanspeed relationships.

Test it out!

With fancontol running, my Ubuntu server is no noisier than the road outside my house. This is an acceptable noise level for me.

However, I wanted to be sure that the fan controller was doing what it said on the tin. As such, I whipped up a simple shell script to log CPU and case temperature against fan speed:


#!/bin/bash
#
# Log temperature information from sensors
#

readonly SAMPLE_INTERVAL=30 # seconds
readonly HEADER_INTERVAL=10 # samples
readonly LOGFILE=/root/templog.csv

while true
do
if test $count -eq 0
then
do_header
count=$HEADER_INTERVAL
fi
do_templog
sleep $SAMPLE_INTERVAL
count=$((count - 1))
done >> $LOGFILE


I tried a quick test of system response by subjecting the CPU to some computational stress:

dd if=/dev/urandom | gzip -c -9 > /dev/null

With a few of these pipelines running in parallel, the CPU stayed more-or-less 100% busy.

Here’s a graph of the system response:

As you can see, the system is behaving as it should. As the CPU is loaded, core temperature increases and fan speed ramps up in response. Removing the CPU load allows the core to cool, and the fan slows down again. Perfect.

Just for fun I left my script running in the background for the next few days. Here’s a graph of the data:

Aside from a few outliers, the fan speed looks pretty much stable until the CPU temperature gets to around 35 degrees, which is the trigger temperature for a speed increase. At that point there is quite a lot of fluctuation in fan speed. It would be interesting to see whether hysteresis could be added to the fancontrol script to compensate for that.

In conclusion

With a little bit of effort you can reduce the noise generated by a standard PC, without necessarily buying any specialist parts. If you’ve got to share a room with that PC, the results can be more than worth it!

I had an old PC on which I wanted to install Ubuntu’s most recent LTS release (“Lucid Lynx”). In ordinary circumstances, this should take about an hour or so. However, it just so happens that my PC has a specific model of NEC DVD drive which is subject to this Ubuntu installer bug.

So what to do?

First off, I tried downloaded the minimal install image and burning that to CD. Some folks on the Ubuntu forum seemed to have success with this approach. But not me. My PC simply turned its nose up at the minimal install CD.

Next, I investigated booting the install image from a USB flash device. I used the most excellent unetbootin tool to extract the ISO image onto a USB flash drive, and asked the PC to boot from that. It worked! Or, at least, it worked insofar as booting the installer. Sadly, one of the installer’s first jobs is to talk to the CD to check the contents of the image. So I was back to square one.

Or was I? After some fruitless banging of my forehead against my desk, I hit upon a cunning workaround. Boot from the USB drive, then insert the CD and let the installer have a look at it. Then, when the installer pauses to ask a question, log in on another virtual terminal, umount the CD drive, and mount the USB device in it’s place. That way, the installer thinks it has the CD mounted, and all the files it needs are there — but it doesn’t have to talk to the CD drive itself, and thereby avoids the bug I tripped over in the first place.

Finally, I have my Ubuntu Server install. And it only took two CDs, a USB flash drive, and an evening of time.

Besides being heart-wrenchingly sad, Poor Kids left me feeling angry and frustrated. The film is crammed with injustice. From the benefits-cutting which left one family prey to short-term loan offers (with the associated prohibitive interest rates), to the utilities companies and hire firms who make low-income families pay over the odds for services, there are countless examples of poor people getting a rough deal simply because they are poor. As an example, If I wanted to take out a personal loan from my bank (the Cooperative), the highest interest rate I could pay would be 20.9% APR. By contrast, if I were forced to use someone like Provident Personal Credit, the lowest interest rate I could pay would be 272.2% APR. I know the finance companies would argue that low income customers represent a higher risk, and the relatively high APR reflects that, but that doesn’t make it fair.

A rough deal from financiers is only the start of it. The kids in the film suffered illness due to poor housing conditions, were bullied at school because they didn’t have the right clothes, were bored on the streets due to a lack of things to do, and felt a lack of self-worth that even drove one girl to attempt suicide. If you’re poor, you get a rough deal from the whole of society.

The thing that saddened me the most about Poor Kids, though, was how little optimism they felt for the future. School seemed less an education than something to be endured, or possibly an opportunity for a free meal. Pessimism about jobs and about the challenges of adulthood abounded. And the terrible thing about that sort of pessimism that it becomes a self-fulfilling prophecy. If you know there is no way out of your situation, you are much less likely to even try.