Quite some time ago, in my post on build farms and BASH queues I described a simple continuous-integration style build server written in BASH. That post concentrated on the IPC queue mechanism I used in my scripting.
Well, since I wrote that post, the build server code has been made available and now resides on github:
So now you can go and check it out in detail!
How it all works 1: theory
Now that babs can be viewed in its entirety, I thought I’d write a short overview of how the system works. For a BASH script it’s reasonably sophisticated, while as a program it is fairly simple, making it an interesting study.
The design of babs revolves around a few key concepts:
- Configuration is read from a file written in the INI file format. All configuration lives in there, from specification of the repositories to monitor, to a definition of the machines to use for building code.
- After the master server has been started (e.g. from an init script), it can be prompted to do work by adding events to its input queue. The babs script takes command line arguments to make that easy. For example, you ask the server to scan the repositories it is monitoring like this:
Within the babs codebase, event queues are just files on disk. Write access to the files is serialised using lockfiles. The server can block on events using inotify to sleep until the event queue file changes.
- The master server has a “build pool” which consists of a number of machines that will run build jobs. Jobs are shared out between machines on the basis of who is least busy (babs’ idea of “busy” is based on how many jobs each machine has outstanding). The job runner machines each run a slave babs process, and work around the same input queue mechanism that the master babs process uses. Communications between the master and slave machines is carried out over ssh using password-less keys.
- babs maintains build state information in a number of lists. As with the event queues, lists are just files on disk, protected using lockfiles. Each list entry consists of a line of text, starting with a unique ID, and followed by space-delimited arbitrary text. Different babs lists contain different sets of information.
- The master server provides various sets of information to the user via. a command line interface. The
babs scancommand above is an example of one such interface — although in real life you would probably run this from a repository checkin hook or a cron job. More useful for human consumption are commands such as:
babs history : to show the build history to date
babs poolinfo : to show information about the babs slave build pool
babs requestbuild : to request a build of a particular code revision
How it all works 2: reality
So much for the theory. How does the process of building a piece of software work in reality? Let’s follow the process through from start to finish.
In the beginning, the administrator creates the babs .ini file, which details the information about the repositories to check for updates, the scripts to run to actually process a build, and the pool of machines babs can command to carry out the building. While they’re at it, the administrator also sets up a cron job which is responsible for periodically running
babs scan on the babs master machine.
When the babs master machine performs its first scan, it looks up the last revision of the code that it built in its list of builds. Of course, it hasn’t carried out a build at all as yet, so it needs to build the latest code the repository has to offer.
Now that babs has a build to process, it needs to pick a machine to run the build on. To do so, it iterates through its list of job runner machines, and asks each in turn how long its job queue is. The machine with the shortest queue is picked for the task. In the case of the first ever build, none of the job runner machines will be busy, so the first machine on the list will be picked.
Once a build machine is selected from the pool, babs creates a set of scripts to tell the build machine what to do. These scripts consist of a build script, and a report script. The former consists of the build script the administrator originally created for the project, plus a bit of babs boilerplate code to set up a build directory and a some useful environmental variables (e.g. for the code revision). The latter is a simple “callback” script that the build machine can use to report his results back to the babs master server. The master server then passes the build and report scripts to the build machine, makes a note of the fact that the build is in progress, and goes back to sleep.
At this point, the story moves over to the build machine. Its job is very simple really: it runs the build script, capturing the output to a build log, and then reports the results back to the babs master using the report script. The build itself may be arbitrarily complex, since it’s driven entirely by a project-specific build script. Neither babs master nor the build machine care about this: they just run the script.
When the build machine reports back to babs master, the updates its list of builds in progress to note that the build has been completed, and pulls the report from the build machine to save it for posterity. Finally, it performs some internal housekeeping (for example, updating its build history file), and then (optionally) emails a build summary to one or more recipients. Then it goes back to sleep again waiting for the next scan event.
There you have it! A multi-process, event-driven, CI build server/client script, implemented in around 1600 lines of shell (1300 lines if you ignore comments), which provides somewhat-reusable libraries for queues, lists, events, and .ini file parsing.
I’m sad I never really got a chance to use babs in anger: the testing I performed during development on small pools of build machines suggested it would work fairly well. However, it was a fun project to implement, and now that the code is available to everyone I hope someone will find it useful, either as a fun project to play with, or as a build server for when they absolutely need something quick, to the point, and implemented in BASH :-)