You are here

Software build farm

Project ID: 
115
Current stage: 
Manager: 
Unit: 
What: 

Description: We really need a way of building software for multiple platforms in parallel in an automated fashion.

Deliverables: System for building software packages (right now this is just SRPMs) for multiple platforms and architectures with minimal effort required from the users.

Why: 

Customer: Informatics computing team. Potentially also external users so they should not be ignored but they are not a priority.

Case statement:

Currently we have to manually build software packages for all target systems. This requires login access to a build host for each platform and the ability to configure the target platform so that it has all the necessary build requirements installed.

Even when only a single platform is being supported this requires access to two systems (one per architecture). This makes the whole process slow, inefficient and liable to lead to problems (such as forgetting to build or submit packages for a particular architecture).

Tools, such as mock, are available for building RPMs within a chroot which require only one system. This makes the process much simpler but still requires more user input than might be strictly necessary. Ideally the user would submit the source package once, possibly without full knowledge of the supported platform and architecture set. The system would then farm out the build process to all supported platforms in a fully automated fashion. This would lead to a more efficient process which is much less liable to user errors.

When: 

Status:

Timescales: The project is expected to require 4 weeks of effort from beginning development to final deployment. If no other major projects conflict with this work it is expected to be completed by the end of November 2010.

Priority:

Time:

How: 

Proposal:

We originally investigated using the Redhat koji software build system. This proved to not really be a good fit with our requirements as it's a heavyweight system more designed to be for building an entire distribution. It expects that all SRPMs for a platform will be imported. It can rebuild whole sets of packages based on their dependencies which is nice but a bit of an overkill for what we need. It also had some various problems with authentication, as it's kerberos support was based around people having redhat accounts. Although this could have been fixed with some effort, it didn't give the impression that other external users had really been properly considered.

Requirements

It must be possible to have multiple machines servicing a single platform. This allows efficient handling of large scale rebuilds, such as when we port LCFG/DICE to a new platform.

It must be possible to have a build job which consists of multiple source packages. The build products from one source package should be made available immediately to fulfil build requirements for the following packages within the same job.

It must be possible to restrict the target platforms for a build job when, for example, it makes no sense to attempt to build software on certain architectures.

It would be nice to have the option of extending the system at a later date to support non-RPM based source packages.

It would be good if this system were to be useful for external groups to deploy. The software should be driven by configuration files and not contain any hard-wired Informatics-specific information.

Design

A command line tool will be provided so that users can submit build jobs into an AFS directory. This has the advantage of being accessible from anywhere and we can use the AFS ACLs to control access.

A daemon will watch the incoming jobs directory. When new jobs appear they will be validated and (if ok) registered in a database as requiring building.

On each build host there will be a daemon which regularly checks the central database for new jobs. When jobs are available it will take the first in chronological order and trigger a build. On successful completion the build products would then be submitted to the usual package repositories. The final status (i.e. success or fail) will be stored in the database and the build log will be stored in a central directory.

There will be a simple web front-end to allow users to query the info stored in the database and the log files from the completed jobs.

Resources: During the development phase there are no big requirements. For deployment we will require a build host for each target platform (could be a VM). There will have to be some AFS space for holding the build jobs and log files from the builds, also needed will be a database to maintain state.

Plan:

Other: 

Dependencies:

Risks:

Milestones

Proposed date Achieved date Name Description
2010-09-05 2010-09-05 1 Design a class to represent a build job and provide a serialisation system for submission and retrieval of jobs.
2010-09-05 2010-09-05 2 Develop a command line tool for submitting build jobs to the incoming directory.
2010-10-27 2010-09-19 3 Design a database schema to hold the registry of build jobs. Also develop a simple perl API for querying and modifying this information.
2010-10-27 2010-09-26 4 Develop a tool for processing the incoming jobs queue which validates and registers jobs.
2010-12-01 2010-11-03 5 Develop a tool for triggering builds of jobs. This must also store the final build status, submit the products of a successful build and store the logs.
2010-11-23 2010-11-10 6 Daemonise the tools used to process the incoming directory and trigger builds.
2011-01-27 2010-11-17 7 Develop a simple web front-end for querying the database and viewing the build logs.
2011-05-10 2010-11-24 8 Code-level testing and documentation. (This will be ongoing throughout the development process).
2010-04-11 2010-11-30 9 User level documentation.
2011-05-10 2010-12-07 10 Deployment and user acceptance testing.
2011-01-21 2010-12-20 11 Develop LCFG components to configure the client and server.
2010-12-20 12 Add basic passive nagios monitoring for the daemons.