You are here

LCFG buildtools rewrite

Project ID: 
54
Current stage: 
Manager: 
Unit: 
What: 

Description: The existing scripts for building LCFG packages (buildtools) are starting to creak seriously. In particular, for non-Linux builds: Solaris packages are created by an old, fragile student script, and Mac packages are created with a hack which can generate incorrect file lists.

The functionality of the existing buildtools is good, but it is old and should probably be rewritten if it is to be sustainable.

Deliverables:

  • Produce a new interface for querying system information at runtime.
  • Use the new runtime query system to move various things which are hardwired into scripts at compile time into runtime config options.
  • Split the release and build systems.
  • We will have better support for non-Linux and non-Redhat based platforms. Including MacOSX support (and maybe Solaris).
  • A system which is easier for external sites to deploy such that they can contribute to LCFG development.
  • The release system will have the ability to support multiple revision control systems (or none).
  • There will be a test-suite system which is less complex and more intuitive.
  • There will be a system for building and testing on multiple platforms in parallel, possibly a "build farm".
Why: 

Customer: The primary customer is the COs in the School of Informatics. There are also external users of LCFG, all of whom currently have difficulties building packages due to the way buildtools is tied to our particular choice of infrastructure (e.g. IS, EPCC, SEE).

Case statement:

The current buildtools system was not really designed, it has really evolved from a set of scripts created by Paul for his own use. The system is very closely related to our choice of infrastructure - e.g. it uses a CVS repository, and builds RPM packages, which limits uptake outwith Informatics. The scripts are showing their age and need to be completely redesigned to be more robust, modular and extensible.

Within Informatics the number of platforms we support has recently increased dramatically. We now have FC3, FC5 (32 and 64 bit), FC6 (32 and 64 bit) and soon we will have SL5 (32 and 64 bit). We also have a need to build packages on Solaris and MacOSX. Consequently we require a new build system which more adequately aids our support of multiple platforms. This needs to be able to build, test and submit packages in parallel on multiple platforms.

When: 

Status:

27/8/07
Project proposal being fleshed out
5/9/07
Proposal put to Development Meeting
3/12/07
Implementation stage fleshed out

Timescales: We expect to begin this project towards the end of September. Firm timescales for the deliverables will be set after the evaluation is complete.

Priority:

High priority.

In particular the requirement to create a build-farm which can be used by Computing Officers within Informatics is very important. We now have far too many different supported platforms to cope with continuing to do manual software builds on each.

Time:

How: 

Proposal:

The first part of the project is really an evaluation phase which should provide some guidance as to how buildtools system should be redeveloped.

  1. Look at how components use and abuse the current build system and come up with proposals for new features and improvements.
  2. Examine what needs to be done to improve the support for non-Linux, non-Redhat systems (including MacOSX support and maybe Solaris). It might be very useful to select Debian as a platform for prototyping the new system
  3. Discuss buildtools with interested parties to find out what they would like to see incorporated into the new buildtools.
  4. Investigate possible solutions for providing system information to components and scripts at runtime.
  5. Investigate possible solutions for the need to provide a parallel build and test infrastructure.
  6. Produce a set of of proposals based on this evaluation to take the project into the implementation stage.

The evaluation phase has been completed, a short summary of the collected thoughts and comments is available. The details for the implementation stages are:

  1. Further cmake investigation

    Carry out further investigation into cmake and develop a set of usage
    examples. This should lead to a short talk that can be given to COs in
    a Development Meeting.

  2. Create new build system

    If cmake is acceptable then a new build system should be created based
    on that software.

  3. Release Tools

    Develop a new set of tools to do release version management for
    CVS. It must be possible to use these independently of the new build
    tools system.

  4. Solve compile-time macro problems

    Mainly this involves developing a new LCFG component to contain system
    information and then extending the ngeneric tools to make it simple to
    access the data. Also required is the converting the core components
    to use this information source.

  5. Convert MPU components to new build tools

    In order to prove that the new build tools are succesful we will
    convert the components managed by MPU to the new system. This should
    hopefully allow the discovery of any remaining issues and result in
    them being fixed before the process of wide-spread adoption is begun.

  6. Investigate RPM build-farm technology

    Look into what is required to install and configure a fedora
    build-farm and produce a plan for the next milestone.

  7. Create a build-farm for RPM-based software

    Based on the results of the previous investigation work install a
    build-farm for use by COs which allows the automated building of
    RPM-based software on all our supported fedora and scientific linux
    platforms. This may require the development of LCFG components to
    manage the system.

  8. Investigate a generic build-farm system

    It is desirable to have a system which can manage the automated
    building of software for any platform not just those which are
    RPM-based. It should be possible to use this to feed build-farms for
    Redhat and Debian based systems and also manage the building of
    packages on MacOSX and Solaris. It is not clear how much work this
    would require so at this stage only an investigation is proposed. It
    may well be that this leads into a separate development
    project.

Resources:

Plan:

  1. We will provide new interfaces (for shell, perl and maybe C) to allow the querying of information which describes the platform on which a component is running. For example, this would give details of the paths for the different constituent parts of the LCFG system, it could give info on the platform name, cpu architecture, etc..
  2. We will do an evaluation of what COs (internal and external) require from a new buildtools system. As well as talking to COs this will involve examining a range of components to see how the current tools are being used (and abused). This will help us design a modular, extensible system which will, hopefully, satisfy all our requirements.
  3. We will construct a set of scripts and makefiles which will support these requirements. If possible it would be good to provide backwards compatibility with the old system so that we do not need to continue providing support for the old system once the new one is available.
  4. We will investigate "build farm" technology to see what best suits our need to build components and carry out suites of tests on many platforms in parallel.
  5. We will implement a parallel-build system based on the results of the investigation into build-farm technology.

Some of these tasks are not dependent on each other and could be done in parallel such as 1,2 and 4.

Other: 

Dependencies: No dependencies.

Risks:

Milestones

Proposed date Achieved date Name Description
2007-12-03 2007-11-30 1.0 Produce proposals for implementation phase
2008-05-31 2008-03-31 2.0 Further cmake investigation
2008-07-14 2008-07-31 3.0 Create new build system
2008-03-04 2008-02-29 4.0 Release Tools
2008-01-15 2007-12-21 5.0 Solve compile-time macro problems
2009-03-04 2008-10-31 6.0 Convert MPU components to new build tools
2008-05-31 2008-02-29 7.0 Investigate RPM build-farm technology
2009-03-04 2008-10-31 8.0 Create a build-farm for RPM-based software - moved to the build farm project.
2009-03-31 2008-12-31 9.0 Investigate a generic build-farm system - moved to the build farm project
2009-03-16 2009-03-30 10.0 Add subversion support for building packages.