You are here

Improved LCFG Distribution

Project ID: 
7
Current stage: 
Manager: 
Unit: 
What: 

Description: The distribution of LCFG products (the headers and component packages) needs reworking to provide better means of access for external (to Informatics) users.

Deliverables: There are a number of tangible deliverables, these would be available for each system type we support e.g. FC3, FC5, Mac OSX, Solaris. They would also be available on a "stable release" basis so that a user can select which version of the headers and components they want to utilise. These are :-

  • LCFG headers via the web
  • LCFG headers via rsync
  • LCFG component packages via the web
  • LCFG install ISO (FC3, FC5)
Why: 

Customer: External users of LCFG, such as EUCS and SEE.

Case statement: The LCFG products are distributed to external users via the website http://www.lcfg.org/. Currently the published stable releases are manually added to the website in an ad-hoc fashion. We need a consistent and effective way of delivering the new weekly stable releases. Also there are currently no published LCFG components for Solaris or Mac OSX. The School of Informatics has much to gain from improving the public profile of LCFG and a necessary step in this process is to improve delivery of the  core  headers and components.

When: 

Status: Current status of the web interface can be seen at http://www.lcfg.org/

2006/11/08
Most of the first two milestones now completed
2007/01/26
Most milestones up to and including 4.0 are now completed.

Timescales: The first target is to complete the first two complete milestones "basic data products" and "web interface" by the Development Meeting on Wednesday 8th November.

Priority: High priority proposal.

Time: It is estimated that it will take 19 days work to complete the first 5 stages of the project. The effort required to support MacOSX and Solaris is currently unclear as separate work will need to be carried out first to get packaging for these systems into good shape. Some of the work for stage 1 has already been carried out during the evaluation stage as it was the most practical way to evaluate the scope of the issues involved.

How: 

Proposal: This can be broken down into a set of milestones as follows:

  1. Basic data products [6 days]
    • Create basic LCFG component infrastructure which can be used to manage the data product generation
    • Develop script that can be used to automatically mirror new LCFG releases.
    • Develop script to package up the headers in tar.gz and RPM formats.
    • Develop script to cache necessary RPMs for the supported Fedora based OSes and create categorised lists of the packages required.
    • Develop script to mirror LCFG Fedora install CD images
  2. Web interface [4 days]
    • Extend component to allow management of the web interface.
    • Create CGI scripts and templates to provide interfaces to the generated data products
    • Create CGI script to generate the "wget" url files
  3. Web interface extensions [5 days]
    • Add script to generate RSS feeds listing the changes between previous and current releases.
    • Add ability to get RPMs using yum.
    • Add ability to diff the headers of two releases.
    • Script to extract documentation from the RPMs - (both for reading via the web and for the lcfg guide)
    • Support other access methods [2 days]
    • Support rsync access as well as http
  4. Move to new server [2 days]
    • Move lcfg.org www and rsync over to the diydice server so everything is on the same machine and it can all be publically available
  5. Add support for MacOSX packages [unknown]
  6. Add support for Solaris packages [unknown]

Resources: Knowledge of web development techniques such as how to create cgi scripts, template web pages

Physical resources:

  • web server (probably somewhere to run a test service for a while as well)
  • file space to store generated data products

Plan:

  • Develop some way of categorising components so we can ship core code as well as (several levels of?) contrib code.
  • Create a script (or set of scripts) so that the external user-facing interfaces to LCFG (website and rsync) tracks the latest stable versions, it must be zero effort from our point-of-view once set up. There must be support for "releases" so that we can release consistent versions and there must be support for the various platforms.
  • Alter the website to display easy to use interfaces to the headers and components that form the new releases and platforms.

A cron job will run each night to look in svn for new releases of LCFG. Whenever a new version appears it will automatically generate the data files, copy across CD images and component RPMs and build  wget  lists of all the RPMs for ease of downloading.

There will be an index web page which lists all available releases of LCFG. This will be generated via a cgi script using perl Template Toolkit (TT) so that when new releases appear they are automatically shown.

For each stable release of LCFG we will present a web page, per supported OS, which gives easy access to the install CD (where relevant), data files (tar-file of all the headers and package lists), and all the required components (RPMs, etc). This will be presented in a similar manner to the way it is done now on www.lcfg.org.

Existing tools such as the perl module LCFG::RPMCache can be used to parse the RPM lists and generate the pages with the package categories being specified via the LCFG category pragma. It should also be possible to re-use some of the code from the lcfg-orgbuilder component which was previously used to generate the LCFG downloads website.

The method of delivery will be slightly altered, each web page will be templated with TT and the final view created via a cgi script with some caching where performance makes it required.

In order to begin delivering something tangible quickly we will initially focus on creating pages for the Linux platforms (probably FC3 and FC5). We will keep in mind the future need to provide pages for Solaris and Mac.

Other: 

Dependencies: LCFG release management project. Package repositories for Solaris and MacOSX

Risks:

URL: https://wiki.inf.ed.ac.uk/DICE/MPUProjectProposalLCFGDist

Milestones

Proposed date Achieved date Name Description
2006-11-08 2006-11-03 1.1 Create basic LCFG component infrastructure which can be used to manage the data product generation.
2007-02-05 2006-11-07 1.5 Develop script to mirror LCFG Fedora install CD images
2006-11-08 2006-11-08 2.1 Extend component to allow management of the web interface.
2006-11-08 2006-11-08 2.2 Create CGI scripts and templates to provide interfaces to the generated data products
2006-11-08 2006-11-03 1.2 Develop script that can be used to automatically mirror new LCFG releases.
2006-11-08 2006-11-07 1.3 Develop script to package up the headers in tar.gz and RPM formats.
2006-11-08 2006-11-07 1.4 Develop script to cache necessary RPMs for the supported Fedora based OSes and create categorised lists of the packages required.
2006-10-08 2006-11-07 2.3 Create CGI script to generate the "wget" url files
2007-02-06 2006-12-13 3.1 Add script to generate RSS feeds listing the changes between previous and current releases.
2006-11-17 2006-12-13 3.2 Add ability to get RPMs using yum.
2006-11-17 2006-12-13 3.3 Add ability to diff the headers of two releases.
2006-11-16 2006-12-13 3.4 Script to extract documentation from the RPMs - (both for reading via the web and for the lcfg guide)
2006-11-16 2006-12-13 3.5 On the individual release page mark up packages which are new or have been updated.
2006-12-08 2006-12-13 3.6 Support other access methods - rsync
2006-12-21 2006-12-02 4.0 Move lcfg.org www and rsync over to the diydice server so everything is on the same machine and it can all be publically available
2007-12-31 5.0 Add support for MacOSX packages
2007-10-11 2007-06-30 6.0 Add support for Solaris packages