You are here

Improved RPM submission tool

Project ID: 
13
Current stage: 
Manager: 
Unit: 
What: 

Description: Convert the rpmsubmit tool to use AFS instead of NFS.

Deliverables: AFS repository, appropriately modified rpmsubmit tool and RPM accelerators

Why: 

Customer: Computing staff and users of DIYDICE machines

Case statement: The current rpmsubmit tool is a crude setuid program which copies the submitted RPMs over NFS to the RPM repository and then runs the genhdfile program over those RPMs.

There are a number of problems with this approach :-

  • it has weak security as it uses NFS as transport
  • RPMs can only be submitted from local machines
  • only COs can contribute RPMs

This project will move the RPM repository to AFS and modify the rpmsubmit tool appropriately.

A further project will be proposed to enhance the functionality of the package submission technology :-

  • extend to include support for MacOS and Solaris packages
  • automatically detecting which bucket to use (based on history)
  • allow submitter to add a tag to explain reason for RPM
  • perform simple validation of RPM being submitted (particularly for user-submitted RPMs)
When: 

Status:

31/7/06
Not much work on this happened due to holidays and post-holidays catch up. Alastair is about to start work proper.
2/8/06
Craig and Alastair met to discuss what was possible with AFS
12/9/06
No work has been carried out on this project, primarily due to illness.
2/10/06
Slipped again due to higher priority work. Now aiming for steps 1-4 to be complete by 1/11/06 (rather than 1-6).
7/11/06
Project yet again slipped due to higher priority work. Propose suspending project until completion of 64bit project.
28/09/07
Proposal to reawaken project and amended milestones

Timescales: Propose suspending project until completion of 64-bit project.

Priority:

Time:

How: 

Proposal: Conclusions from discussion with Craig on 2nd August 2007 were :-

  • RPMs and SRPMs will live on a set of replicated AFS volumes
  • there is a preference for avoiding huge volumes, so we will need several volumes
  • the AFS root is likely to be /afs/inf.ed.ac.uk/packages
  • we will need to consider the structure under the above root
  • We need flexible host based access control to the RPMs; AFS host-based access is flakey/primitive so we will continue to deliver RPMs via HTTP.
  • RPMs/SRPMs will not be openly (ie beyond informatics) readable via AFS
  • the two existing RPM accelerators will now run apache with identical configs
  • to allow apache to serve RPMs, will need to create a new principal with appropriate rights and use kstart as with backup server(s)
  • might need to hack apache component to do the relevant kinit as "packages" principal.
  • pezenas (now figgy) will become redundant

Discussion between Alastair and Simon on 31/07/07 :-

  • one volume per bucket (eg fc5/dice)
  • having one volume per bucket means that filesystem structure (for rpmsubmit) can be different from structure for HTTP access.
  • RPM accelerators will just use large local AFS client cache for local caching of RPMs
  • initially just using host based access control to give access to RPM accelerators.
  • RO copies of volumes for backup purposes. Will look later at using these for performance
  • One question - who creates the rpmlist files ?

Resources:

  • estimate 10 working days
  • RAIDed and replicated AFS volume for repository - current use is 100Gb, but this is currently 93% full, so 150Gb is probably more appropriate.
  • Some expertise from the Services Unit will be required to setup an AFS volume and advise on AFS ACLs

Plan:

  1. design structure within /afs/inf.ed.ac.uk/packages (2 days)
    Two views :-
    • view from rpmsubmit - by platform
    • view from RPM servers - by layer (eg DICE, ED, LCFG)
  2. arrange for packages AFS partition to be created (1 day services-unit)
  3. for one initial platform - create and populate AFS volumes apply appropriate access controls (1 day)
  4. identify minimally required changes to rpmsubmit, implement and test (1 day )
  5. author new program to generate rpmlist files and release any changed AFS volumes (5 days)
  6. identify required changes to RPM servers, implement and test (3 days)
  7. produce script for creating volumes for new platforms and for deleting old volumes (2 days)
  8. create and populate AFS volumes for remaining platforms (1 day)
  9. add new functionality to rpmsubmit (2 days)
  10. deploy new repository, rpmsubmit and RPM servers (2 days)
Other: 

Dependencies:

  • provision of AFS volume for repository (see Resources)
  • expertise from Services Unit (see resources)

Risks:

URL: https://wiki.inf.ed.ac.uk/DICE/MPUProjectProposalRPMsubmit

Milestones

Proposed date Achieved date Name Description
2007-09-21 2007-09-21 Stage 1 Design repository structure
2007-09-21 2007-09-21 Stage 2 Arrange for creation of AFS partition for RW copy of repository
2007-09-21 2007-09-21 Stage 3 Populate repository with one complete platform for testing
2009-04-17 2009-04-22 Stage 4 Identify minimal changes to rpmsubmit, implement and test
2009-05-13 2009-05-06 Stage 5 Create a program/daemon? to periodically generate rpmlist files and release any changed AFS volumes
2009-05-15 2009-05-20 Stage 6 Identify required changes to RPM web servers, implement and test
2009-05-15 2009-05-27 Stage 7 Scripts for creating new volumes (for new platforms) and deleting old volumes
2009-08-01 2009-06-01 Stage 8 Populate repository with remaining platforms
2009-07-15 2009-06-08 Stage 10 Procure new server hardware for service
2007-09-21 2007-09-21 Stage 7.5 Arrange for AFS RO copies of repository
2009-10-01 2009-06-08 Stage 13 Clean up rpmsubmit and add extra functionality (check existence of RPMs already in buckets, implement remove function)
2009-10-22 2009-06-15 Stage 11 Produce procedures documentation
2009-09-30 2009-07-01 Stage 12 Deploy into service
2009-12-02 Signoff Signoff