You are here
Improved RPM submission tool
Fri, 01/25/2013 - 15:45 - boss
Project ID:
13
Current stage:
Manager:
Unit:
What:
Description: Convert the rpmsubmit tool to use AFS instead of NFS.
Deliverables: AFS repository, appropriately modified rpmsubmit tool and RPM accelerators
Why:
Customer: Computing staff and users of DIYDICE machines
Case statement: The current rpmsubmit tool is a crude setuid program which copies the submitted RPMs over NFS to the RPM repository and then runs the genhdfile program over those RPMs.
There are a number of problems with this approach :-
- it has weak security as it uses NFS as transport
- RPMs can only be submitted from local machines
- only COs can contribute RPMs
This project will move the RPM repository to AFS and modify the rpmsubmit tool appropriately.
A further project will be proposed to enhance the functionality of the package submission technology :-
- extend to include support for MacOS and Solaris packages
- automatically detecting which bucket to use (based on history)
- allow submitter to add a tag to explain reason for RPM
- perform simple validation of RPM being submitted (particularly for user-submitted RPMs)
When:
Status:
- 31/7/06
- Not much work on this happened due to holidays and post-holidays catch up. Alastair is about to start work proper.
- 2/8/06
- Craig and Alastair met to discuss what was possible with AFS
- 12/9/06
- No work has been carried out on this project, primarily due to illness.
- 2/10/06
- Slipped again due to higher priority work. Now aiming for steps 1-4 to be complete by 1/11/06 (rather than 1-6).
- 7/11/06
- Project yet again slipped due to higher priority work. Propose suspending project until completion of 64bit project.
- 28/09/07
- Proposal to reawaken project and amended milestones
Timescales: Propose suspending project until completion of 64-bit project.
Priority:
Time:
How:
Proposal: Conclusions from discussion with Craig on 2nd August 2007 were :-
- RPMs and SRPMs will live on a set of replicated AFS volumes
- there is a preference for avoiding huge volumes, so we will need several volumes
- the AFS root is likely to be /afs/inf.ed.ac.uk/packages
- we will need to consider the structure under the above root
- We need flexible host based access control to the RPMs; AFS host-based access is flakey/primitive so we will continue to deliver RPMs via HTTP.
- RPMs/SRPMs will not be openly (ie beyond informatics) readable via AFS
- the two existing RPM accelerators will now run apache with identical configs
- to allow apache to serve RPMs, will need to create a new principal with appropriate rights and use kstart as with backup server(s)
- might need to hack apache component to do the relevant kinit as "packages" principal.
- pezenas (now figgy) will become redundant
Discussion between Alastair and Simon on 31/07/07 :-
- one volume per bucket (eg fc5/dice)
- having one volume per bucket means that filesystem structure (for rpmsubmit) can be different from structure for HTTP access.
- RPM accelerators will just use large local AFS client cache for local caching of RPMs
- initially just using host based access control to give access to RPM accelerators.
- RO copies of volumes for backup purposes. Will look later at using these for performance
- One question - who creates the rpmlist files ?
Resources:
- estimate 10 working days
- RAIDed and replicated AFS volume for repository - current use is 100Gb, but this is currently 93% full, so 150Gb is probably more appropriate.
- Some expertise from the Services Unit will be required to setup an AFS volume and advise on AFS ACLs
Plan:
- design structure within /afs/inf.ed.ac.uk/packages (2 days)
Two views :-- view from rpmsubmit - by platform
- view from RPM servers - by layer (eg DICE, ED, LCFG)
- arrange for packages AFS partition to be created (1 day services-unit)
- for one initial platform - create and populate AFS volumes apply appropriate access controls (1 day)
- identify minimally required changes to rpmsubmit, implement and test (1 day )
- author new program to generate rpmlist files and release any changed AFS volumes (5 days)
- identify required changes to RPM servers, implement and test (3 days)
- produce script for creating volumes for new platforms and for deleting old volumes (2 days)
- create and populate AFS volumes for remaining platforms (1 day)
- add new functionality to rpmsubmit (2 days)
- deploy new repository, rpmsubmit and RPM servers (2 days)
Other:
Dependencies:
- provision of AFS volume for repository (see Resources)
- expertise from Services Unit (see resources)
Risks:
URL: https://wiki.inf.ed.ac.uk/DICE/MPUProjectProposalRPMsubmit
Milestones
| Proposed date | Achieved date | Name | Description |
|---|---|---|---|
| 2007-09-21 | 2007-09-21 | Stage 1 | Design repository structure |
| 2007-09-21 | 2007-09-21 | Stage 2 | Arrange for creation of AFS partition for RW copy of repository |
| 2007-09-21 | 2007-09-21 | Stage 3 | Populate repository with one complete platform for testing |
| 2009-04-17 | 2009-04-22 | Stage 4 | Identify minimal changes to rpmsubmit, implement and test |
| 2009-05-13 | 2009-05-06 | Stage 5 | Create a program/daemon? to periodically generate rpmlist files and release any changed AFS volumes |
| 2009-05-15 | 2009-05-20 | Stage 6 | Identify required changes to RPM web servers, implement and test |
| 2009-05-15 | 2009-05-27 | Stage 7 | Scripts for creating new volumes (for new platforms) and deleting old volumes |
| 2009-08-01 | 2009-06-01 | Stage 8 | Populate repository with remaining platforms |
| 2009-07-15 | 2009-06-08 | Stage 10 | Procure new server hardware for service |
| 2007-09-21 | 2007-09-21 | Stage 7.5 | Arrange for AFS RO copies of repository |
| 2009-10-01 | 2009-06-08 | Stage 13 | Clean up rpmsubmit and add extra functionality (check existence of RPMs already in buckets, implement remove function) |
| 2009-10-22 | 2009-06-15 | Stage 11 | Produce procedures documentation |
| 2009-09-30 | 2009-07-01 | Stage 12 | Deploy into service |
| 2009-12-02 | Signoff | Signoff |