You are here

large file distribution and management system

Project ID: 
196
Current stage: 
Manager: 
Unit: 
What: 

Description: The project will develop a mechanism for distributing and managing large files/packages efficiently to DICE workstations. Subject to appropriate requirements it should extend this to distributing files across the School, Unitersity and the Internat.

Deliverables:

  • A system for managing large files on DICE machines, it should at least.
    • support efficient file distribution across DICE via torrent/multicast type technologies
    • Allow distributed files to be managed (untarred, removed...) via standard DICE/LCFG tools
    • Allow jobs to be scheduled to use the network at off peak times, whilst still allowing for immediate distribution.
  • A system for securely(?) distributing large files across the school non dice machines, University or internet
  • An interface allowing select users to manage the above.
Why: 

Customer: For the DICE part the C(S)Os, for the extended part staff. Students and the general public may be users of the system

Case statement: We have a number of files and applications on DICE which are too big to ship in rpms and which cannot be broken down into a number of smaller rpms. NFS and AFS are inappropriate technologies for shipping these files to a large number of machines in a short period, torrent or multicast would be more suitable technologies for such a deployment. The Virtualised DICE project is likely to produce more large files and we need a machanism to deploy these files to DICE machines and to manage the files on the DICE machines.

Given such a system it is invisioned that it could be used to distribute the same or similar files to non dice machines in the school, within the university or across the internet.

It is also possible that various research groups may wish to take advantage of having files or datasets available via such technologies.

When: 

Status: proposal

Timescales: Before the next academic session would be nice

Priority:

Time:

How: 

Proposal: Find out what the requirements are within the school for a torrent/multicast type file sharing service and how appropriate it would be to deploy. Assuming this requirement at least implement a system which would allow us to manage the deployment of large (>2GB files) across DICE. Optionaly deploy a system which would allow staff to make files available to non DICE machines.

Resources: Staff time:
3 days to gather requirements
1 week to look at available technologies
1-2 weeks to implement distribution system
2 weeks to implement management system

Members of the services, MP and infrastructure units would be needed for sort periods on a consultancy basis to determine the best ways to integrate with existing systems.

Possibly further 2 weeks to implement something to distribute files beyond DICE.

Hardware
Depending on the implementation there may be a need for disk space on a number of servers which are geographicall and topologically distributed. It's expected that there would be minimal new hardware requirements.

This project is not likely to be a good candidate for virtualisation.

Plan: The exact plan depends on how widely we want to make this available.

  • Inverstigate requirements from C(S)Os and non C(S)Os.
  • Investigate file transportation technologies.
  • Integrate management of transportation technology with LCFG/DICE.
  • produce documentationfor C(S)Os

If non DICE distribution is required and both services can use the same platform then:

  • Check with IS about transport technology across EDLAN if access from outside school is required.
  • Develop interface to allow staff to distribute files.
  • Package or document client installations for major OSs (linux, windows, OS, Solaris.
Other: 

Dependencies: None

Risks: Even if used correctly this could be a wonderful denial of service mechanism against the school network. We should also be aware of Copyright issues with any external service.

Milestones

Proposed date Achieved date Name Description