DICE development projects - cc

Evaluate move of LCFG configuration from svn to git

ascobie — Fri, 23 Aug 2013 14:19:30 +0000

Project ID:

289

Current stage:

2_Evaluation

Manager:

Unit:

mp-unit

Summary:

We want to know if it is possible and desirable to move the LCFG configuration data (headers) from subversion to git

What:

Evaluate the possibility of switching from subversion to git for the management of the LCFG configuration data. Test a prototype based on a snapshot of the current data. Ensure that we can continue to have an efficient, low-overhead weekly testing/stable release cycle.

Why:

Current Situation

The LCFG configuration files (headers and package lists) are currently stored in a subversion revision-control repository which is run by the School of Informatics. This repository can be split down
into 3 separate strands:

lcfg: Public data which will create sensible default configuration
policies for the general case
ed: Public data which configures computers to suit the University of
Edinburgh computing environment
dice: Private data which configures computers to suit the School of
Informatics computing environment

The separation of public and private data, which is all stored in the same single repository, is achieved using subversion ACLs. These ACLs ensure that only computing staff from the School of Informatics can read or modify files related to the dice environment. Further to these access restrictions, although the public data is readable by any user it is only modifiable by Informatics computing staff.

Each week a snapshot of the development "trunk" is taken and tagged as a new release. That release is first deployed to a number of testing machines. Once a release has been tested and any problems resolved it is tagged again as the stable release. The full stable release is rolled out to DICE machines, the public portion (the lcfg and ed sections) is bundled up and pushed to the LCFG public repository (http and rsync). Downstream users, such as the MDP team in IS, then test and apply the changes to their own systems.

Currently the only way in which external users can contribute new features or improve existing configurations is to create a patch file containing the required changes and then submit a bug report in the LCFG bug tracker. This is a rather unwieldy process which results in external contributors being treated as second-class citizens. As well as creating a situation where contributing to the LCFG project is more difficult than it should be it also creates more overhead for Informatics staff. Each patch has to be reviewed and then manually applied to the repository, there is no automation to simplify the process or assist in reducing the risk of patches being badly applied. Although not impossible to do with the subversion revision control system this is not a trivial task.

Broadening the Community

There has never been an explicit policy of excluding external users. Rather the current situation has occurred naturally because the project was created and managed by Informatics computing staff. There simply was not originally any specific requirement for extending write access beyond that team. In recent years the LCFG project has become much more widely adopted across the University of Edinburgh. Each succesful deployment has increased the number of people who know how to use and create LCFG components. If the project is to continue to grow and thrive then contributions must be welcomed and actively encouraged. To encourage more external contributions we should aim to make it possible for anyone to make contributions. Clearly we cannot provide direct write access into the main repository, a good strategy, which is used by many Open Source projects, would be to allow public contributions into a review system. Gatekeepers can then review submissions, respond to the contributors and commit or reject the changes in an efficient, transparent process.

We have previously considered restructuring the LCFG headers repository so that we can allow external contributors direct write access to the headers for components which they manage. The restructuring is probably a good idea to help improve comprehensibility and discoverability, however it was decided that allowing access like this is too much of a security risk for Informatics. In Informatics we have approximately 100 machines which follow the develop release in real time, any mistake to a widely used header could result in us having badly broken machines. This risk is considered acceptable when there is a local CO involved who can check the likely impact of any change and be relied upon to notify others in advance of any risky changes. They are also more easily contacted if a problem occurs. This problem pretty much rules out subversion as a choice for our revision control system if we wish to involve external contributors. A better alternative would be to use a revision control system with a distributed model (e.g. git). This would allow contributors to fork repositories, test their own changes locally and push back changes when ready. We can also then build a review system around that approach which should help raise the quality of all changes.

Good Practices

Beyond just lowering the barriers for external contributors a review-based system has a number of additional benefits. By using a review system for all changes to the configuration files we get "more eyes on the code". It would actively increase the number of people who have an understanding of the purpose of an LCFG component. There would also be a wider awareness of upcoming changes. It can also be a useful tool for helping to tutors new contributors on topics of good-practice such as expected style, syntax, config layout, etc, when it is not possible to work side-by-side in "real life". A review system also encourages users to use good, descriptive changelogs, there's no way to lazily submit changes with an overly terse, unhelpful or just plain wrong changelog entry.

How:

Import a snapshot of the current LCFG configuration data (the headers) into a git repository.
Configure the gerrit review system for the git repository
Get local COs to play with the system and give feedback
Get external contributors to play with the system and give feedback
Investigate how the dice headers could be split from the public lcfg and ed levels.
Investigate how we would do the weekly testing/stable releases

Effort estimate:

Two weeks

Other:

Dependencies:

Risks:

Absence Reporting Management

boss — Fri, 25 Jan 2013 15:46:26 +0000

Project ID:

257

Current stage:

3s_Stalled

Manager:

Unit:

mp-unit

What:

Description: A combination of factors; external compliance via UKBA and HR legislation or compliance with US accounting rules necessitate the need to update absence management procedures within Informatics. On the flipside the intention is to introduce the lightest touch possible reporting and management requirements.

One option is to introduce a webform to simplify absence reporting. An example from another School would be https://www.wiki.ed.ac.uk/display/SBSIntranet/Home+Page in School of Biological Sciences (but note secure access permissions to site). This tool allows an SBS staff member to securely notify HR of absence.

Overall requirements

a need to scope remit of webform, noting that the simpler the form the better
consideration of linked email notifications (to whom ?, saying what ?)
identify website location and login permissions as a submitter
a need to verify all against technical contraints and Data Protection Act.
data access to InfHR and then Oracle.
webform construction and data repository.

Deliverables: Effectively a web form for recording absence. Something/somewhere to hold the data submitted via the web form.

Why:

Customer: Staff

Case statement: Legislative requirement to do something.

When:

Status:

Timescales: Required as soon as possible as current position is non-compliant with University policy or external requirements.

Priority: See timescales.

Time:

How:

Proposal: See plan.

Resources:

Plan: Initially just scope requirements and propose an implementation. Consider whether a standard WebMark form (which can be fed with external data and can send results by email) may be the quickest option.

Other:

Dependencies: Unclear if there are dependencies on any external data to drive web form and where to send/store the output of the web form.

Risks: Needs to be available very soon. Currently still a fuzzy requirement.

Milestones

Proposed date	Achieved date	Name	Description

DICE virtual image with teaching software for students - trial

boss — Fri, 25 Jan 2013 15:46:23 +0000

Project ID:

239

Current stage:

3_Implementation

Manager:

Unit:

mp-unit

What:

Description: Produce a virtualised image with all teaching software - or as much as would be feasible given the awkwardness of some of our teaching software - for students to install on their personal laptops and home PCs.

This project will concentrate on producing a useable image that can be used to gauge interest. The image distribution need not be scalable at this stage. Should there be sufficient interest, a follow-on project would produce a scalable image distribution system.

There is more than one possible approach to this exercise.

Produce an image based roughly on a DIY-DICE install. Updated, say, twice a semester.
Produce an image based on stock SL6 but using DICE repos for additional software. Updated, say, twice a semester - but could allow remote updates from DICE repos using yum. Advantage is that this is closer to stock SL6 and hence easier and more understandable for students to tinker with.

Whatever approach was used we would have to be careful to avoid providing a solution which might later lead us into providing support for individual images. Support should be limited to the production of replacement images.

Deliverables: a DICE-like virtual machine image which could be used for simpler student work, runnable on personal machines and on IS lab machines.

Why:

Customer: Informatics students, although other DICE users might use it too

Case statement: We expect soon to have to move all our student labs to a more remote location for a year. We also expect to be catering for significantly more students than at present - more than our student labs could handle if we aimed to provide each student with the opportunity of sufficient time sitting physically in front of a machine. It may well be the case that even when our student labs moved back from their remote location, we would not have the resources to provide enough machines for such an aim. We therefore want to try to reduce demand on the student lab machines by moving some simpler work to VMs which could run elsewhere.

When:

Status:

Timescales:

Priority:

Time:

How:

Wiki: VirtuallyDICEModes

Plan:
Try these two approaches and find out which seems easier and quicker:

Start with DIY DICE

Install a DIY DICE machine, including teaching software.

Local Account

Provide a generic local user account.

De-LCFG

Stop the machine from being managed by LCFG.

Networking

Remove any fixed IP address from the machine, make it networkable with minimal alteration or hassle on an average new VMware installation, bearing in mind that multiple copies of it will be running simultaneously.

AFS for backups

Enable the local user to sign in to DICE and access AFS directories from the VM, for backup purposes.
Start with SL

Start with a standard Scientific Linux installation.

Kerberos

Get Kerberos installed and working with DICE

AFS

Get OpenAFS installed and working with DICE

Teaching Software - Generic

Get some idea of which teaching software is available on SL anyway and how easy it would be to install and cofigure that appropriately.

Teaching Software - Local

Find out which teaching software is not from SL repositories but could be distributed locally, for instance from a student-accessible yum repository.

Teaching Software - Awkward

Find out which software would be likely to be awkward to distribute/install/configure as above and why - e.g. legal issues, technical problems, resource problems.

Other:

Dependencies:

Risks:

Milestones

Proposed date	Achieved date	Name	Description

Increasing Energy Savings from DICE Desktops

boss — Fri, 25 Jan 2013 15:46:22 +0000

Project ID:

234

Current stage:

5_Completed

Manager:

Unit:

mp-unit

What:

Description: This project aims to assess and if necessary improve the effectiveness of energy-saving measures on DICE desktops. It builds on project 94 which delivered the LCFG sleep component.

Deliverables: A data feed of statistics showing the sleep rates of DICE desktops; possible improvements in the LCFG sleep component; possible improvements in the energy-saving guidance given to users of DICE desktops

Why:

Customer: The School of Informatics and the University of Edinburgh should benefit from lower energy bills and a lower environmental impact.

Case statement: For some years the DICE desktops in the student labs have used energy-saving sleep software, the LCFG sleep component, which was developed in project 94. More recently the sleep component was also installed on staff and postgrad DICE desktops. We should now follow up on this work by finding out how successful it has been and what might be done to increase the energy savings. The project would start by collecting statistics from
machines to see how much of the time they are asleep - in other words
trying to understand how much of a contribution that sleep might be
making to energy saving. Depending on what the statistics say, some
changes to the LCFG sleep component could be made or users could be informed
about how to get their machines to sleep.

Gathering the statistics should be straightforward as machines running the sleep component currently log when they enter and exit from a sleep state. The statistics could be made available not just for this project but also for other purposes. For instance it may well be useful and interesting to use them to produce periodic graphs of desktop sleep rates, or to calculate monetary savings.

There is at least one obvious change which could be made to the sleep component: it currently lacks a way to find out the idleness of a machine's keyboard and mouse. This means that the sleep component has been conservatively configured to avoid suspending any machine which has an X session in progress. Adding such a capability, and safely sleeping idle machines running a user X session, could be a way to increase machine sleep substantially.

As for explanations for users, we already have some blog posts and a page of guidance for DICE desktop users on machine sleep but this could be expanded and made more detailed if necessary. It could perhaps also be publicised more widely from time to time.

When:

Status:

Timescales:

Priority:

Time:

How:

Proposal:

Resources: Estimate of 2 weeks effort, and temporary use of a few DICE desktop machines.

Plan:

Final report URL:

https://wiki.inf.ed.ac.uk/DICE/FinalProjectReport234

Other:

Dependencies:

Risks:

Milestones

Proposed date	Achieved date	Name	Description

Provide support for REF 2012

boss — Fri, 25 Jan 2013 15:46:21 +0000

Project ID:

232

Current stage:

3_Implementation

Manager:

Unit:

compexec

What:

Description: Provide computing support for REF 2012. Act as liaison between academic staff responsible for our REF submission (led by Stephen Gilmore) and computing staff.

Deliverables:

Why:

Customer:

Case statement:

When:

Status:

Timescales:

Priority:

Time:

How:

Proposal:

Resources: Provisionally allocated 2 weeks in each of T1 and T2.

Plan:

Other:

Dependencies:

Risks:

Milestones

Proposed date	Achieved date	Name	Description

Wake-on-LAN

boss — Fri, 25 Jan 2013 15:46:18 +0000

Project ID:

213

Current stage:

5_Completed

Manager:

Unit:

mp-unit

What:

Description: Implement a wake-on-LAN system

Deliverables: A wake-on-LAN system

Why:

Customer: Anyone with a machine which might be put to sleep

Case statement: We've had a few queries now about wake-on-LAN, the implication being that folk would be happier to have their machines sleep if they could have a way to wake them up as needed.

When:

Status: Delivered. The service can be found here.

Timescales: Putting together a cgi-bin script would probably only take a week or so. Testing and ironing out any platform- and machine-specific quirks might take another couple of weeks. There might be another week or so of Support effort to add the necessary entri

Priority: High priority.

Time:

How:

Proposal:

Resources: The system would probably run on the network infrastructure machines, or perhaps the network services machines if that's easier, so no additional kit would be needed.

Plan: The netinf machines are on all the right subnets and already have "all" machines' MAC addresses through the dhcpd resources. The idea would be to provide a cgi-bin script which would look up the MAC address and call some piece of code (externally-sourced or locally-written) to send the magic packet.

To be accessible from outside, there would need to be cosign-protected https access to the cgi-bin. If this conflicted with the need to have simple http access to the netmon pages for DR purposes then the system might be better run on the network services machines than the network infrastructure machines, but that's something to determine at the time.

Alternatively, hook into the central IS-provided system. There are issues with configuration-data transfer and directed broadcasts which make this appear rather more expensive and less flexible than doing it ourselves.

Other:

Dependencies: Given that not all machines sleep/wake properly, there might need to be some kind of (spanning-map based?) configuration by machine type. Since ?most self-managed machines don't currently have a machine-type entry in their profiles, for this to be done reliably could require some Support effort.

Risks:

Milestones

Proposed date	Achieved date	Name	Description

Hadoop cluster

boss — Fri, 25 Jan 2013 15:46:10 +0000

Project ID:

175

Current stage:

5_Completed

Manager:

Unit:

rat-unit

What:

Description: Make a Hadoop cluster for teaching, for the "Extreme Computing" course.

Deliverables: A Hadoop cluster of half a dozen or so data nodes, with the number of data nodes quickly expandable to 20 or so if necessary near practical handin time. It's not clear whether or not Hadoop 0.18.3 will be sufficient or if we'll have to use the more recent Hadoop 0.20.2.

Why:

Customer: Stratis Viglas who co-runs the Extreme Computing course.

Case statement:

When:

Status:

Timescales: An LCFG-managed Hadoop 0.18.3 cluster is up and running

Priority:

Time:

How:

Proposal:

Resources: We're borrowing the necessary number of machines from the hcrc beowulf cluster.

Plan:

Final report URL:

https://wiki.inf.ed.ac.uk/DICE/HadoopClusterFinalReport

Other:

Dependencies:

Risks:

Milestones

Proposed date	Achieved date	Name	Description

Investigate install scripts for self-managed machines

boss — Fri, 25 Jan 2013 15:46:07 +0000

Project ID:

161

Current stage:

5_Completed

Manager:

Unit:

mp-unit

What:

Description: Investigate potential techniques for providing install scripts, for configuring self-managed machines against DICE services.

Deliverables: A report describing what is and isn't feasible. Perhaps some prototypes

Why:

Customer: Users of self-managed machines

Case statement: Users have often asked for install scripts which configure their machines against DICE services - they often find following documentation awkward and error prone.

When:

Status:

Timescales:

Priority:

Time:

How:

Proposal:

Resources: One week of resource has been allocated for 2010 T2.

Plan: Poll Mac users to find out their attitude to possible automation. Investigate possible automation methods.

Other:

Dependencies: None

Risks:

URL:https://wiki.inf.ed.ac.uk/DICE/MpuInstallScripts

Milestones

Proposed date	Achieved date	Name	Description

Lightweight Platform

boss — Fri, 25 Jan 2013 15:46:02 +0000

Project ID:

143

Current stage:

0_Pending

Manager:

Unit:

mp-unit

What:

Description: A port of LCFG (to "inf" level) to every new version of Fedora that comes out.

Deliverables: LCFG ported to "inf" level for each new Fedora release, starting with Fedora 12.

Why:

Customer: The School of Informatics, although the other LCFG-using Schools would also have the opportunity to use it.

Case statement: Fedora changes quickly, in theory twice a year, and has a short lifespan. The short lifespan is a big reason for our abandoning it in favour of an RHEL-based distribution. However by the same token Fedora tends to have lots of recent software which RHEL lacks. One of the reasons given for postgrads abandoning DICE is the out of date software on SL5. This wouldn't be a problem on Fedora. We'd have the opportunity to provide a "bleeding edge" managed platform for those who wanted it. By tackling every Fedora release we'd also reduce the workload for each single LCFG port: the technological gap between one Fedora release and the next isn't likely to be as big as that between e.g. RHEL 5 and RHEL 6. Providing LCFG for Fedora would also make the ports to new versions of RHEL-based distros like Scientific Linux considerably easier. A suggestion is that the port starts when a Fedora beta release comes out. Providing LCFG for each Fedora release needn't be an intolerable support burden as Fedora support for each release ends relatively quickly, and our LCFG support for a release could end at the same time as the Fedora support ends.

When:

Status:

Timescales:

Priority:

Time:

How:

Proposal: We should port LCFG (to "inf" level) to every Fedora release.

Resources: Two PCs and a blog. Around 8 weeks

Plan:

Other:

Dependencies: Project 140 (Automated upstream repository mirroring) would make it far easier/quicker to do the ports: making the package lists by hand can take ages and be extremely awkward.

Project 138 (Rewrite the LCFG syslog component) is essential as Fedora has used rsyslog for a while now.

Project 68 (LCFG Linux Install redevelopment) is needed for Fedora & RHEL 6 as the current install system almost certainly won't work on them.

Project 49 (LCFG Boot component rewrite) may also be needed since Fedora uses upstart. We may be able to squeeze by for a short while without this as it appears that upstart can also handle old-stye rc scripts.

Risks:

Milestones

Proposed date	Achieved date	Name	Description

Server Hardware Interaction

boss — Fri, 25 Jan 2013 15:46:00 +0000

Project ID:

134

Current stage:

5_Completed

Manager:

Unit:

mp-unit

What:

Description: This project aims to improve the management of and interaction with the server hardware.

Deliverables: Easy, automatic updates for server firmware and RAID firmware; improved monitoring of RAID controllers; monitoring of ambient temperature and clean server shutdown when it gets too high; an OMSA configuration which is useful to us and which we can live with; an improved appreciation of what is and isn't useful and easily automatable in this area. The highest priority deliverable is automated monitoring of the RAID controllers.

Why:

Customer: Directly, the computing staff of the School of Informatics. Indirectly, users of School of Informatics services, through improved reliability and quicker & easier maintenance tasks.

Case statement: We need to improve our interaction with server hardware. For example we have in the past lost data through RAID controller firmware not being kept up to date. It's useful to keep server BIOS versions upgraded. It's useful to know when disks in a RAID array have developed a fault. And we should be able to automatically and cleanly shut down machines when the ambient temperature gets too high. Software exists to help manage these sort of tasks; we need to work out which software would help us and how and design and implement lightweight solutions accordingly. This would provide a way to improve our server reliability (for those servers which aren't being painstakingly monitored and upgraded by their managers) or automate some time-consuming tedious tasks (for those managers who do take pains to upgrade firmware etc.)

When:

Status:

Timescales: The project has been allocated two weeks of time.

Priority: CEG has judged this to be a high priority project w.r.t. temperature measurement and RAID monitoring.

Time: The project has been allocated two weeks.

How:

Proposal: see Plan.

Resources: Plentiful access to examples of our more popular models of rack-mounted servers.

Plan: The highest priority deliverable for this project is now clean automatic shutdown on excessive ambient temperature. The next highest is RAID monitoring. The plan for that is to find out about our RAID facilities; try out possible monitoring software; get this working; pick the most suitable; write or obtain a Nagios translator for it.
If time is left over after achieving this then the project will go on to work on other possible deliverables such as more automatic and easier firmware updates for server hardware.

Other:

Dependencies: The project will depend on getting access to servers...

Risks:

Milestones

Proposed date	Achieved date	Name	Description
2010-02-04	2010-02-12	temp1	Design an emergency temperature protection system
2010-02-05	2010-02-12	temp2	Implement an emergency temperature protection system
2010-02-05	2010-02-12	temp3	Test the emergency temperature protection system
2010-02-08	2010-02-12	temp4	Document the emergency temperature protection system
2010-05-21	2010-05-21	hwmon1	Learn about Nagios and RAID status in preparation for developing a simple hardware status monitor for DICE servers.
2010-05-31	2010-05-28	hwmon2	Implement and document a simple hardware status check for DICE servers for use with the Nagios monitoring system. It should check at least RAID disk status.
	2010-07-02	hwmon3	Test and promote the simple hardware status check system for DICE servers.