DICE development projects - mp-unit

LCFG systemd component

ascobie — Wed, 27 Nov 2013 17:45:07 +0000

Project ID:

297

Current stage:

2_Evaluation

Assigned Priority:

Manager:

ascobie

Unit:

mp-unit

Summary:

Produce an LCFG systemd component

What:

Produce an LCFG component that configures systemd, the system initialisation system used in RHEL 7.

Why:

Traditionally, Redhat based distributions have used the SystemV initscripts system for stopping and starting system services such as network, dns, http etc. Whilst an LCFG managed system doesn't use the SystemV initscripts system directly to bring up a system, it does make use of the individual scripts that each system service provides.

RHEL 7 uses the systemd system for stopping and starting system services. A new component is required to configure this system.

When:

This is required for the LCFG RHEL7 port

How:

Blog on the RHEL7 port blog

Read up about systemd
Consider how systemd might interact with LCFG components
Presentation to COs
Decide how components will be started/stopped with systemd
Design schema for component
Circulate schema for discussion
Implement component
Configure systemd to start a system (with no LCFG components)
Configure systemd to start system with lcfg-client component

Effort estimate:

LCFG SL7 port (inf level)

ascobie — Wed, 27 Nov 2013 10:15:43 +0000

Project ID:

296

Current stage:

3_Implementation

Assigned Priority:

Manager:

ascobie

Unit:

mp-unit

Summary:

Develop LCFG inf level support for Scientific Linux 7 (SL7)

What:

Develop Inf level support for Scientific Linux 7 (SL7) targetting the desktop and server environments the x86_64 cpu architecture. This work will form the basis for the next DICE desktop and server platforms.

This will utilise DICE authentication/authorization and name services in client mode (no local services). AFS home directories will be supported. This will also create an SL7 platform at the LCFG level which can be utilised by groups which are external to Informatics.

Why:

The current DICE platform, Scientific Linux 6, was released in 2011 and is showing its age in terms of software component versions. It is important, from both teaching and research perspectives, that we keep the platform up-to-date so that recent versions of software can be used.

When:

We are dependent on RHEL released RHEL 7. As soon as RHEL 7 beta is released (expected early 2014 Q1) we should start work. Experience has shown that we need at least 4-5 months to port the DICE desktop to a new platform. If RHEL 7 beta does not ship by early March, it is unlikely that we will be able to complete the DICE desktop in time for deployment before start of academic year 2014/2015.

How:

See SL7 port home

Effort estimate:

10w

Virtual desktop infrastructure

squinney — Fri, 11 Oct 2013 09:05:32 +0000

Project ID:

295

Current stage:

0_Pending

Manager:

squinney

Unit:

mp-unit

Summary:

Virtual desktop infrastructure

What:

Provide a decent virtual desktop infrastructure. For researchers (e.g. staff, postdocs, postgrads) who have DICE desktops, the service might utilise the computing resources of the desktop machine allocated to the particular user. This is not possible with NX but there are probably other systems out there which can be used, this project will evaluate the various options to come up with a proposal for future implementation.

Why:

The recently deployed NX service has been wildly popular and users would like an even better service. It's clear that running just one server for this purpose is not going to be sufficient. We have to limit resource access (e.g. memory) so that it is reasonably usable by all. Most staff have DICE desktop machines so it would be nice to have some system to allow remote graphical access.

Effort estimate:

2 weeks

Other:

Dependencies:

Risks:

Evaluate move of LCFG configuration from svn to git

ascobie — Fri, 23 Aug 2013 14:19:30 +0000

Project ID:

289

Current stage:

2_Evaluation

Manager:

Unit:

mp-unit

Summary:

We want to know if it is possible and desirable to move the LCFG configuration data (headers) from subversion to git

What:

Evaluate the possibility of switching from subversion to git for the management of the LCFG configuration data. Test a prototype based on a snapshot of the current data. Ensure that we can continue to have an efficient, low-overhead weekly testing/stable release cycle.

Why:

Current Situation

The LCFG configuration files (headers and package lists) are currently stored in a subversion revision-control repository which is run by the School of Informatics. This repository can be split down
into 3 separate strands:

lcfg: Public data which will create sensible default configuration
policies for the general case
ed: Public data which configures computers to suit the University of
Edinburgh computing environment
dice: Private data which configures computers to suit the School of
Informatics computing environment

The separation of public and private data, which is all stored in the same single repository, is achieved using subversion ACLs. These ACLs ensure that only computing staff from the School of Informatics can read or modify files related to the dice environment. Further to these access restrictions, although the public data is readable by any user it is only modifiable by Informatics computing staff.

Each week a snapshot of the development "trunk" is taken and tagged as a new release. That release is first deployed to a number of testing machines. Once a release has been tested and any problems resolved it is tagged again as the stable release. The full stable release is rolled out to DICE machines, the public portion (the lcfg and ed sections) is bundled up and pushed to the LCFG public repository (http and rsync). Downstream users, such as the MDP team in IS, then test and apply the changes to their own systems.

Currently the only way in which external users can contribute new features or improve existing configurations is to create a patch file containing the required changes and then submit a bug report in the LCFG bug tracker. This is a rather unwieldy process which results in external contributors being treated as second-class citizens. As well as creating a situation where contributing to the LCFG project is more difficult than it should be it also creates more overhead for Informatics staff. Each patch has to be reviewed and then manually applied to the repository, there is no automation to simplify the process or assist in reducing the risk of patches being badly applied. Although not impossible to do with the subversion revision control system this is not a trivial task.

Broadening the Community

There has never been an explicit policy of excluding external users. Rather the current situation has occurred naturally because the project was created and managed by Informatics computing staff. There simply was not originally any specific requirement for extending write access beyond that team. In recent years the LCFG project has become much more widely adopted across the University of Edinburgh. Each succesful deployment has increased the number of people who know how to use and create LCFG components. If the project is to continue to grow and thrive then contributions must be welcomed and actively encouraged. To encourage more external contributions we should aim to make it possible for anyone to make contributions. Clearly we cannot provide direct write access into the main repository, a good strategy, which is used by many Open Source projects, would be to allow public contributions into a review system. Gatekeepers can then review submissions, respond to the contributors and commit or reject the changes in an efficient, transparent process.

We have previously considered restructuring the LCFG headers repository so that we can allow external contributors direct write access to the headers for components which they manage. The restructuring is probably a good idea to help improve comprehensibility and discoverability, however it was decided that allowing access like this is too much of a security risk for Informatics. In Informatics we have approximately 100 machines which follow the develop release in real time, any mistake to a widely used header could result in us having badly broken machines. This risk is considered acceptable when there is a local CO involved who can check the likely impact of any change and be relied upon to notify others in advance of any risky changes. They are also more easily contacted if a problem occurs. This problem pretty much rules out subversion as a choice for our revision control system if we wish to involve external contributors. A better alternative would be to use a revision control system with a distributed model (e.g. git). This would allow contributors to fork repositories, test their own changes locally and push back changes when ready. We can also then build a review system around that approach which should help raise the quality of all changes.

Good Practices

Beyond just lowering the barriers for external contributors a review-based system has a number of additional benefits. By using a review system for all changes to the configuration files we get "more eyes on the code". It would actively increase the number of people who have an understanding of the purpose of an LCFG component. There would also be a wider awareness of upcoming changes. It can also be a useful tool for helping to tutors new contributors on topics of good-practice such as expected style, syntax, config layout, etc, when it is not possible to work side-by-side in "real life". A review system also encourages users to use good, descriptive changelogs, there's no way to lazily submit changes with an overly terse, unhelpful or just plain wrong changelog entry.

How:

Import a snapshot of the current LCFG configuration data (the headers) into a git repository.
Configure the gerrit review system for the git repository
Get local COs to play with the system and give feedback
Get external contributors to play with the system and give feedback
Investigate how the dice headers could be split from the public lcfg and ed levels.
Investigate how we would do the weekly testing/stable releases

Effort estimate:

Two weeks

Other:

Dependencies:

Risks:

MPU - internal documentation review

alisond — Thu, 22 Aug 2013 09:47:03 +0000

Project ID:

281

Current stage:

0_Pending

Unit:

mp-unit

Summary:

Review of MPU internal documentation

What:

Review internal documentation relating to MPU and move existing documentation to new site where appropriate.

Why:

See project 280.

Other:

Dependencies: project 280

Risks:

Use LCFG to pre-configure Apple Macs

ascobie — Mon, 25 Mar 2013 14:30:12 +0000

Project ID:

277

Current stage:

0_Pending

Unit:

mp-unit

Summary:

Use LCFG to pre-configure Apple Macs

What:

At least one other CSE School uses LCFG purely to pre-configure new Apple Macs prior to delivery to their new owners. This project would deliver the same facility to our users.

The project is a follow on from the Investigate install scripts for self-managed machines project.

Effort estimate:

3 weeks

Other:

Dependencies:

Risks:

Virtualised DICE image - follow on

ascobie — Mon, 25 Mar 2013 14:24:23 +0000

Project ID:

276

Current stage:

0_Pending

Unit:

mp-unit

Summary:

Virtualised DICE image - follow on project

What:

Container project for follow on work from Virtualised DICE image trial project.

Effort estimate:

Unknown - depends on result of current project

Other:

Dependencies:

Risks:

Improve KVM guest migration using shared storage

ascobie — Mon, 25 Mar 2013 14:21:23 +0000

Project ID:

275

Current stage:

0_Pending

Unit:

mp-unit

Summary:

Improve KVM guest migration using shared storage (eg ceph)

What:

The disk images of KVM guests are currently stored local to their current KVM host. This means that guest migration involves copying a guest's disk image between KVM hosts - this is slow and not 100% reliable.

This project will investigate storing the KVM hosts on a high performance shared filesystem, eg Ceph.

Other:

Dependencies:

Risks:

LCFG client - remove obsolete module dependencies

ascobie — Mon, 25 Mar 2013 14:10:58 +0000

Project ID:

274

Current stage:

3_Implementation

Unit:

mp-unit

Summary:

LCFG client - remove dependency on obsolete modules (eg W3C::SAX)

What:

The current LCFG client depends on a number of obsolete Perl modules (eg W3C::SAX). This project will replace these modules with more modern, supported equivalents (libxml2 based XML::LibXML).

The plan is to take the opportunity to create a new object-oriented API for accessing the profile/component/resource and packages information for an LCFG client profile. Initially the API will only be used for the rdxprof client process reading the XML profile but it will be designed in such a way that we can support multiple source file formats (e.g. the DBM file, a switch to SQLite or a new improved XML format). The aim is also that it should be possible to use this framework to store the information into multiple target file formats. In the longer term all the other LCFG client libraries could be converted to using this framework to avoid the current code duplication which prevents us from fully modernising the client-side code (particularly the components).

Why:

The W3C::SAX obsolete modules aren't in use anywhere else and we have to port them for each platform upgrade. There's a real risk that they won't port to the next platform upgrade, which may cause problematic delays.

How:

Work out the best way to parse the LCFG XML structure using the XML::LibXML Perl module (3 days)
Design an object-oriented API for the component/lcfg/packages data gathered when parsing the XML (4 days)
Create a test suite for the new API (2 days)
Document the new API (1 day)
Rework the LCFG client modules to use the new API when parsing the XML and generating the DBM and rpmcfg files (5 days)

Effort estimate:

2-3 weeks

Other:

Dependencies:

Risks:

A new project to check the migration

ascobie — Fri, 25 Jan 2013 16:46:16 +0000

Project ID:

271

Current stage:

4d_Dropped

Assigned Priority:

Manager:

ascobie

Unit:

mp-unit

Summary:

A proposal to create a new project

What:

what stuff

Why:

why stuff

When:

when stuff

How:

how stuff

Effort estimate:

5 weeks

Other:

Dependencies: just my time

Risks: Risk free