DICE development projects - 2_Evaluation

LCFG systemd component

ascobie — Wed, 27 Nov 2013 17:45:07 +0000

Project ID:

297

Current stage:

2_Evaluation

Assigned Priority:

Manager:

ascobie

Unit:

mp-unit

Summary:

Produce an LCFG systemd component

What:

Produce an LCFG component that configures systemd, the system initialisation system used in RHEL 7.

Why:

Traditionally, Redhat based distributions have used the SystemV initscripts system for stopping and starting system services such as network, dns, http etc. Whilst an LCFG managed system doesn't use the SystemV initscripts system directly to bring up a system, it does make use of the individual scripts that each system service provides.

RHEL 7 uses the systemd system for stopping and starting system services. A new component is required to configure this system.

When:

This is required for the LCFG RHEL7 port

How:

Blog on the RHEL7 port blog

Read up about systemd
Consider how systemd might interact with LCFG components
Presentation to COs
Decide how components will be started/stopped with systemd
Design schema for component
Circulate schema for discussion
Implement component
Configure systemd to start a system (with no LCFG components)
Configure systemd to start system with lcfg-client component

Effort estimate:

Produce a talks.cam not-a-service for trialling

ascobie — Fri, 30 Aug 2013 16:06:42 +0000

Project ID:

292

Current stage:

2_Evaluation

Manager:

toby

Unit:

inf-unit

Summary:

Produce a talks.cam trial service for disseminating information about School talks

What:

Install the talks.cam software (http://talks.cam.ac.uk/document/documentation) on a VM and configure to authenticate against DICE.

Why:

Iain Murray suggested this a few project periods ago.

"I won't rehash all of the advantages (please look at the docs). But advantages for us would include: 1) the webforms are nicer than plone and announcements and requesting abstracts from speakers are dealt with by the system. 2) people can subscribe to any mix of talk streams in their calendars, email, and RSS readers. 3) Any combination of lists can be listed on the system, or embedded in any webpage. The School could list all talks going on in informatics on the front display screen, the main website, etc.

The system was developed by the academics that first used it, and after going through a couple of versions it's very usable and has proved popular. Maybe ultimately this is a system that should be talks.ed for the whole University. But trialling it with a couple of institutes and then a whole School first seems sensible, and is how talks.cam developed in Cambridge."

We were going to put some effort into quickly putting it up, but it hadn't been ported to SL6. It looks like this has now been done, so perhaps worth spending a small amount of effort to get it going.

Effort estimate:

1 week

Other:

Dependencies:

Risks:

Evaluate move of LCFG configuration from svn to git

ascobie — Fri, 23 Aug 2013 14:19:30 +0000

Project ID:

289

Current stage:

2_Evaluation

Manager:

Unit:

mp-unit

Summary:

We want to know if it is possible and desirable to move the LCFG configuration data (headers) from subversion to git

What:

Evaluate the possibility of switching from subversion to git for the management of the LCFG configuration data. Test a prototype based on a snapshot of the current data. Ensure that we can continue to have an efficient, low-overhead weekly testing/stable release cycle.

Why:

Current Situation

The LCFG configuration files (headers and package lists) are currently stored in a subversion revision-control repository which is run by the School of Informatics. This repository can be split down
into 3 separate strands:

lcfg: Public data which will create sensible default configuration
policies for the general case
ed: Public data which configures computers to suit the University of
Edinburgh computing environment
dice: Private data which configures computers to suit the School of
Informatics computing environment

The separation of public and private data, which is all stored in the same single repository, is achieved using subversion ACLs. These ACLs ensure that only computing staff from the School of Informatics can read or modify files related to the dice environment. Further to these access restrictions, although the public data is readable by any user it is only modifiable by Informatics computing staff.

Each week a snapshot of the development "trunk" is taken and tagged as a new release. That release is first deployed to a number of testing machines. Once a release has been tested and any problems resolved it is tagged again as the stable release. The full stable release is rolled out to DICE machines, the public portion (the lcfg and ed sections) is bundled up and pushed to the LCFG public repository (http and rsync). Downstream users, such as the MDP team in IS, then test and apply the changes to their own systems.

Currently the only way in which external users can contribute new features or improve existing configurations is to create a patch file containing the required changes and then submit a bug report in the LCFG bug tracker. This is a rather unwieldy process which results in external contributors being treated as second-class citizens. As well as creating a situation where contributing to the LCFG project is more difficult than it should be it also creates more overhead for Informatics staff. Each patch has to be reviewed and then manually applied to the repository, there is no automation to simplify the process or assist in reducing the risk of patches being badly applied. Although not impossible to do with the subversion revision control system this is not a trivial task.

Broadening the Community

There has never been an explicit policy of excluding external users. Rather the current situation has occurred naturally because the project was created and managed by Informatics computing staff. There simply was not originally any specific requirement for extending write access beyond that team. In recent years the LCFG project has become much more widely adopted across the University of Edinburgh. Each succesful deployment has increased the number of people who know how to use and create LCFG components. If the project is to continue to grow and thrive then contributions must be welcomed and actively encouraged. To encourage more external contributions we should aim to make it possible for anyone to make contributions. Clearly we cannot provide direct write access into the main repository, a good strategy, which is used by many Open Source projects, would be to allow public contributions into a review system. Gatekeepers can then review submissions, respond to the contributors and commit or reject the changes in an efficient, transparent process.

We have previously considered restructuring the LCFG headers repository so that we can allow external contributors direct write access to the headers for components which they manage. The restructuring is probably a good idea to help improve comprehensibility and discoverability, however it was decided that allowing access like this is too much of a security risk for Informatics. In Informatics we have approximately 100 machines which follow the develop release in real time, any mistake to a widely used header could result in us having badly broken machines. This risk is considered acceptable when there is a local CO involved who can check the likely impact of any change and be relied upon to notify others in advance of any risky changes. They are also more easily contacted if a problem occurs. This problem pretty much rules out subversion as a choice for our revision control system if we wish to involve external contributors. A better alternative would be to use a revision control system with a distributed model (e.g. git). This would allow contributors to fork repositories, test their own changes locally and push back changes when ready. We can also then build a review system around that approach which should help raise the quality of all changes.

Good Practices

Beyond just lowering the barriers for external contributors a review-based system has a number of additional benefits. By using a review system for all changes to the configuration files we get "more eyes on the code". It would actively increase the number of people who have an understanding of the purpose of an LCFG component. There would also be a wider awareness of upcoming changes. It can also be a useful tool for helping to tutors new contributors on topics of good-practice such as expected style, syntax, config layout, etc, when it is not possible to work side-by-side in "real life". A review system also encourages users to use good, descriptive changelogs, there's no way to lazily submit changes with an overly terse, unhelpful or just plain wrong changelog entry.

How:

Import a snapshot of the current LCFG configuration data (the headers) into a git repository.
Configure the gerrit review system for the git repository
Get local COs to play with the system and give feedback
Get external contributors to play with the system and give feedback
Investigate how the dice headers could be split from the public lcfg and ed levels.
Investigate how we would do the weekly testing/stable releases

Effort estimate:

Two weeks

Other:

Dependencies:

Risks:

Project Submission

boss — Fri, 25 Jan 2013 15:46:29 +0000

Project ID:

270

Current stage:

2_Evaluation

Manager:

timc

Unit:

rat-unit

Summary:

Sort out UG4/PGT project submission/access

What:

Description: Project to tie together project and dissertation submission for UG4 and possibly MSc and other project / materials storage.

This is required because the existing retrieval mechanism (based on 2g Theon data) needs to be scrapped but there is no clear alternative in the new world. And in any event it would make considerable sense to tie together the many different scripts and data sources which make it so difficult to find project materials at present.

Some other points / interactions:

Retention and archiving policies

these need to be clarified, though retention can be bound into lifecycle if Theon / Prometheus is involved
mechanism for editing of erroneous submissions
potential continuing requirement for (live, public web) publication
Preferred plan is to interface with PURE subject to various conditions / requirements
Access to project (and materials) by external examiners
Data interface with Theon and/or projectsdb

Deliverables:

Why:

The existing hodge podge of scripts is 15+ years old and dependent on the legacy school database infrastructure (and data as it turns out). Certain aspects are broken. Access to projects and materials (outside school) must be done on an adhoc basis by admin staff.

When:

By April for UG4, August for PGT.

How:

Initially just a requirements capture to define the academic/student side.

Resources: Previous sources of information:
https://www.inf.ed.ac.uk/cgi-bin/submission_form_ug4.pl
https://www.inf.ed.ac.uk/cgi-bin/submission_links_ug4.cgi
http://www.inf.ed.ac.uk/teaching/courses/diss/guide.html#Submission
https://wiki.inf.ed.ac.uk/DICE/MScSubmission
http://www.inf.ed.ac.uk/publications/thesis/msc.html

Plan:

Effort estimate:

1 week (for requirements capture, scoping and proposal)

Investigate AFS on ECDF storage

boss — Fri, 25 Jan 2013 15:46:29 +0000

Project ID:

268

Current stage:

2_Evaluation

Manager:

cms

Unit:

services-unit

Summary:

AFS access to the ECDF provided 500GB per research user.

What:

Description: The University is proposing to provide each research active user with 500GB disk space. Unfortunately, IS are currently proposing that this space will be presented by NFS, CIFS or SSHFS. This project would investigate whether we (working with IS) can add AFS as an access mechanism.

Deliverables: Report of the investigations, and a working AFS path to the space if possible.

Why:

Customer:
All research active users within School

Case statement:
It wouldn't make sense to not use the space being made available to us, however for better integeration with our file system, AFS access would seem a better option than the proposed filesystems.

When:

Status:
Proposal

Timescales:
It's been resourced this third (T1, 2013) for 3 weeks.

Priority:
Medium/low - It's not mission critical, and at the last CCPAG (Feb 2013) ECDF were still waiting for the go ahead to provide the space.

Time:
1 hour so far

How:

Resources: Total of 3 weeks.

Proposal:
Two weeks to create an .ed.ac.uk AFS cell on GPFS nodes. One week to benchmark end-user performance and effect on GPFS cluster.

Plan:

Get to know how ECDF GPFS works.
Research strategies for running AFS on GPFS.
Assuming it is possible, mount GPFS on machine in Forum, or get access to machine at ECDF that does have access and setup AFS. Probably a test cell first.
If it isn't possible, can we get access to raw disk space and mount as a normal /vicep
Test and bench mark, check performance and affect on cluster.
If all is good, then make what ever changes to run it as a full service - eg hardware, final cell name.

What if a researcher doesn't want "their" space as AFS. Some may want to opt out and use one of the IS proposed protocols. eg researchers with Windows machines?

Final report URL:

https://wiki.inf.ed.ac.uk/DICE/Project268AFSonECDF

Other:

Dependencies:
Access to ECDF storage.

Risks:
This would be a new service, not impacting on current services, so risks would be low. Perhaps performance issues affecting network/ECDF.

Milestones

Proposed date	Name	Description
22/3/2013	access	Get access to ECDF and GPFS
	testcell	Setup test AFS cell
	perf	Performance testing
	results	Present results/findings/thoughts
	production	Turn into production service if all went well
	writeup	Final Report

Do we still do these?

Meeting the School's Archiving Needs

boss — Fri, 25 Jan 2013 15:46:14 +0000

Project ID:

194

Current stage:

2_Evaluation

Assigned Priority:

Manager:

cms

Unit:

services-unit

What:

Description: This project will investigate how best to meet the School's requirements for archiving data

Deliverables: A cohesive archival policy for the School's data along with guidence on how this policy may achieved

Why:

Customer: School

Case statement: The report produced by https://devproj.inf.ed.ac.uk/project/show/149 concluded that the School's current approach to archiving was no longer appropriate and that investigation should be carried out on how best the School's archiving needs could be met. This project will carry out that investigation. This project will conduct that investigation and in addition formualte and publicise a policy on when it is permissible for personal information such as home directories to be retrieved from archive.

When:

Status:

Timescales: Identifying the requirements and constraints which an archiving solution for the school needs to meet may take some time given the many different contributers to those requirements. Allow a week for this. Identifying archive methods and evaluating how they meet the needs of the School may also take up to two weeks. Categorising and assigning ownership of data and engaging those owners in initiating the archiving process will take a considerable amount on time (just how much depends on how much data we are dealing with and how many separated owners) but much of this work can be done by CSOs.

Priority: High. Archiving of data in an appropriate way is vital for the School and is something that is just not happening at the moment

Time: 3 weeks

How:

Proposal:

Resources:

Plan:

Formulate specification of School's archiving needs
Quantify data needing archived
Survey available archiving methods/services and assess best fit for 1) and 2)
Produce and implement plan/policy for migration or initiation of data archiving

Effort estimate:

3 weeks + CSO time

Other:

Dependencies: Familiarity with the School's data

Risks: We greatly increase the chances of losing vital data if we do not address this issue urgently.

Milestones

Proposed date	Name	Description
19/8/2013	Identify needs	Identify the archiving needs of the School
26/8/2013	quantify data	Estimate the amount of data (both existing and potential) needing archived
9/9/2013	Identify methods/services	Identify and characterise the archiving services we can potentially make use of
23/9/2013	Plan implimentation	Produce plan for implementation of new archiving strategy and implement

EUCLID Informatics Enhancements

boss — Fri, 25 Jan 2013 15:45:52 +0000

Project ID:

110

Current stage:

2_Evaluation

Manager:

timc

Unit:

rat-unit

What:

Description: EUCLID won't do everything we need. Some critical local processes will not be support. This project encapsulates all of them although some could potentially be forked off into separate projects in the future.

Deliverables: Various local enhancements to EUCLID. On the critical list however are system account creation, system authentication control, online practical submission and student mailing lists.

Why:

Customer: Systems staff. ITO staff. IGS staff.

Case statement: These are systems which are pre-existing - not new functionality, essential to business function, do not duplicate existing functionality within EUCLID and will not be provided by EUCLID now or in the near future.

When:

Status:

Timescales: Critical systems must be working by start of the 09/10 academic session. Other systems have different timescales.

Priority: Very high.

Time: 1 month for critical systems.

How:

Proposal: See Plan.

Resources: Effort.

Plan: Evaluate data requirements for each critical system and feed this back to IS. Test IS interface. Adapt local systems to new interface.

Other:

Dependencies: Largely dependent on the EUCLID Interoperability project which is largely dependent on IS producing a suitable interface to EUCLID.

Risks: IS don't deliver. Change to EUCLID data structures enforces incompatible changes onto our existing systems resulting in significant rewriting effort.

URL:https://wiki.inf.ed.ac.uk/DICE/Euclidie

Milestones

Proposed date	Achieved date	Name	Description
2009-05-15	2009-04-30	CollegeMeet	Arrange college level technical meeting to discuss common requirements
	2009-03-31	CoreData	Identify generic core data requirements for each critical system.
	2009-08-31	NonCritCD	Identify generic core data requirements for each non-critical system.
	2009-03-31	CoreAffect	Identify all known possible functional impact/changes to each critical system.
	2009-08-31	NonCritCA	Identify all known possible functional impact/changes to each non-critical system.
	2009-12-31	Systems	Continue identifying local systems that should be part of this project.
	2010-03-31	EUCLID	IS should have developed EUCLID interface and local support should be in place. This ia a guess, based on IS start date on interface work in Sep2009 - there is no other supporting evidence for this timeframe as yet.
	2010-06-30	NewSystems	Update each critical system to work with locally cached EUCLID data obtained through the IS interface.

Intrusion Detection System

boss — Fri, 25 Jan 2013 15:45:50 +0000

Project ID:

102

Current stage:

2_Evaluation

Manager:

gdmr

Unit:

inf-unit

What:

Description: We don't have any automated mechanism at the moment for spotting compromised machines. We could do with one somewhere (or perhaps one or more at each site) watching the network for the usual traffic signatures.

Deliverables: A working prototype IDS, or alternatively a report as to why the project couldn't produce one.

Why:

Customer: All.

Case statement: As things stand, we're relying on our edge filter rules to keep out things we don't like, but have no way to check that they're actually doing what we want. Furthermore, we have no automated way to make sure that our internal machines are behaving themselves on the network. Running a parallel IDS would be expected to provide independent assurance that traffic in and out is reasonably clean. See also DevProj#98 and DevProj#103.

When:

Status:

Timescales: Suggest a time budget, taking two or three weeks to prototype something, and then see what's left.

Priority:

Time:

How:

Proposal:

Resources: Depends on where it's run. Most likely is on one or more of the edge routers. Part of the evaluation phase is to see if that's a viable option. If it turns out to slug the machine too much (and we think the service is actually useful) then we might have to invest in some dedicated hardware.

Plan:

Look at available IDSes."snort" seems a likely candidate, at least from an initial glance, but there may be others out there.
Prototype something.
Pick the best (assuming there's more than one, of course).
Review time budget. Either write a report and wind up, or polish up prototype for a full service.

Other:

Dependencies:

Risks:

Milestones

Proposed date	Achieved date	Name	Description