DICE development projects - rat-unit

Manager:

Unit:

Summary:

Migrate the systems used by InfHR in the old School database into Theon

What:

This covers a number of areas. The first is driven by a business analysis being done by admin. This will determine the shape of the structure under Theon. It is unlikely that there will be much migration (data or structure) from legacy in the end. It is more likely that the processes will be changed to be driven by upstream data from HR systems and the VRS, reducing the duplicated data entry currently done. Associated with this migration will be review/conversion of old conduits generating staff and institute landing pages to move into the Portal.

Why:

The InfHR systems will in due course be the only thing remaining in the old School DB setup (database, outgoing conduits framework). After this migration the old systems can be shut down removing a maintenance burden. At the moment data in 3g comes from the HR upstream source, but web pages are generated from the old 2g InfHR maintained data. There is an increasing interference/drift of the main Theon data processing from the data that is still maintained locally by InfHR resulting in discrepancies. This would be corrected by migration. The migration is an opportunity for admin to improve/automate more of the HR processes (to take advantage of features supported by Theon but not in the legacy systems).

When:

Largely this is waiting on business analysis by admin, probably not until mid 2014.

How:

TBD based on business analysis.

Effort estimate:

Not possible at this stage but likely to be more than 4 weeks

Other:

Dependencies:

Risks:

IGS Misc Development

timc — Tue, 03 Sep 2013 08:15:39 +0000

Project ID:

293

Current stage:

3_Implementation

Manager:

Unit:

https://svn.theon.inf.ed.ac.uk/trac/milestone/Consolidation%20of%20Research%2...

Summary:

Process the operational tasks requiring small development effort arising out of the live Theon deployment. Tasks are individually identified in Trac.

Why:

Live use of Theon continuously identifies issues and enhancements for the IGS that are generally queued due to other work. Completing these assist the IGS in using the administrative processes now driven from Theon. Each issue is however always assessed for cost/benefit on a case by case basis.

When:

Due to the Trac being a live system with new tickets going in all the time there can be no explicit effort calculated, instead an effort budget should be allocated and tickets addressed one by one until the budget is used up.

How:

In constant contact ISS to identify prioritisation of tasks. Work on each task in priority order until the allocated effort budget is expired.

Final report URL:

Other:

Dependencies:

Risks:

Hadoop cluster revamp

iainr — Mon, 26 Aug 2013 15:09:49 +0000

Project ID:

291

Current stage:

Manager:

Unit:

Summary:

Rework the existing hadoop component and headers to provide better management and monitoring

What:

Rework the hadoop component to
* Allow individual nodes to be added and removed from jobtracker
* Allow additional diskspace on nodes to be added and removed from the hdfs filesystem
* Better support the use of multiple clusters.

Develop a small test suite that could be used to automatically test the operation of the cluster and possibly integrate this with nagios.

Deploy performance/usage monitoring software to better quantify the cluster usage.

Why:

The current component works well at configuring a fairly static cluster but adding and removing nodes, both to run jobs and as disk nodes in hdfs is awkward and time consuming. The configuration often does not reflect the actual nodes in the cluster which has led to data loss in the past.

Being able to remove nodes when the cluster was not being used would allow us to sleep nodes (where supported) and reduce the cluster power footprint. Ideally this could be scripted and allow the cluster to automatically minimise it's power usage without affecting it's usability.

The current hadoop configuration (version and installed packages) is largely dictated by the extreme computing course. Being able to set up multiple hadoop or hdfs clusters on the existing hardware with a minimum of overhead would allow us to respond more flexibly to requests from other groups of users where better security or a more recent version of hadoop is required.

Having a standard set of test jobs would allow us to monitor the clusters health and hopefully reduce downtime.

How:

Core:
Rewrite the hadoop component to fully manage the configuration of all the clients and the starting and stopping of the various daemons.
Produce a mechanism that would as much as possible automate managing the membership of the cluster (say via a spanning map) and also manage which member nodes are currently active within the cluster.
Take the basic hadoop example tutorial and automate execution of it so that it could be run as a nightly cron job. Investigate any hadoop nagios clients and integrate them into our nagios setup.
Generate basic monthly logs of cluster usage.

Optionally:
Write a daemon that monitors the cluster usage and dynamically activates and sleeps nodes to match the cluster usage.
Investigate any hadoop nagios clients and integrate them into our nagios setup.
Check for any useage/accounting type packages available for hadoop or write one if there is none available.

Effort estimate:

2-3 weeks for reworking the component, 2-3 days for the test setup 2+ weeks for the optional stuff.

Other:

Dependencies: Any development would have to fit in alongside current cluster usage

Risks:

RAT Unit internal documentation review

alisond — Thu, 22 Aug 2013 09:51:34 +0000

Project ID:

284

Current stage:

Unit:

Summary:

Review of RAT Unit internal documentation

Other:

Dependencies: project 280

Risks:

Display SAT info on AT foyer display screens

ascobie — Thu, 28 Mar 2013 10:46:04 +0000

Project ID:

278

Current stage:

4_Signoff

Manager:

Unit:

https://wiki.inf.ed.ac.uk/DICE/FinalProjectReport278

Summary:

Display SAT personal timetables on AT foyer display screens

What:

We are obliged to present an individual timetable for students from AY 2013/2014 onwards. Our plan is to present info from our RBS onto displays in the foyers in AT. This project will implement this info display

When:

Needs to be implemented for start of AY 2013/2014

How:

modify RBS:
Add course feed from theon
Add unique identifier to appropriate repeated bookings
Add scripts to import/export required data
add to theon
Incoming conduit to take feed from RBS
Marry groups assigned in theon with bookings in RBS
generate html to be published on AT Display screens

Final report URL:

Other:

Dependencies: I will have to lean heavily on Tim and Grahams Theonn knowledge and Neils knowledge of the RBS.

Risks: We may not get it done in time

Milestones

Proposed date	Achieved date	Name	Description
2009-06-18	2009-06-20	rbscourse	Add course feed to RBS development
2013-07-01	2013-06-14	rbskey	Add unique identifer to RBS entries
2013-07-14	2013-07-28	theonfeed	Add feed to import data from rbs into Theon
2013-08-07		theonsync	Sync data to active Theon tables
2013-08-14	2013-08-14	theoncode	Marry rbs bookings with Theon generated groups
2013-08-20		theonreport	Write report to generate html for AT screens and website
2013-08-24		test	Test deployment
2013-09-07		deploy	Final depolyment
2009-06-18	2013-06-20	rbscourse	Add course feed to RBS development

Project Submission

boss — Fri, 25 Jan 2013 15:46:29 +0000

Project ID:

270

Current stage:

2_Evaluation

Manager:

Unit:

Summary:

Sort out UG4/PGT project submission/access

What:

Description: Project to tie together project and dissertation submission for UG4 and possibly MSc and other project / materials storage.

This is required because the existing retrieval mechanism (based on 2g Theon data) needs to be scrapped but there is no clear alternative in the new world. And in any event it would make considerable sense to tie together the many different scripts and data sources which make it so difficult to find project materials at present.

Some other points / interactions:

Retention and archiving policies

these need to be clarified, though retention can be bound into lifecycle if Theon / Prometheus is involved
mechanism for editing of erroneous submissions
potential continuing requirement for (live, public web) publication
Preferred plan is to interface with PURE subject to various conditions / requirements
Access to project (and materials) by external examiners
Data interface with Theon and/or projectsdb

Deliverables:

Why:

The existing hodge podge of scripts is 15+ years old and dependent on the legacy school database infrastructure (and data as it turns out). Certain aspects are broken. Access to projects and materials (outside school) must be done on an adhoc basis by admin staff.

When:

By April for UG4, August for PGT.

How:

Initially just a requirements capture to define the academic/student side.

Resources: Previous sources of information:
https://www.inf.ed.ac.uk/cgi-bin/submission_form_ug4.pl
https://www.inf.ed.ac.uk/cgi-bin/submission_links_ug4.cgi
http://www.inf.ed.ac.uk/teaching/courses/diss/guide.html#Submission
https://wiki.inf.ed.ac.uk/DICE/MScSubmission
http://www.inf.ed.ac.uk/publications/thesis/msc.html

Plan:

Effort estimate:

1 week (for requirements capture, scoping and proposal)

Lab Exam Virtual Machine Image

boss — Fri, 25 Jan 2013 15:46:27 +0000

Project ID:

261

Current stage:

Manager:

gdutton

Unit:

What:

Description: Produce a VM Image which allows a variant of the current DICE Lab Exam environment to be provided on non-DICE hardware.

Deliverables:

Secure VM Environment on IS Labs

Secure VM Environment on DICE lab desktops

Virtual Machine Image which can be deployed to both of the above

Backend on DICE servers which allows existing exam functions to work with the above

Why:

Customer:

Case statement: As class sizes exceed lab capacity and the popularity of the lab-exam format for assessment increases, it will be necessary to find a mechanism to allow all students to take an exam simultaneously in a way which does not disadvantage any students.

When:

Status:

Timescales: Preferably in time for next year's mock exam diet.

Priority:

Time:
A rough estimate follows.

Notes:

The requirement to perform much of this work offsite and in collaboration with other schools will contribute to latency and effort, the latter of which is hopefully accounted for here.
The below does not include CSO time which should be assumed at approximately 25% of the defiend FTE for starred* and 50% for double-starred** items.

1-2 FTE weeks image creation
3* FTE week image application management, of which
1. 1 wk initial environment creation
2. 2 wks application / environment management tools
2 FTE week redevelopment of paper retrieval / submission tools
3 FTE week redevelopment of server-side tools, of which:
1. 1 wk support of paper retrieval / submission
2. 2 wks logging, backup and recovery mechanisms
3-5* FTE week image deployment and runtime mechanism, of which:
1. 1-2 wk DICE lab deployment incl. redevelopment of "Exam Mode"
2. 2-3 wks IS lab deployment
3-5** FTE weeks testing, of which
1. 1-2 wks (lab / firewall) security testing
2. 2-3 wks procedural and application testing

Risks to the above lie in assumptions made on:

Virtualisation technology
Chosen OS / release for exam image
Ability to achieve parity with teaching software requirements
??

How:

Proposal:

Resources: Will require significant CSO time to assist in physical lab configuration
Will require IS' CO time for IS lab consultation and configuration

Plan: See time below for the rough outline.

Url: https://wiki.inf.ed.ac.uk/DICE/LabExamsVirtual

Other:

Dependencies: IS Public Lab environment.

Risks:

Risk that one or more of the given environments cannot be secured adequately

High risk of unforeseen obstacles on IS equipment given our lack of knowledge in this area

Risk that IS cannot devote the effort required

Risk that project costs more than the repeated setting of multiple (staggered) exams

Risk the development overruns past exam setting / sitting deadlines

Milestones

Proposed date	Achieved date	Name	Description

DICE VM for Staff

boss — Fri, 25 Jan 2013 15:46:25 +0000

Project ID:

252

Current stage:

Manager:

Unit:

What:

Description: This project will develop and roll out a virtualised version of the current DICE installation that will be usable on staff(and research student) desktops within the schools network

Deliverables: A working VM image that can be kept up to date and installed on DICE and Non DICE desktops (windows, mac, linux).
Infrastructure for allowing staff to self install.
Infrastructure to keep the image updated.
Documentation on install ing the image on Windows, Macos, Linux

Why:

Customer: All staff and research students

Case statement: This would allow staff and students who run self managed machiens access to a DICE desktop on their desktop, it would also allow staff direct access to any customised DICE configuration (lab machine, testing release etc) for testing software.

When:

Status:

Timescales:

Priority:

Time:

How:

Proposal: The project should deliver a system allowing users to install a vm on their main client desktop.
It's highly desirable that this be done with the minimum of work from support.

Resources: 4 weeks effort
Consultancy from C(S)Os with expertise in various platforms to test the VM and generate appropriate documentation

Plan: Investigate install methods (e.g. create vm and run pxeinstall vs pregenerated vm image) and implement one.

Other:

Dependencies: Installroot, probably needs to tie into the installroot build process.
networking
develop a delivery mechanism for staff to create an appropriate VM and install the image

Risks: We are effectively handing out rooted DICE images and should look at the implications this involved wrt networked services and DICE

Milestones

Proposed date	Achieved date	Name	Description

Documentation update - RAT unit

boss — Fri, 25 Jan 2013 15:46:24 +0000

Project ID:

246

Current stage:

3_Implementation

Manager:

Unit:

What:

Description: Review and update end-user documentation. Includes porting across any remaining documentation not already on new drupal based site.

Deliverables:

Why:

Customer:

Case statement:

When:

Status:

Timescales:

Priority:

Time:

How:

Proposal:

Resources: 1 week effort

Plan:

Other:

Dependencies:

Risks:

Url: https://wiki.inf.ed.ac.uk/DICE/246RatUserDoc

Milestones

Proposed date	Achieved date	Name	Description

ISS Misc Development

boss — Fri, 25 Jan 2013 15:46:21 +0000

Project ID:

228

Current stage:

3_Implementation

Manager:

Unit: