DICE development projects - timc

Manager:

Unit:

Summary:

Migrate the systems used by InfHR in the old School database into Theon

What:

This covers a number of areas. The first is driven by a business analysis being done by admin. This will determine the shape of the structure under Theon. It is unlikely that there will be much migration (data or structure) from legacy in the end. It is more likely that the processes will be changed to be driven by upstream data from HR systems and the VRS, reducing the duplicated data entry currently done. Associated with this migration will be review/conversion of old conduits generating staff and institute landing pages to move into the Portal.

Why:

The InfHR systems will in due course be the only thing remaining in the old School DB setup (database, outgoing conduits framework). After this migration the old systems can be shut down removing a maintenance burden. At the moment data in 3g comes from the HR upstream source, but web pages are generated from the old 2g InfHR maintained data. There is an increasing interference/drift of the main Theon data processing from the data that is still maintained locally by InfHR resulting in discrepancies. This would be corrected by migration. The migration is an opportunity for admin to improve/automate more of the HR processes (to take advantage of features supported by Theon but not in the legacy systems).

When:

Largely this is waiting on business analysis by admin, probably not until mid 2014.

How:

TBD based on business analysis.

Effort estimate:

Not possible at this stage but likely to be more than 4 weeks

Other:

Dependencies:

Risks:

IGS Misc Development

timc — Tue, 03 Sep 2013 08:15:39 +0000

Project ID:

293

Current stage:

Manager:

Unit:

https://svn.theon.inf.ed.ac.uk/trac/milestone/Consolidation%20of%20Research%2...

Summary:

Process the operational tasks requiring small development effort arising out of the live Theon deployment. Tasks are individually identified in Trac.

Why:

Live use of Theon continuously identifies issues and enhancements for the IGS that are generally queued due to other work. Completing these assist the IGS in using the administrative processes now driven from Theon. Each issue is however always assessed for cost/benefit on a case by case basis.

When:

Due to the Trac being a live system with new tickets going in all the time there can be no explicit effort calculated, instead an effort budget should be allocated and tickets addressed one by one until the budget is used up.

How:

In constant contact ISS to identify prioritisation of tasks. Work on each task in priority order until the allocated effort budget is expired.

Final report URL:

Other:

Dependencies:

Risks:

Project Submission

boss — Fri, 25 Jan 2013 15:46:29 +0000

Project ID:

270

Current stage:

2_Evaluation

Manager:

Unit:

Summary:

Sort out UG4/PGT project submission/access

What:

Description: Project to tie together project and dissertation submission for UG4 and possibly MSc and other project / materials storage.

This is required because the existing retrieval mechanism (based on 2g Theon data) needs to be scrapped but there is no clear alternative in the new world. And in any event it would make considerable sense to tie together the many different scripts and data sources which make it so difficult to find project materials at present.

Some other points / interactions:

Retention and archiving policies

these need to be clarified, though retention can be bound into lifecycle if Theon / Prometheus is involved
mechanism for editing of erroneous submissions
potential continuing requirement for (live, public web) publication
Preferred plan is to interface with PURE subject to various conditions / requirements
Access to project (and materials) by external examiners
Data interface with Theon and/or projectsdb

Deliverables:

Why:

The existing hodge podge of scripts is 15+ years old and dependent on the legacy school database infrastructure (and data as it turns out). Certain aspects are broken. Access to projects and materials (outside school) must be done on an adhoc basis by admin staff.

When:

By April for UG4, August for PGT.

How:

Initially just a requirements capture to define the academic/student side.

Resources: Previous sources of information:
https://www.inf.ed.ac.uk/cgi-bin/submission_form_ug4.pl
https://www.inf.ed.ac.uk/cgi-bin/submission_links_ug4.cgi
http://www.inf.ed.ac.uk/teaching/courses/diss/guide.html#Submission
https://wiki.inf.ed.ac.uk/DICE/MScSubmission
http://www.inf.ed.ac.uk/publications/thesis/msc.html

Plan:

Effort estimate:

1 week (for requirements capture, scoping and proposal)

ISS Misc Development

boss — Fri, 25 Jan 2013 15:46:21 +0000

Project ID:

228

Current stage:

Manager:

Unit:

What:

Description: Process the operational tasks requiring small development effort arising out of the live Theon deployment. Tasks are individually identified in Trac.

Deliverables: A number of closed Trac issues.

Why:

Customer: ISS and Academic Staff

Case statement: Live use of Theon highlighted a number of enhancement and loop closing issues for the IGS that were queued due to higher priority ITO work. Completing these will assist the ISS in using the administrative processes now driven from Theon and help the IGS aspect of the larger Theon project to be closed.

When:

Status:

Timescales:

Priority:

Time: Due to the Trac being a live system with new tickets going on all the time there can be no explicit effort calculated, instead an effort budget should be allocated and tickets addressed one by one until the budget is used up.

How:

Proposal:

Resources:

Plan: Contact ISS to identify a prioritisation of the tasks.
Work on each task in priority order until the allocated effort budget is expired.

Other:

Dependencies: RAT support.

Risks:

URL: https://svn.theon.inf.ed.ac.uk/trac/report/24

Milestones

Proposed date	Achieved date	Name	Description

Teaching/Research Software Requirements Management

boss — Fri, 25 Jan 2013 15:46:21 +0000

Project ID:

231

Current stage:

Manager:

Unit:

What:

Description: The project will develop tools that allow staff and C(S)O's to check on current software and it's status. It should generate reports and reminders to ensure that these lists are kept up to date.

Deliverables: A dynamic set of web pages showing current teaching and research sotware requirements, the current status of the requirement and the status of the software on DICE.

A tool that allows staff to query what software is currently available on DICE, or is in the various repositories we use to build dice.

An interface that allows staff to specify and alter software requirements.

A set of reports which drive the various deadlines involved and inform staff/COs of the current status of required software.

Why:

Customer: staff and COs

Case statement: We do not currently have a definitive set of software requirements for each of the teaching courses. This leads to a messy scramble at the start of some courses where staff are asking for software at the last minute, software is unavailable because of licensing issues, upgrades break coursework and software is removed for various reasons ranging from obsolescence to security issues. We are also probably expending resources maintaining software that is no longer used by anyone. To fit in with the new software requirements process some management software would assist in keeping staff aware of what their courses have/will require, allow them to update the requirements, keep track of the status of their requirements and allow CO staff to manage and plan upgrades more efficiently.

When:

Status:

Timescales: Ideally we should have something in place in the run up to 2012/13 session. But too late for that so more likely to be something for 2013/14.

Priority:

Time: 4 weeks

How:

Proposal:

Resources:

Plan:

Agree functionality of a set of admin tools, that, for example:

allows staff/students to query requirements based on a course
allows them to edit requirements if they teach on the course.
restricts editing to valid times depending on course development schedule and # of weeks before course is taught
allows C(S)Os to update current status
where suitable, shows current version of software installed
shows any suitable additional information (updates, etc)
allow staff to easily query the standard dice installed software
allow staff to easily query software in reqositories we are
using
allow staff to define tests to verify their coursework
produce various reports to drive the process including:
- midsummer reminder of software status for staff teaching
  course
- 3-2-1 weeks to deadline reports for staff where software is
  unverified
- weekly monthly reports to interested parties on software for
  all courses

Other:

Dependencies: This should hook into the "course in a box stuff" somehow even if it's just look and feel. Some of the data will come from the school database. For new courses it should integrate with webmark somehow

Risks:

Milestones

Proposed date	Achieved date	Name	Description

Consider how we can make use of new IS IDMS

boss — Fri, 25 Jan 2013 15:46:18 +0000

Project ID:

216

Current stage:

Manager:

Unit:

compexec

What:

Description: Consider how we can make use of new IS IDMS (eg for visitor registration). This is from the CEG wish-list. So far as we can remember, the idea was that once the new IDMS has come fully on-line we should be able to take some/more feeds from it, either to give us new data that we don't have or have to re-key, or to provide a more direct feed for things we already take from elsewhere.

Deliverables: Ultimately less effort and fewer errors entering data.

Why:

Customer:

Case statement: This is from the CEG wish-list.

When:

Status: From the CEG wish-list

Timescales:

Priority:

Time:

How:

Proposal:

Resources:

Plan:

Other:

Dependencies: People who know their way around: prometheus; the database; the new IDMS.

Risks:

Milestones

Proposed date	Achieved date	Name	Description

School Database Revamp: Phase 1B & 2B

boss — Fri, 25 Jan 2013 15:46:16 +0000

Project ID:

207

Current stage:

Manager:

Unit:

What:

Description: Phase 1 and 2 of this project were concerned primarily with the re-factoring of the back-end and front-end of the service. In addition phase 2 incorporated the intergation of the old Access systems used by the IGS for PGR management and ISS for duties allocation. Work to do with replacing the old feeds came under the EUCLID Interoperability project although there was crossover. As the project closes out it leaves a few core requirements left unaddressed. From a front end perspective these are moving the ITO out of the old database model and into the new one, including in particular an IGS level of EUCLID sync integration and a removal of the obsolete Class structure which causes a high level of additional complexity, ITO effort and support effort in the system. From a back end perspective these are closing the change management control loop (managed schema maintenance).

Deliverables: The primary deliverable (2B) is the ITO moved across to using the new 3G model joining the IGS. This entails a number of things.

update to use the new generic sync framework for EUCLID data (replacing the legacy retrofit feed)
create new desktops to replace legacy desktops in the new web front end
migrate live data from the old model to the new model allowing for model changes and now sync'ed data
update of all internal functions to correlate and in particular assessment calculation functions for BoE processes (significant for dropping of Class) and student registration functions
creation of new portal reports and/or migration of old reports to the new model
modification or deprecation of any remaining legacy reports
migrate all remaining ITO/IGS historical data and deprecate/archive all associated aspects of the old model
implement prioritised enhanced functionality made available in the new model and as originally requested by the ITO
replacement for genrep (either portal reports or completion of reporting interface in new UI)

The secondary deliverable (1B) is the DDL change management process. This would mean

changes to "schematic" and XSLT transforms to support "differential DDL's" between revisions (for at least the core types of schema change)
a mechanism to push/pull changes to the DDL
implementation of the schema-schema within XML schema to allow for maintenance desktops using the new web front end (replacing the old client).

Why:

Customer: Primarily the ITO but also IGS/ISS, School and Computing.

Case statement:

Phase 2B:

The existing ITO model is now fragile. It contains many legacy artifacts (such as Class) which are no longer necessary. Their existence unnecessarily complicates the administration process for ITO staff and causes a much larger than necessary maintenance burden both for ITO staff and systems staff. It also increases the risk of errors in assessment data as much "meta" data needs to be created to drive the process manually and which is not obvious to administration staff (and consequently requires extensive systems staff support). The structure of courses and programmes has changed significantly over the last 15 years and with the introduction of EUCLID. This makes it hard to cleanly and reliably sync/integrate the old model with the center. The old model has also been "extended" with kludges over this time period to cater with assessment structure which it originally was never designed to handle (EUMI being one case, but it also never automatically handled P/T taught status and those have always required manual intervention). All of this introduces instability, risk and an increased maintenance burden on the system.

The new EUCLID feed is currently retrofitted onto the existing ITO model as it would have been very complex to back port the feed and generic sync framework into the old model. Hence the old feed remains in all but name. This is very fragile as the new EUCLID data does not fit particularly well with the old scheme, there is mapping data to be maintained manually and there is much "local knowledge" built into the feed process itself which is no longer valid and increases the frequency of errors (both in the raw data and then in knock on for BoE processes).

Both the above result in an additional cross-checking and anomaly reporting administration burden for the ITO and a considerable concern for over all data integrity in the system.

Phase 1B:

At the moment change management has to be carried out manually, i.e. the schema is updated using the old database client and the same change is then made manually on the database server using raw DDL. The risks of this are self-apparent. It also makes it harder to run parallel development servers to trial changes across the whole system, much like the LCFG develop -> testing -> stable release mechanism which was part of the original plan. This is a largely unsustainable situation in the longer term.

When:

Status:

Timescales: To be worthwhile the user fronting changes need to be completed well in advance of the 2011 session starting. It has been suggested that the resit period could be used for QA testing of the modified exam process and assessment functions.

Priority: High -> Critical

Time: 24 FTE Weeks (Moving ITO to 3G) + 8 FTE Weeks (Closing Change Management Loop)

Breakdown of effort (most concurrent) for ITO move as follows:

Custom Desktops 4wks
There are at least four to be built (Course, Programme, Student, Finance)
but more as well, each with numbers of separate panels. Estimate includes
repeated QA/refine cycles. Entirely replace all legacy ITO desktops.

Model Refinement 2wks
This runs concurrent with other blocks - picks up any model changes that
are required by actual development of desktops, sync and functional
processing. Also changes for tutor allocation.

Anomaly Correction 2wks
Handling of "programme change" and other edge case conditions requires a
solution (also for PGR but more so PGT/UG). Not strictly critical in this
block.

EUCLID Interoperability 2wks
Extend the existing sync configuration (PGR only) to cover PGT and UG.
This will also mean adding new supplemental sync feeds. The new syncs
will entirely replace the existing "retrofit" mechanism, meaning we
also need a "course registration" sync. We will also need to extend
the "Staff" internal sync as more data will be needed for the ITO from
the old model which is not migrating yet.

Functional Migration 6wks
Functional processing needs to be modified to account for removal of the
"Class" layer. This includes both registration and assessment functions.
Registration will also need changes due to EUCLID status changes and to
modified sync processes. Also covers implementation of session roll
forward sync and functional processing (new, integrated into database
rather than the existing report and manual process mish mash). The
assessment functions need adapted for new P/T, sessionless programme
and year structures. Much QA needed.

Data Migration 4wks + 8wks
The first effort estimate is for migrating current student data
(plus 2012 upcoming) only (in this block), the latter for historical
data (in this block).
The approach will probably require development of a table by table
mapping function that will allow old and new to run in parallel.
Complexities abound in here and include mapping old class structures
into course structures. There is also a question of how much we can
one-time sync and complement instead. Vague for now.

Roles 2wks
Existing per-person roles replaced with per-function roles (as for PGR)
and migrated to new model.

Report Migration 2wks + 4wks (BoE) + 4wks (remainder)
4wks (genrep replacement)
Existing reports need to be reviewed and those that need to be retained
migrated. This is expected to be a manual report by report process. Most
will be candidates for the portal. Something needs to be done with "genrep"
and paramaterised reports (potentially the original "Reporter" aspect of
the new client/server which is as-yet unimplemented). The BoE reports need
to have the functional processing contained in the reports folded back
into the database (and changed to account for "class" drop and other
structural assessment alterations, e.g. project handling). The effort
is given as a minimum up-front + guesstimate on migration costs. With
the exception of BoE (and related) reports these would be planned to be
done on-demand (at short notice during the session rather than up-front
as part of the main project block). The genrep replacement effort is
given as implementation of Reporter (including conduit meta-data markup),
there may be other options. Effort on BoE reports could potentially be
done during the first semester.

Enhancement ?wks
On top of basic migration there is enablement of new functionality built
into the new model to meet original requirements. Effort required is to
expose data via "desktops" and present data via new "reports" in most
cases. Many enhancements will however also require additional functional
processing. It is not possible to estimate this - depends on which of
the enhancements are prioritised. Not clear that any are "critical" in
this block though (as over anything above).

How:

Proposal:

Resources: Effort.

Plan:

Other:

Dependencies: Outside of the database service itself none of note.

Risks:

Milestones

Achieved date	Name	Description
2011-08-10	Desktop	Create Theon desktop for core ITO functionality (as specified).
2011-08-10	Feeds and Syncs	Create new feeds and syncs under TheonSync to provide all the data needed for core ITO functionality.
2011-08-17	Integration	Transfer functional processing for core ITO functionality (course registration processes). Update conduits as necessary (e.g. submit data).
2011-08-24	Full QA	Completion of ITO desktop and data QA. Also includes one-off transfer of assessment structure to new model. Also includes upcoming sync work for continuous rollover support.
2012-04-30	Year/Programme	Implement Assessment Functionality (L787)
2012-03-31	Course	Complete Course Assessment Functionality (L788)
2012-01-07	Course	Implement Course Assessment Functionality (Sem1/VUG)
2012-05-30	BoE	Complete BoE process desktops/reports, including portal replacement for genrep
2012-05-07	Diets	Complete multi-diet management, or pend and use fixed diet for now

Collaborative LaTeX System - Remedial Work

boss — Fri, 25 Jan 2013 15:46:16 +0000

Project ID:

206

Current stage:

Manager:

Unit:

What:

Description: The Collaborative LaTeX system has been running for a number of years using prototype code. Although this is still working there have been an increasing number of bugs, particularly associated with the move to F13, causing user frustration. The original repository framework was designed for a small number of users but now there are many more repositories and it has become very slow and could do with being changed to accommodate this. An earlier project did some redesign work on the Coltex client command but was ultimately de-prioritised and stalled and this work was left unfinished. Backup is still only a daily mirror and with the large number of users and amount of data held there could now be quite a lot of lost effort were the disks to fail.

Deliverables: A completed production client with bug fixes. Changes made in client code and backend code for a re-structured repository model. An svnsync backup mechanism to complement the existing mirroring.

Fix and/or resolution to RT #s
46005,
37181,
48853,
48907,
51299,
51858,
51931,
52024,
52178,
52256,
52319,
52375,
52375,
52624,
52684,
52953,
53763,
54779,
56091,
54778

Why:

Customer: All School Staff, but primarily Academic/Research.

Case statement: Day to day operational maintenance load for Coltex had been effectively non-existent up until the last year or so where it has started to ramp up significantly (more tickets in the last six months than the last few years put together). This is a sign that the prototype code is starting to creak. The admin commands have become very slow and unwieldy, particularly in initial use, due to the large number of repositories now being managed. Substantially more critical data is now held in the system and the use of an svnsync (near real time) like backup is increasingly necessary.

When:

Status:

Timescales: Although the existing system has been patched up to work it is costing increasing effort to maintain the code base now. The risk of disk failure is ever present.

Priority: Medium -> High

Time: 4 FTE Weeks

How:

Proposal:

Resources:

Plan:

Other:

Dependencies: None of note.

Risks:

Milestones

Proposed date	Achieved date	Name	Description

Push R&T Packages Upstream

boss — Fri, 25 Jan 2013 15:46:10 +0000

Project ID:

176

Current stage:

Manager:

Unit:

What:

Description: Re-package our research and teaching software for F13/4 packaging compliance and submit upstream so they are built as part of the standard Fedora distribution. This project is an extension of Package Teaching & Research Software for Fedora which did the groundwork. It was hoped that packages would then be rebuilt to be compliant and submitted upstream as part of the normal release re-build cycle (or as and when individually upgraded) but this has not happened due to operational timing constraints, hence this more focused approach.

Deliverables: All our locally packaged software that is publically deployable and is not packaged anywhere else available in the standard Fedora13/4 distribution.

Why:

Customer: All

Case statement: For us is efficiency gains in moving from one release to the next. For the user packages should be available faster on new platforms, as well as the ability to obtain most of our research and teaching packages on non-DICE platforms.

When:

Status:

Timescales: There is no specific deadline to meet, although in advance of the next summer release upgrade cycle would make sense.

Priority:

Time: There are probably about 40 core packages to do, plus dependencies. They vary greatly in complexity. While the upstream push process can take a lot of time (months) it takes only hours per-package in real time. The main cost is building for Fedora packaging compliance, one days per package on average maybe.

How:

Proposal:

Resources: Effort only.

Plan:

Other:

Dependencies: None.

Risks:

Milestones

Proposed date	Achieved date	Name	Description

School Database - Changes for Prometheus

boss — Fri, 25 Jan 2013 15:46:09 +0000

Project ID:

169

Current stage:

Manager:

Unit: