You are here

School Database Revamp: Phase 1B & 2B

Project ID: 
Current stage: 

Description: Phase 1 and 2 of this project were concerned primarily with the re-factoring of the back-end and front-end of the service. In addition phase 2 incorporated the intergation of the old Access systems used by the IGS for PGR management and ISS for duties allocation. Work to do with replacing the old feeds came under the EUCLID Interoperability project although there was crossover. As the project closes out it leaves a few core requirements left unaddressed. From a front end perspective these are moving the ITO out of the old database model and into the new one, including in particular an IGS level of EUCLID sync integration and a removal of the obsolete Class structure which causes a high level of additional complexity, ITO effort and support effort in the system. From a back end perspective these are closing the change management control loop (managed schema maintenance).

Deliverables: The primary deliverable (2B) is the ITO moved across to using the new 3G model joining the IGS. This entails a number of things.

  • update to use the new generic sync framework for EUCLID data (replacing the legacy retrofit feed)
  • create new desktops to replace legacy desktops in the new web front end
  • migrate live data from the old model to the new model allowing for model changes and now sync'ed data
  • update of all internal functions to correlate and in particular assessment calculation functions for BoE processes (significant for dropping of Class) and student registration functions
  • creation of new portal reports and/or migration of old reports to the new model
  • modification or deprecation of any remaining legacy reports
  • migrate all remaining ITO/IGS historical data and deprecate/archive all associated aspects of the old model
  • implement prioritised enhanced functionality made available in the new model and as originally requested by the ITO
  • replacement for genrep (either portal reports or completion of reporting interface in new UI)

The secondary deliverable (1B) is the DDL change management process. This would mean

  • changes to "schematic" and XSLT transforms to support "differential DDL's" between revisions (for at least the core types of schema change)
  • a mechanism to push/pull changes to the DDL
  • implementation of the schema-schema within XML schema to allow for maintenance desktops using the new web front end (replacing the old client).

Customer: Primarily the ITO but also IGS/ISS, School and Computing.

Case statement:

Phase 2B:

The existing ITO model is now fragile. It contains many legacy artifacts (such as Class) which are no longer necessary. Their existence unnecessarily complicates the administration process for ITO staff and causes a much larger than necessary maintenance burden both for ITO staff and systems staff. It also increases the risk of errors in assessment data as much "meta" data needs to be created to drive the process manually and which is not obvious to administration staff (and consequently requires extensive systems staff support). The structure of courses and programmes has changed significantly over the last 15 years and with the introduction of EUCLID. This makes it hard to cleanly and reliably sync/integrate the old model with the center. The old model has also been "extended" with kludges over this time period to cater with assessment structure which it originally was never designed to handle (EUMI being one case, but it also never automatically handled P/T taught status and those have always required manual intervention). All of this introduces instability, risk and an increased maintenance burden on the system.

The new EUCLID feed is currently retrofitted onto the existing ITO model as it would have been very complex to back port the feed and generic sync framework into the old model. Hence the old feed remains in all but name. This is very fragile as the new EUCLID data does not fit particularly well with the old scheme, there is mapping data to be maintained manually and there is much "local knowledge" built into the feed process itself which is no longer valid and increases the frequency of errors (both in the raw data and then in knock on for BoE processes).

Both the above result in an additional cross-checking and anomaly reporting administration burden for the ITO and a considerable concern for over all data integrity in the system.

Phase 1B:

At the moment change management has to be carried out manually, i.e. the schema is updated using the old database client and the same change is then made manually on the database server using raw DDL. The risks of this are self-apparent. It also makes it harder to run parallel development servers to trial changes across the whole system, much like the LCFG develop -> testing -> stable release mechanism which was part of the original plan. This is a largely unsustainable situation in the longer term.



Timescales: To be worthwhile the user fronting changes need to be completed well in advance of the 2011 session starting. It has been suggested that the resit period could be used for QA testing of the modified exam process and assessment functions.

Priority: High -> Critical

Time: 24 FTE Weeks (Moving ITO to 3G) + 8 FTE Weeks (Closing Change Management Loop)

Breakdown of effort (most concurrent) for ITO move as follows:

Custom Desktops 4wks
There are at least four to be built (Course, Programme, Student, Finance)
but more as well, each with numbers of separate panels. Estimate includes
repeated QA/refine cycles. Entirely replace all legacy ITO desktops.

Model Refinement 2wks
This runs concurrent with other blocks - picks up any model changes that
are required by actual development of desktops, sync and functional
processing. Also changes for tutor allocation.

Anomaly Correction 2wks
Handling of "programme change" and other edge case conditions requires a
solution (also for PGR but more so PGT/UG). Not strictly critical in this

EUCLID Interoperability 2wks
Extend the existing sync configuration (PGR only) to cover PGT and UG.
This will also mean adding new supplemental sync feeds. The new syncs
will entirely replace the existing "retrofit" mechanism, meaning we
also need a "course registration" sync. We will also need to extend
the "Staff" internal sync as more data will be needed for the ITO from
the old model which is not migrating yet.

Functional Migration 6wks
Functional processing needs to be modified to account for removal of the
"Class" layer. This includes both registration and assessment functions.
Registration will also need changes due to EUCLID status changes and to
modified sync processes. Also covers implementation of session roll
forward sync and functional processing (new, integrated into database
rather than the existing report and manual process mish mash). The
assessment functions need adapted for new P/T, sessionless programme
and year structures. Much QA needed.

Data Migration 4wks + 8wks
The first effort estimate is for migrating current student data
(plus 2012 upcoming) only (in this block), the latter for historical
data (in this block).
The approach will probably require development of a table by table
mapping function that will allow old and new to run in parallel.
Complexities abound in here and include mapping old class structures
into course structures. There is also a question of how much we can
one-time sync and complement instead. Vague for now.

Roles 2wks
Existing per-person roles replaced with per-function roles (as for PGR)
and migrated to new model.

Report Migration 2wks + 4wks (BoE) + 4wks (remainder)
4wks (genrep replacement)
Existing reports need to be reviewed and those that need to be retained
migrated. This is expected to be a manual report by report process. Most
will be candidates for the portal. Something needs to be done with "genrep"
and paramaterised reports (potentially the original "Reporter" aspect of
the new client/server which is as-yet unimplemented). The BoE reports need
to have the functional processing contained in the reports folded back
into the database (and changed to account for "class" drop and other
structural assessment alterations, e.g. project handling). The effort
is given as a minimum up-front + guesstimate on migration costs. With
the exception of BoE (and related) reports these would be planned to be
done on-demand (at short notice during the session rather than up-front
as part of the main project block). The genrep replacement effort is
given as implementation of Reporter (including conduit meta-data markup),
there may be other options. Effort on BoE reports could potentially be
done during the first semester.

Enhancement ?wks
On top of basic migration there is enablement of new functionality built
into the new model to meet original requirements. Effort required is to
expose data via "desktops" and present data via new "reports" in most
cases. Many enhancements will however also require additional functional
processing. It is not possible to estimate this - depends on which of
the enhancements are prioritised. Not clear that any are "critical" in
this block though (as over anything above).



Resources: Effort.



Dependencies: Outside of the database service itself none of note.



Proposed date Achieved date Name Description
2011-08-10 Desktop Create Theon desktop for core ITO functionality (as specified).
2011-08-10 Feeds and Syncs Create new feeds and syncs under TheonSync to provide all the data needed for core ITO functionality.
2011-08-17 Integration Transfer functional processing for core ITO functionality (course registration processes). Update conduits as necessary (e.g. submit data).
2011-08-24 Full QA Completion of ITO desktop and data QA. Also includes one-off transfer of assessment structure to new model. Also includes upcoming sync work for continuous rollover support.
2012-04-30 Year/Programme Implement Assessment Functionality (L787)
2012-03-31 Course Complete Course Assessment Functionality (L788)
2012-01-07 Course Implement Course Assessment Functionality (Sem1/VUG)
2012-05-30 BoE Complete BoE process desktops/reports, including portal replacement for genrep
2012-05-07 Diets Complete multi-diet management, or pend and use fixed diet for now