DICE development projects - iainr

Hadoop cluster revamp

iainr — Mon, 26 Aug 2013 15:09:49 +0000

Project ID:

291

Current stage:

0_Pending

Manager:

iainr

Unit:

rat-unit

Summary:

Rework the existing hadoop component and headers to provide better management and monitoring

What:

Rework the hadoop component to
* Allow individual nodes to be added and removed from jobtracker
* Allow additional diskspace on nodes to be added and removed from the hdfs filesystem
* Better support the use of multiple clusters.

Develop a small test suite that could be used to automatically test the operation of the cluster and possibly integrate this with nagios.

Deploy performance/usage monitoring software to better quantify the cluster usage.

Why:

The current component works well at configuring a fairly static cluster but adding and removing nodes, both to run jobs and as disk nodes in hdfs is awkward and time consuming. The configuration often does not reflect the actual nodes in the cluster which has led to data loss in the past.

Being able to remove nodes when the cluster was not being used would allow us to sleep nodes (where supported) and reduce the cluster power footprint. Ideally this could be scripted and allow the cluster to automatically minimise it's power usage without affecting it's usability.

The current hadoop configuration (version and installed packages) is largely dictated by the extreme computing course. Being able to set up multiple hadoop or hdfs clusters on the existing hardware with a minimum of overhead would allow us to respond more flexibly to requests from other groups of users where better security or a more recent version of hadoop is required.

Having a standard set of test jobs would allow us to monitor the clusters health and hopefully reduce downtime.

How:

Core:
Rewrite the hadoop component to fully manage the configuration of all the clients and the starting and stopping of the various daemons.
Produce a mechanism that would as much as possible automate managing the membership of the cluster (say via a spanning map) and also manage which member nodes are currently active within the cluster.
Take the basic hadoop example tutorial and automate execution of it so that it could be run as a nightly cron job. Investigate any hadoop nagios clients and integrate them into our nagios setup.
Generate basic monthly logs of cluster usage.

Optionally:
Write a daemon that monitors the cluster usage and dynamically activates and sleeps nodes to match the cluster usage.
Investigate any hadoop nagios clients and integrate them into our nagios setup.
Check for any useage/accounting type packages available for hadoop or write one if there is none available.

Effort estimate:

2-3 weeks for reworking the component, 2-3 days for the test setup 2+ weeks for the optional stuff.

Other:

Dependencies: Any development would have to fit in alongside current cluster usage

Risks:

Display SAT info on AT foyer display screens

ascobie — Thu, 28 Mar 2013 10:46:04 +0000

Project ID:

278

Current stage:

4_Signoff

Manager:

iainr

Unit:

rat-unit

Summary:

Display SAT personal timetables on AT foyer display screens

What:

We are obliged to present an individual timetable for students from AY 2013/2014 onwards. Our plan is to present info from our RBS onto displays in the foyers in AT. This project will implement this info display

When:

Needs to be implemented for start of AY 2013/2014

How:

modify RBS:
Add course feed from theon
Add unique identifier to appropriate repeated bookings
Add scripts to import/export required data
add to theon
Incoming conduit to take feed from RBS
Marry groups assigned in theon with bookings in RBS
generate html to be published on AT Display screens

Final report URL:

https://wiki.inf.ed.ac.uk/DICE/FinalProjectReport278

Other:

Dependencies: I will have to lean heavily on Tim and Grahams Theonn knowledge and Neils knowledge of the RBS.

Risks: We may not get it done in time

Milestones

Proposed date	Achieved date	Name	Description
2009-06-18	2009-06-20	rbscourse	Add course feed to RBS development
2013-07-01	2013-06-14	rbskey	Add unique identifer to RBS entries
2013-07-14	2013-07-28	theonfeed	Add feed to import data from rbs into Theon
2013-08-07		theonsync	Sync data to active Theon tables
2013-08-14	2013-08-14	theoncode	Marry rbs bookings with Theon generated groups
2013-08-20		theonreport	Write report to generate html for AT screens and website
2013-08-24		test	Test deployment
2013-09-07		deploy	Final depolyment
2009-06-18	2013-06-20	rbscourse	Add course feed to RBS development

VM / cloud options survey

boss — Fri, 25 Jan 2013 15:46:26 +0000

Project ID:

256

Current stage:

3_Implementation

Manager:

iainr

Unit:

compexec

What:

Description: A survey of options for providing a local virtual machine hosting / cloud type services.

Deliverables: A document listing options and known requirements.

Why:

Customer: Primarily computing team to decide whether producing a service is worth doing and, if so, what technology to use.

Case statement:

When:

Status:

Timescales: This should be done between now and Christmas

Priority:

Time:

How:

Proposal:

Resources: One week

Plan: Look at cloud technologies and generate a list of those we could deliver
Look at technologies Currently available in the university.
Look at any current provision within the school.

Formulate a list of questions
generate an online survey and submit to staff/phd students

Other:

Dependencies: None

Risks: None

Milestones

Achieved date	Name	Description
2013-01-03	results	Collate results of survey
2012-12-03	Publish	Publish the survery to general staff
2012-11-25	pilot	Generate a pilot survey and pass to cos for comments
2012-11-10	survey	Decide on survey provider and familiarise with it's use.
2012-11-22	review	Review current tech and finalise possible platforms.

DICE VM for Staff

boss — Fri, 25 Jan 2013 15:46:25 +0000

Project ID:

252

Current stage:

0_Pending

Manager:

iainr

Unit:

rat-unit

What:

Description: This project will develop and roll out a virtualised version of the current DICE installation that will be usable on staff(and research student) desktops within the schools network

Deliverables: A working VM image that can be kept up to date and installed on DICE and Non DICE desktops (windows, mac, linux).
Infrastructure for allowing staff to self install.
Infrastructure to keep the image updated.
Documentation on install ing the image on Windows, Macos, Linux

Why:

Customer: All staff and research students

Case statement: This would allow staff and students who run self managed machiens access to a DICE desktop on their desktop, it would also allow staff direct access to any customised DICE configuration (lab machine, testing release etc) for testing software.

When:

Status:

Timescales:

Priority:

Time:

How:

Proposal: The project should deliver a system allowing users to install a vm on their main client desktop.
It's highly desirable that this be done with the minimum of work from support.

Resources: 4 weeks effort
Consultancy from C(S)Os with expertise in various platforms to test the VM and generate appropriate documentation

Plan: Investigate install methods (e.g. create vm and run pxeinstall vs pregenerated vm image) and implement one.

Other:

Dependencies: Installroot, probably needs to tie into the installroot build process.
networking
develop a delivery mechanism for staff to create an appropriate VM and install the image

Risks: We are effectively handing out rooted DICE images and should look at the implications this involved wrt networked services and DICE

Milestones

Proposed date	Achieved date	Name	Description

Documentation update - RAT unit

boss — Fri, 25 Jan 2013 15:46:24 +0000

Project ID:

246

Current stage:

3_Implementation

Manager:

iainr

Unit:

rat-unit

What:

Description: Review and update end-user documentation. Includes porting across any remaining documentation not already on new drupal based site.

Deliverables:

Why:

Customer:

Case statement:

When:

Status:

Timescales:

Priority:

Time:

How:

Proposal:

Resources: 1 week effort

Plan:

Other:

Dependencies:

Risks:

Url: https://wiki.inf.ed.ac.uk/DICE/246RatUserDoc

Milestones

Proposed date	Achieved date	Name	Description

SL6 upgrade for RAT Unit

boss — Fri, 25 Jan 2013 15:46:19 +0000

Project ID:

220

Current stage:

3_Implementation

Manager:

iainr

Unit:

rat-unit

What:

Description: Upgrade all Research and Teaching Unit servers and services to use the SL6 Operating system.

Deliverables: All Services and Servers using SL6.

Why:

Customer: School

Case statement: This is inevitable as required by the eventual loss of SL5 and discontinuation of support by RHEL, SL and Informatics.

When:

Status: RAT servers are already starting to be converted to SL6 as new hardware or services are installed. The project is effectively underway so this is an attempt to track and coordinate progress.

Timescales:

Plan for conversion in place by the end of 2011 with some key services converted by then.

All services to be theoretically SL6-compatible by Feb 2012 with plan drawn up for any exceptions.

Research servers to be running SL6 by end

Priority:

Time: Estimated at four weeks, not including Beowulf cluster.

How:

Proposal:

Resources: One CO to coordinate; additional CO time to convert own services; CSO to schedule and perform machine re-installation.

One (existing) SL6_64 server for purposes of compute server user testing.

Plan:

Produce planning document detailing work required.

Notify component/header managers of any changes required.

Request user acceptance / notify user base for each server.

Schedule reinstallation - then perform it.

Final report URL:

https://wiki.inf.ed.ac.uk/DICE/Project220FinalReport

Other:

Dependencies: Multiple services

Risks: Research users unwilling/unable to convert code.
Proprietary software where modern (64-bit / glibc) builds are unavailable.

URL: https://wiki.inf.ed.ac.uk/DICE/RATSL6Upgrade

Milestones

Proposed date	Achieved date	Name	Description

Sl6 Dice desktop

boss — Fri, 25 Jan 2013 15:46:17 +0000

Project ID:

210

Current stage:

5_Completed

Manager:

iainr

Unit:

rat-unit

What:

Description: additional work required to bring LCFG SL6(inf) (#131) up to working DICE desktop

Deliverables: components and configuration to bring the sl6 inf release up to DICE standard.

Why:

Customer: The primary customer is the School of Informatics. Other schools may take some of the software/components or modify it fortheir own use.

Case statement:

When:

Status:

Timescales:

Priority:

Time:

How:

Proposal:

Resources: 6 weeks base on the sl5->f13 port

Plan:

Other:

Dependencies:

Risks:

URL: https://wiki.inf.ed.ac.uk/DICE/SL6DICEDesktop

Milestones

Proposed date	Achieved date	Name	Description
2011-06-01	2011-06-01	command line	get the base command line config in place
	2011-06-20	X tweaks	Finalise any DICE tweaks of the X system in place
	2011-06-14	tweaked command	tweak the command line environment for SL6
	2011-06-08	X Environment	Configure Base X Environment for use in SL6
	2011-07-01	core env packag	Have all core env packages in place
	2011-07-08	signoff	submit project for signoff

large file distribution and management system

boss — Fri, 25 Jan 2013 15:46:14 +0000

Project ID:

196

Current stage:

0_Pending

Manager:

iainr

Unit:

rat-unit

What:

Description: The project will develop a mechanism for distributing and managing large files/packages efficiently to DICE workstations. Subject to appropriate requirements it should extend this to distributing files across the School, Unitersity and the Internat.

Deliverables:

A system for managing large files on DICE machines, it should at least.
- support efficient file distribution across DICE via torrent/multicast type technologies
- Allow distributed files to be managed (untarred, removed...) via standard DICE/LCFG tools
- Allow jobs to be scheduled to use the network at off peak times, whilst still allowing for immediate distribution.
A system for securely(?) distributing large files across the school non dice machines, University or internet
An interface allowing select users to manage the above.

Why:

Customer: For the DICE part the C(S)Os, for the extended part staff. Students and the general public may be users of the system

Case statement: We have a number of files and applications on DICE which are too big to ship in rpms and which cannot be broken down into a number of smaller rpms. NFS and AFS are inappropriate technologies for shipping these files to a large number of machines in a short period, torrent or multicast would be more suitable technologies for such a deployment. The Virtualised DICE project is likely to produce more large files and we need a machanism to deploy these files to DICE machines and to manage the files on the DICE machines.

Given such a system it is invisioned that it could be used to distribute the same or similar files to non dice machines in the school, within the university or across the internet.

It is also possible that various research groups may wish to take advantage of having files or datasets available via such technologies.

When:

Status: proposal

Timescales: Before the next academic session would be nice

Priority:

Time:

How:

Proposal: Find out what the requirements are within the school for a torrent/multicast type file sharing service and how appropriate it would be to deploy. Assuming this requirement at least implement a system which would allow us to manage the deployment of large (>2GB files) across DICE. Optionaly deploy a system which would allow staff to make files available to non DICE machines.

Resources: Staff time:
3 days to gather requirements
1 week to look at available technologies
1-2 weeks to implement distribution system
2 weeks to implement management system

Members of the services, MP and infrastructure units would be needed for sort periods on a consultancy basis to determine the best ways to integrate with existing systems.

Possibly further 2 weeks to implement something to distribute files beyond DICE.

Hardware
Depending on the implementation there may be a need for disk space on a number of servers which are geographicall and topologically distributed. It's expected that there would be minimal new hardware requirements.

This project is not likely to be a good candidate for virtualisation.

Plan: The exact plan depends on how widely we want to make this available.

Inverstigate requirements from C(S)Os and non C(S)Os.
Investigate file transportation technologies.
Integrate management of transportation technology with LCFG/DICE.
produce documentationfor C(S)Os

If non DICE distribution is required and both services can use the same platform then:

Check with IS about transport technology across EDLAN if access from outside school is required.
Develop interface to allow staff to distribute files.
Package or document client installations for major OSs (linux, windows, OS, Solaris.

Other:

Dependencies: None

Risks: Even if used correctly this could be a wonderful denial of service mechanism against the school network. We should also be aware of Copyright issues with any external service.

Milestones

Proposed date	Achieved date	Name	Description

Virtualised DICE image - scoping

boss — Fri, 25 Jan 2013 15:46:10 +0000

Project ID:

172

Current stage:

5_Completed

Manager:

iainr

Unit:

rat-unit

What:

Description: A virtualised DICE image has been suggested in a number of meetings.
This would be, for example, a virtualbox image with the current supported
desktop OS and all the teaching applications installed that students
(and staff) could download to their own laptops and desktops. The image would be
preconfigured for use against DICE services (eg AFS, printing etc)

IS have indicated willing to consider running DICE VMs in the IS public labs,
so this should be investigated as part of this project.

This project would investigate the feasibility of creating and maintaining
such an image, including :-

investigating real demand
identifying what is the minimum useful functionality and what extras functionality would be useful
investigating issues such as authentication, image updates etc

This project would probably be a joint project involving MPU and RAT.

Deliverables: A report covering the possible solutions and the likely costs involved.
some exemplar images

Why:

Customer: Staff, students...

Case statement:

When:

Status:

Timescales:

Priority:

Time:

How:

Proposal:

Resources:

Plan:

Get requirements from original requestors.
check for other requirements
check with IS about what they would accept in their labs
check technologies available, shortlist possible technologies
produce exemplar images
look at updating images and authentication
look at deploying images.
write report

Other:

Dependencies:

Risks:

Milestones

Proposed date	Achieved date	Name	Description
2011-07-01	2011-04-12	requirements ca	Get a requirements capture from anyone likely to want to use a virtual DICE (or virtual DICEish) machine
2011-05-18	2011-04-18	techreview	Quick review of available technologies.
2012-02-01	2011-04-18	IScheck	Check with IS about whether they're agreeable to us installing DICE images in their labs.
2011-05-13	2011-04-24	buildimages	create some exemplar images to play with
2011-09-04	2011-04-29	testimages	See how viable using these images in anger is likely to be.
2012-03-03	2011-05-01	report	write report

SICSA web

boss — Fri, 25 Jan 2013 15:46:08 +0000

Project ID:

163

Current stage:

3s_Stalled

Manager:

iainr

Unit:

rat-unit

What:

Description: Redevelop the "look and feel" of www.sicsa.ac.uk keeping the current content and platform (plone)

Deliverables: A new plone theme and associated graphics.

Why:

Customer: SICSA

Case statement: SICSA have requested and CEG have approved.

When:

Status: writup https://wiki.inf.ed.ac.uk/DICE/163SICSA_Web

Timescales: As soon as possible within the T2 framework

Priority: Medium

Time: 2 FTE weeks

How:

Proposal:

Resources: iainr

Plan: look for possible themes, install these on test copies of the sicsa website and hold a meeting of interested persons. Hopefully choose a theme and rework it, have a meeting to discuss results. repeat until the principals are happy with the results. Declare victory.

Other:

Dependencies: None

Risks: Possible time black hole if customer does not have a clear idea of their requirements.

Milestones

Proposed date	Achieved date	Name	Description
2010-04-01	2010-04-01	search	Look for suitable themes matching sicsa's requirements
2010-04-01	2010-04-01	createclone	Generate a number of clone copies of the sicsa website for testing
2010-04-02	2010-04-02	createtests	install each theme on a clone site and tweak graphics etc to match
2010-04-10	2010-04-10	custmeet	Meet with customers to identify best match (if any) and what changes need to be made
	2010-05-11	tweak	make changes to possible theme
	2010-04-30	approvemeet	Meet with customer to approve tweaks
	2010-05-13	implement	install the final theme on live website.
2011-03-24	2011-03-14	writeup	Writeup report
	2011-03-14	Signoff	sign off project