DICE development projects - squinney

Manager:

Unit:

Summary:

Virtual desktop infrastructure

What:

Provide a decent virtual desktop infrastructure. For researchers (e.g. staff, postdocs, postgrads) who have DICE desktops, the service might utilise the computing resources of the desktop machine allocated to the particular user. This is not possible with NX but there are probably other systems out there which can be used, this project will evaluate the various options to come up with a proposal for future implementation.

Why:

The recently deployed NX service has been wildly popular and users would like an even better service. It's clear that running just one server for this purpose is not going to be sufficient. We have to limit resource access (e.g. memory) so that it is reasonably usable by all. Most staff have DICE desktop machines so it would be nice to have some system to allow remote graphical access.

Effort estimate:

2 weeks

Other:

Dependencies:

Risks:

User accessible login reports

boss — Fri, 25 Jan 2013 15:46:26 +0000

Project ID:

254

Current stage:

5_Completed

Manager:

Unit:

https://wiki.inf.ed.ac.uk/DICE/FinalProjectReport254

What:

Description:

The recent development of a database to store interesting events that have been filtered out from the centrally-stored syslog files means that it is now relatively straightforward to generate reports of logins for our users.

The aim of this project is to produce a mechanism (eg individual email reports, cosign-protected web page) for users to see from which remote machines their account has been accessed - including ssh and weblogin logins.

Deliverables: A working mechanism.

Why:

Customer: All users

Case statement: Recent evidence shows us that compromised accounts are a serious, and not infrequent, threat to the integrity of our computing systems.

When an account has been compromised, an attacker will often log on to systems accessible by that account to check for any local root exploits. If there are no current root exploits on those systems, the attacker will quietly logoff and keep the account details for a future attempt.

Providing a mechanism by which users could see from which external machine their account was accessed would allow users to spot suspicious activity on their account, hopefully before any malicious damage is done.

When:

Status:

Timescales:

Priority:

Time:

How:

Proposal:

Resources: Two weeks

Plan:

The simplest approach to providing access to this information for our users is to create a cosign-protected CGI script. This would use SQL to securely query the data stored in the buzzsaw database stored on the central log server.

The intention is to write this script using the python programming language. This provides the opportunity to gain valuable experience with an alternative programming language. This clearly means that the project will take longer but that has been factored into the effort allocation.

The web interface does not need to be complicated but will ideally have facilities to allowing basic paging (e.g. month by month) and sorting of the data (e.g. by date, source host, target host).

Separation of the query code from the presentation by using a templating system is preferable to make it easier to modify the interface in the future.

Along with the web interface we will send an email to each user every month summarising all the logins from the previous month. This means that all users will be encouraged to check their login history regularly.

Effort taken (days):

Final report URL:

Other:

Dependencies:

Risks:

Milestones

Achieved date	Name	Description
2012-11-23	sql	Develop the python code to do the necessary SQL queries against the buzzsaw database.
2012-11-30	cgi	Develop a cosign-protected cgi script which will use the sql query functions to find a list of logins for the user.
2012-12-14	presentation	Develop a web interface using a templating system to present the data to the user in a reasonable format with basic paging and sorting functionality.
2012-12-21	email	Develop a system to send an email to each user every month which summarises their login activity.

Build Farm enhancements

boss — Fri, 25 Jan 2013 15:46:24 +0000

Project ID:

245

Current stage:

Manager:

Unit:

What:

Description: This project will fix various bugs in the PkgForge build farm software and deliver a number of feature enhancements.

Deliverables:

Why:

Customer: Informatics computing team. Potentially also external users so they should not be ignored but they are not a priority.

Case statement:

Since its entry into service in Informatics the PkgForge software build farm has been very successfully used to port LCFG and DICE to the SL6 platform. A number of minor bugs and niggles with the interface have been discovered that users would like resolved.

Due to a shortage of time with the original project the web interface is rather basic and needs a lot of work to provide a better user experience, it also needs reworking internally to use the correct APIs.

There are also a few new features which are very desirable, for instance, gpg signing of packages would help enhance the security of our systems.

When:

Status:

Timescales:

Priority:

Time:

How:

Proposal:

Resources:

Plan:

Add systemd support for daemons (1 day)
Fix reported bugs (1 day)
Rework the web interface (4 days)
Feature enhancement: Add support for signing packages (2 days)
Feature enhancement: Soft/hard build timeouts (1 day)
Feature enhancement: Job resubmission (2 days)
Feature enhancement: Live view of build logs (2 days)
Feature enhancement: Allow building against a target without submission (1 day)
Feature enhancement: Add query functionality to the command line client (2 days)
Feature enhancement: Enhance package quality checks (rpmlint, etc) (2 days)

Other:

Dependencies:

Risks:

Milestones

Proposed date	Achieved date	Name	Description

LCFG client refactor - code cleanup

boss — Fri, 25 Jan 2013 15:46:20 +0000

Project ID:

225

Current stage:

3_Implementation

Manager:

Unit:

https://wiki.inf.ed.ac.uk/DICE/FinalProjectReport225

What:

Description:

The aim of this project is to cleanup up the code of the LCFG client
(primarily rdxprof). This will not be a complete rewrite and we do not
intend to add new features. The project will however go beyond basic
refactoring by addressing a couple of longstanding bugs which have
been seen to regularly cause problems for users. The aim is to bring
the code up to "Modern Perl" standards and make it more portable.

Note that completely redesigning the code (such as reimplementing
using a proper Object-Oriented framework like Moose) is outside of the
scope of this project.

This is expected to require 4 weeks effort, the project has been split
up into 11 tasks with 19 days allocated which leaves 1 day of
leeway. Time allocations for tasks have been rounded up to the nearest
half day of effort, some will take less, some might take more. For
some of the tasks the changes might be quite small but a good
understanding of the code will need to be acquired first.

This is a child project of project 139

Deliverables: Cleaner LCFG client component code

Why:

Customer: Internal

Case statement: There is a significant risk that the existing code base might not run "as is" on future DICE platforms. Further developments of LCFG are also stalled because of the difficulty of extending the existing code.

When:

Status:

Timescales:

Priority:

Time:

How:

Proposal:

Resources: Four weeks.

Plan:

1. Rework the rdxprof script as a Perl module. (0.5 days)

This makes the code more manageable and potentially reuable, it also
allows the creation of a code-level test suite for the various
subroutines. This will reduce the risk involved in future development
and maintenance. The rdxprof script will still exist but will just
call functions from the new module (a similar approach was taken with
the LCFG server mkxprof script).

2. Basic test suite (0.5 days)

Add basic compilation tests for all modules which can be run each time
the package is built.

3. Run code through perltidy (1 day)

This will make the code indentation and formatting more consistent and
thus more readable. It will be done to the same rules as the LCFG
server code base.

4. Refactor code to satisfy perl critic level 5 (2 days)

This stage will refactor the code to deal with the worst cases of
badly written Perl based on the findings of the perlcritic tool.

5. Refactor code to satisfy perl critic level 4 where possible (2 days)

This stage will refactor the code to eliminate the majority of the
lesser code issues. The experience of improving the LCFG server code
showed that it is not entirely possible to satisfy level 4. There are
likely to be some cases where it is essential to implement
functionality in a way which might not be considered "best practices".

6. Enable 'warnings' and eradicate any use of undefined variables (4 days)

None of the LCFG client modules or the rdxprof script have the Perl
warnings pragma enabled. We should enable this in all code to avoid
nasty surprises. Based on the experiences of refactoring the LCFG
server code this is likely to result in a substantial number of
warnings being generated related to the usage of undefined
variables. Eradicating this use of undefined variables is likely to be
the biggest part of the project, in particular, finding them takes
quite a bit of effort when they lie in uncommonly used code paths. It
also requires a good working knowledge of the code to understand when
there is a specific meaning attached to a variable being in an
undefined state.

7. Improve option handling (1 day)

The current handling of command-line options in rdxprof is fairly poor
and does not support the longer descriptive option name style. It
would also be very useful to add support for a simple configuration
file. This would make it easier to separate the client code and the
LCFG component into separate projects in SVN.

8. Eradicate hard wired paths (1 day)

There are a number of paths hardwired into the code using the CMake
macro syntax (e.g. @FOO@). This means it is currently not possible to
run rdxprof unless you are the root user, even if you can read the XML
profiles. They should all be converted into command-line options with
the current values being the defaults for root and new defaults being
introduced for the non-root user scenario. These macros also need to
be removed so that we can convert the code into a standard Perl
package.

9. Split client and LCFG component (1 day)

The LCFG client code should be split out into a separate project in
SVN. This would allow the packaging to be reworked so that it is
buildable as a standard Perl module which can be submitted to
CPAN. The LCFG component would still be built as normal using the
CMake based system.

This follows the way in which the LCFG server code is packaged. It
should be relatively straightforward, and can reuse the Module::Build
module created for that package.

10. Bug Fix: Rework the network port handling (4 days)

On startup the client attempts to listen on a network port so that it
can receive signals from the server when a new profile is
available. If the port has already been taken by another process then
the client hangs forever. This is particularly undesirable behaviour
as it leaves a machine in an unmanageable state, we should rework this
code so that it fails better. The client should be able to just log
the error and carry on. There is no issue with continuing without the
ability to listen for notifications since the client will still be
able to regularly poll the server for changes, the responsive will
only be slightly degraded.

11. Bug Fix: Regularly lookup the addresses for servers (2 days)

Currently, once the client has looked up the address for a server it
continues to use that address until the process is restarted. This
makes it very difficult to move a server (for instance, by just moving
a DNS CNAME from one host to another). This data should be stored in a
time-limited cache so that lookups are redone occasionally.

Final report URL:

Other:

Dependencies:

Risks:

Milestones

Proposed date	Achieved date	Name	Description

System Security Enhancements

boss — Fri, 25 Jan 2013 15:46:20 +0000

Project ID:

224

Current stage:

5_Completed

Manager:

Unit:

https://wiki.inf.ed.ac.uk/DICE/FinalProjectReport224

What:

Description: This project will enhance the security of our systems in various ways based on the proposals made in the report into the compromise of the SSH servers.

Deliverables: More secure DICE infrastructure.

Why:

Customer: Internal.

Case statement:

When:

Status:

Timescales:

Priority:

Time:

How:

Proposal:

The expectation is that all of the "main aims" will be completed within the time allotted. If there is spare time then work will be done on the "other possibilities" section of the project.

Resources: 4 weeks effort.

Plan:

Main Aims

System Auditing:
We should enable the Linux audit daemon on all DICE machines with a reasonable set of default rules. This will, for instance, make it possible to trace all modifications to important parts of the filesystem and monitor the execution of setuid root files. For important servers we would also like to have a facility to securely send the audit logs to a remote central server.
Enhance logging and automate the log monitoring:
In particular we need to automatically monitor the syslogs for all DICE systems to spot events, such as kernel panics, which might indicate attempts by a user to leverage a local security hole.
Regularly monitor the file-system:

It is clear that we need to monitor the file-system of our servers for unauthorized changes. In particular we should monitor:
1. All setuid root scripts which are runnable by other users.
2. All files which are not owned by RPMs
3. Important directories such as /etc/.
Some of this can be done using the Linux audit daemon but it would be good to have additional coverage. This can be done with intrusion-detection software such as AIDE (Advanced Intrusion
Detection Environment) which is a file and directory integrity checker.
Regularly sweep for root-kits:

We should regularly run software to check for the evidence of root-kits having been installed, e.g. chkrootkit and rkhunter. In particular, running more than one check would denitely be useful as they function quite differently.
Bootable image for investigations:
Develop a bootable ISO which can be used to carry out investigations on suspect machines. This would be based on our standard installroot image with additional packages, such as chkrootkit and rkhunter, which are useful to have readily available. This could be made available via CD, USB and PXE. There is also a need for a read-only filesystem (in our case delivered via AFS) for running scripts such as chkrootkit which use a number of separate tools which could be subverted if they are coming from the local disk.

Other Possibilities

Use SELinux:

We do not currently use the SELinux framework which would almost certainly have prevented this intrusion occurring. To run it in full "strict" mode might require a great deal of effort to make it work with LCFG. However, it should be possible to get a lot of
the benefits whilst using the lesser "targetted" mode, which is the standard for Redhat systems. It is denitely worth some further investigation.
Shared database for failed logins:

We should investigate whether it is possible to create a database of failed logins which is shared across all DICE machines with external SSH access. This could be used to
automatically block all SSH access for hosts which are attacking our systems.
SSH Honeypot:

We could use an SSH honeypot which has an SSH firewall hole but which cannot actually be accessed by any users due to restrictions in the SSH daemon config and the
access.conf file. The location of this service would not be advertised to our users so it
would only be found by attackers who are port-scanning our network for targets. This
could then feed into a shared database of banned hosts.

Some parts of the work for this project have already been done or just need some extensions. In particular, the audit daemon work is mostly complete but needs rolling out to all machines.

Final report URL:

Other:

Dependencies:

Risks: We get hacked again...

Milestones

Proposed date	Achieved date	Name	Description
	2012-06-16	3. Monitor fs	File-system intrusion monitoring
2012-09-30	2012-06-30	4. Check for ro	Regularly sweep for root-kits
	2012-07-30	6. SELinux	Consider using SELinux
	2012-07-16	5. ISO	Develop a bootable image for investigations.
	2012-08-30	7	Consider using a shared database for failed logins
2012-08-17	2012-05-31	1. Log Checking	Enhance logging and automate log monitoring.
2012-06-18	2012-05-31	0. Auditing	System Auditing

Remote hardware management for desktops

boss — Fri, 25 Jan 2013 15:46:06 +0000

Project ID:

156

Current stage:

Manager:

Unit:

What:

Description: It would be useful to be able to reliably remotely manage the power settings for desktop machines. In particular it would be good to offer our users the facility to remotely turn on/off and reboot their own desktop machines. When a user is logged in via the console they are allowed to do all these things so they shouldn't be limited when working remotely.

This could also provide the opportunity to improve power savings in Informatics as we would not have to leave any desktop machine permanently turned on just in case a user wanted to use it remotely.

Further opportunities are available to remotely configure BIOS settingsand to do asset management for the Inventory without requiring machines be turned on.

Deliverables: Interface for users to remotely control the power settings of their own desktop machine.

Why:

Customer: Informatics users.

Case statement: Requested by user: https://rt3.inf.ed.ac.uk/Ticket/Display.html?id=45583

When:

Status:

Timescales: A rough estimate of 3 weeks effort for delivery of a system which allows users to remotely control their machines.

Priority:

Time:

How:

Proposal:

Resources:

Plan: Investigate Intel AMT. Develop an authenticated system for remote power control using AMT, maybe based around remctl. Develop a user interface, possibly available via the web, authenticated with cosign, to allow the desktop 'owner' to control their machine.

Other:

Dependencies: Currently, Intel AMT is only available in desktop PCs with Intel Core 2 processor with vPro technology so the right hardware is required.

Risks:

Milestones

Proposed date	Achieved date	Name	Description

LCFG Beginners Guide

boss — Fri, 25 Jan 2013 15:46:02 +0000

Project ID:

141

Current stage:

Manager:

Unit:

What:

Description: A guide which would cover all the basics needed to get started with LCFG. This would be mainly based on the LCFG workshops run in June 2007.

Deliverables: A guide to getting started with LCFG, probably in both HTML and PDF formats.

Why:

Customer: External users of LCFG.

Case statement: It's currently quite hard to quickly get started with LCFG.

When:

Status:

Timescales:

Priority:

Time:

How:

Proposal:

Resources:

Plan:

Other:

Dependencies:

Risks:

Milestones

Proposed date	Achieved date	Name	Description

Rewrite the LCFG syslog component

boss — Fri, 25 Jan 2013 15:46:01 +0000

Project ID:

138

Current stage:

4d_Dropped

Manager:

Unit:

What:

Description: It has noted that the syslog component is in need of some attention. It suffers from a number of problems:

The paths to the syslogd and klogd programs are hardwired in the shell code - prevents use with any other compatible daemon.
All the log file paths are hardwired in the shell code.
It uses the old-style LCFG templating system to generate m4 files which are then processed to generate the final syslog configuration file. This also means that resources contain m4 statements rather than simple values.

Fedora now uses rsyslog as the default syslogd, and has done since November 2007. This means it is fairly likely to be in RHEL6. It is also used by Debian and Ubuntu so it looks like it is becoming the de facto standard choice for a modern syslogd. It supports the old config file but also has lots of other nice features which could be useful. It is also highly desirable to make the component generic enough to support multiple different syslogds. Alternatively it might be good to write a specific rsyslog component which makes it possible to enable the fancy features via LCFG. We might even want to do both of these.

See the rsyslog website for more details on rsyslog.

There is an rsyslog package available in SL5 but it's version 2.0 which is somewhat behind the current stable version.

Deliverables: A new LCFG syslog component.

Why:

Customer:

Case statement:

When:

Status:

Timescales:

Priority:

Time:

How:

Proposal:

Resources:

Plan:

Other:

Dependencies:

Risks:

Milestones

Proposed date	Achieved date	Name	Description

LCFG Client Refactoring

boss — Fri, 25 Jan 2013 15:46:01 +0000

Project ID:

139

Current stage:

Manager:

Unit:

What:

Description: A project to rewrite the LCFG client to be cleaner and easier to maintain and develop, without significant change in functionality.

This has been deliberately split out from the LCFG Core Refactoring project so that we can track server and client redevelopment separately.

Deliverables: New LCFG client code.

Why:

Customer: All LCFG users.

When:

Status:

Timescales:

Priority:

Time:

How:

Proposal: Include porting the LCFG core, and some other critical components, to some other Linux distribution, eg Debian, to maintain portability.

Resources:

Plan:

Other:

Dependencies:

Risks:

URL:http://wiki.lcfg.org/bin/view/LCFG/LcfgBrainstorm

Milestones

Proposed date	Achieved date	Name	Description

Automated upstream package repository mirroring

boss — Fri, 25 Jan 2013 15:46:01 +0000

Project ID:

140

Current stage:

Manager:

Unit: