ADD378 Hosting and development of London DataStore
The London DataStore is one of the GLA’s flagship projects and is a platform through which many of the Smart London Plan objectives are delivered. The London Datastore provides:
Integration of data with the main london.gov.uk website;
Automatic, less labour intensive, gathering of data from other data stores and catalogues;
modern interfaces, better able to provide data in machine readable formats (for use by web and app developers);
Better data search and management tools, making it more user friendly for Londoners and technical developers and easily updateable;
Opportunities to share our investment and intellectual property with the London boroughs and other cities through Open Source development.
The DataStore is hosted on behalf of the GLA on secure cloud servers to provide sufficient flexibility and robustness.
That the Assistant Director approves expenditure of up to £40,000 for two years’ hosting and development of the London DataStore.
Part 1: Non-confidential facts and advice
1.1. When it was launched in 2009, the original London DataStore (LDS) was a highly innovative first step into open data for London and the UK. In the intervening period, a number of other cities around the world have copied the GLA. Digital technology and the open data agenda have also advanced significantly in this time.
1.2. The development of the enhanced LDS was procured using a competitive procurement process in April/May 2014. The contract for £60,000 was won by DataPress whose team had carried out much of the work on the national data.gov.uk.
1.3. A full working product was delivered and launched on 23rd October 2014 and included the initial 12 months of development and support. Provision under this arrangement is to continue until March 2016 with additional costs of £10,000 incurred. The DataStore is built on open source products CKAN and WordPress and uses Amazon Web Services and as such is not restricted to one service provider and as such a mini-competition will be held to identify further provision.
1.4. The provision of support to date covered, and future provision would seek:
• Development of the platform
• Web hosting
• updates to CKAN
1.5. The DataStore consists of:
- A searchable repository of approximately 1,600 tables of key data about London
- supporting information about each dataset (including licensing and descriptions)
- A system of user accounts, which allows people to register for notifications when datasets that they are interested in are updated
- A publishing area which allows other trusted organisations (for instance LFB) to publish and update their own datasets
- Management functions to help ensure that datasets are kept up to date
- A website system for publishing pages of further information and tools to help people make use of the data
The following objectives and outcomes are expected through continued support to the DataStore:
- 24 months cloud hosting;
- Deployment, testing and updating of security to prevent unauthorised access to user information or the back end systems;
- Daily backups;
- Implementing updates to CKAN; and
- Development of additional functions to the platform, including:
- Improvements to the data preview functions;
- Reporting on dataset usage; and
- Applying design guidelines from new GLA website.
The London DataStore is delivered in way that meets the GLA’s duty not to discriminate against people with protected characteristics. In particular, it has been designed to be at least as accessible to people with visual impairment as per the main GLA website.
4.1 Key risks and issues
Compatibility with other data stores and catalogues
The DataStore uses internet standards and open data standards to ensure that it is able to harvest data from other data stores and catalogues regardless of their software platform. It is harvested itself by the national data.gov.uk and the ODI certification process.
Capacity and resilience of infrastructure
The DataStore is hosted in an Amazon Cloud (where multiple instances are kept in different physical data centres) leading to over 99.97% up time in 2015.
Problems caused by either not applying updates to the underlying software, or through the update process
This takes up a significant amount of staff time for the data.gov.uk team, but is handled by DataPress for the London DataStore.
4.2 links to Mayoral strategies and priorities
Smart London Plan
Delivery of an enhanced London DataStore directly fulfils the following ‘Measures of Success’:
- Evolve the London Datastore into a global exemplar platform by 2016
- Double the number of users on the Datastore and Dashboard by 2018
It also supports the Smart London Borough Partnership in its work to encourage boroughs to free up London’s local level data – either by harvesting data from borough data stores or by providing a ready-made platform for boroughs to use if they wish.
Supporting Open Source
The GLA has a strong track record in using Open Source software (for instance Drupal for the main GLA website) and we are continuing to move away from proprietary software to open source (e.g. moving from Oracle to Postgres). This approach is economically advantageous to the GLA, avoids ‘vendor lock-in’ and supports the SME / high-tech community.
4.3 Impact assessments and consultations
The DataStore receives nearly 50,000 visits per month and is a flagship project for the GLA, receiving an international award for Open Data Publishing from the Open Data Institute in June 2015. The impact to the GLA of not continuing to operate it and improve it would be damaging to the organisation’s reputation.
Consultation on the functional requirements took place across the Intelligence Unit (who use and maintain the current DataStore on a daily basis) and more widely across the GLA. Through feedback forms and the Borough Data Partnership would also have feedback from users on the next set of developments.
Consultation on the interaction with the main GLA website and the GLA’s infrastructure has taken place with the Technology Group.
5.1. The total estimated cost of the £40,000 expenditure will be funded from the Ordnance Survey Budget code (GG.0220.003.003). The financial implications per financial year will be as follows:
5.2. Any changes to this proposal, including budgetary implications will be subject to further approval via the Authority’s decision-making process. All appropriate budget adjustments will be made.
5.3. The Intelligence Unit within the Communities & Intelligence Directorate will be responsible for managing this contract and ensuring that all expenditure and associated activities comply with the Authority’s Financial Regulations, Contracts & funding Code and Expenses & Benefits Framework.
Procurement route: Mini-competition for quotes
Announcement [if applicable]
Delivery Start Date [for project proposals]
Project Closure: [for project proposals]