LDD Automation Frequently Asked Questions

We have noted a number of consistent queries from boroughs on the London Development Database Automation project, which we have answered below.

Note that the live data feed we are producing will be called the ‘Planning London Datahub’—you’ll see reference to that in our answers.

For an overview of the project, please visit the main page.

When does the LDD shut down?

Due to the server situation we expect the existing LDD system to be switched off by the end of May 2020. Not every back office system will meet the LDD data requirements, so manual data entry to complete LDD returns will still be required for most authorities, until we can switch on the new questions on the Planning Portal. 

How will this year’s starts and completions exercise work?

LDD Officers should continue with their LDD work (including the starts and completions) until the database is switched off. Once your back office system connector is in place we will pull through all live applications into our new Elasticsearch. Authorities will be able to view this feed on Kibana and will be asked to send supplementary data to the project team. Manual data entry will be required until we can switch on the new questions on the Planning Portal. Please note that we will only be asking for the current LDD data so we can maintain our time series and carry out the statutory returns in September. The project team will provide forms for any supplementary data required and you will also be able to use these if the LDD is switched off before your connector is in place. This year’s starts and completion exercise will look slightly different because of the Covid-19 pandemic. The GLA does not consider site visits essential and recommends that officers utilise desk based solutions. We are working with the UK Space Agency to get you further assistance with the exercise.

What about applications that are not made online?

Our objective is to ensure that applications made through means other than an online portal (e.g. by paper, email) are still treated as valid by the borough, but also comply with the new data standard. The goal of this project is to reduce administrative burden, not to increase it.

First, it’s important to consider the scale of paper applications across London. Based on data we received from LPAs between July 2018 and July 2019, a total of 66,917 applications were received that would meet our data requirements*. Of these:

  • 16% (10,747) of all applications received across London were outside of the Planning Portal system (i.e. paper submissions).
  • 6,283 Prior Notifications submissions were received of which 80% (5,087) were not submitted through the Planning Portal.
  • 47% of all applications received outside of Planning Portal were Prior Notifications. Therefore, we anticipate the number of paper submissions received is likely to halve when boroughs adopt the new data standard, and in concert with this, when Planning Portal releases its new Prior Notifications digital forms.

*Our data does not include applications to Listed Building Consent, Trees Works, Advertisement Consents, Non-Material Amendments and Pre-Apps as we are not collecting these types of applications in our data standard.

When applications are received outside of the Planning Portal (or other supported electronic submission portals) we will be supporting boroughs to collect the relevant data from the applicant by providing the borough with an alternative online form to send to the applicant to complete.  Once completed, the applicant or borough can submit this form to us, and we will upload this into our system, verifying it has been successful. We will keep boroughs up to date on applicants that haven’t supplied the required data, so together we can try to keep the dataset comprehensive.

In the rare event that data is not submitted to us via this digital form, we will treat it as a null return. Initially the GLA will not chase for data unless the application entails creation or loss of a dwelling.


Won’t this make the planning application forms long, unwieldy, and an undue burden on developers? 

In many cases, the GLA’s additional fields logically fit within or alongside existing questions. The GLA and Planning Portal have ensured that any new requirements either amend existing questions or dovetail within them wherever possible, so the data is only requested once. This will limit the impact of answering questions when filling out a planning application. 

Please follow this link to see an online demonstration of how Planning Portal’s new submission forms, including our questions, will work.  We invited members of all LPAs to attend this session, which included an open question and answer forum.

The GLA has consulted with the development industry through workshops and requests for responses to the Planning Data Standard circulated through industry publications. To date, the majority of developers and agents have not deemed the Planning Data Standard to be an undue burden.

The GLA has been careful to delineate different data requirements for different types of development—Prior Approvals, Homeowners, and Others (including Major schemes). This will ensure that those least able to answer complex questions (homeowners completing their own applications without professional service providers involved) are still able to submit applications without additional burdens placed upon them. For more information on the questions asked for each type of development, see the LDD Automation Developer Questions spreadsheet.

The GLA is providing comprehensive guidance against each new question, to be included within the Planning Portal interface.


Won’t this impose excessive validation requirements on boroughs? How will data quality be assured? What if boroughs want to check the data? 

The GLA does not expect boroughs to validate the contents of the new fields—the intention is to impose no additional burden on validation officers when going live this summer. The GLA will closely monitor the data received, assessing its completeness, particularly in the first months of the project. This will provide feedback to the GLA—for instance, if the same question is left blank repeatedly, the GLA will investigate the issues with that particular question and the guidance surrounding it.

Boroughs seeking to correct or amend data submitted by applicants might limit our ability to learn from the data.  All published data will have a statement explaining the source of the data.

The Planning Data Standard is not expected to be a static data requirement imposed through validation from Day 1. Instead, by allowing the data to be collected as-is, without the additional step of validation, the GLA can maximise what it learns in this initial release and can make fixes to the Standard, in partnership with boroughs, going forward. The GLA intends to convene a working group comprised of voluntary borough representatives to monitor the data standard and the data received in the early stages of the project.

By publishing the data in a public, easily accessible feed, the GLA expects that the data quality will improve over time—agents and developers will be aware that their scheme’s information will be visible widely, which should disincentivise any intentional misrepresentations. We do predict that the Planning Officer responsible for a particular case will quickly double-check key fields to confirm they conform with any negotiations made throughout the planning process before proceeding to a decision, but this is similar to the types of activities currently undertaken at this stage of the process.

Although a borough is not obligated to validate, update, or confirm any of the new fields, boroughs will have the option to submit corrections to the London Planning Datahub throughout the lifecycle of an application.  If you are keen to do this, ideally please do so post decisions being made. In order for Boroughs to submit changes to an application we will initially be providing an online form that will allow borough staff to submit their changes to us to update the system. At a later date, we will be looking to provide a more elegant and user-friendly change submission portal.

Some boroughs may be concerned about ensuring the data is accurate at the point a decision is made.  After recent discussions with most boroughs in London, we propose to create a two-week window after decisions are made for boroughs to ensure the data is correct before it is published on our website, if they wish to. As part of the solution we have built, we are creating two versions of our new reporting tool – an internal one and a public-facing one. The internal version will be available to the GLA and all LPAs to check and correct the data before it is published on the public version. This, we hope, will alleviate some of the concern about the accuracy of the data at this key stage in the process, but we welcome any additional suggestions in this area.


Is all of the information requested relevant to Planning?

Yes.  The requirement for all of the additional information is based on sound land use planning considerations. The GLA is preparing a full explanation and justification on the london.gov.uk website. This will enable LPAs to point applicants querying the need for the data to a consistent answer. We will also step in and support any appeal process should it be required. 


What about Permitted Development Requirements?

Questions have been raised about the legality of asking additional information to support prior approvals.  

Government are increasingly looking to LPAs to give them evidence about the impact of prior approvals. We are not making the new data mandatory; however the gaps in the dataset from any applicant not wanting to provide the information will be obvious, at which point we will have to take a view as to whether to complete them, or issue any evidence with an acknowledgement that some of the core data is missing.

We will continue to work with MHCLG (one of the funders of this project) to review the data submitted and establish whether the base form needs updating to match our data requirement.


Does this project raise any GDPR issues? 

All data carries potential GDPR implications now. The issue is whether the data qualifies as ‘personal data.’ We are not seeking to extract or collect any personal data, with the exception of the site address (we are not looking for the applicant, agent, or contact details).  At present the site address appears on a number of public registers; as such it carries minimal risk and liability.

We will, however, need to complete a data sharing agreement defining the data we are looking to extract from borough systems and how it will be used. 


What happens when a planning application is amended on the Planning Portal?  

Given our aim to reduce burden on LPAs wherever possible, we have identified that there is room for improvement in the current process for updating planning application information within Planning Portal. Currently, if an applicant makes a change to their application in Planning Portal, only updated supporting documents are passed to LPAs’ back office systems. There is no ability to update machine-readable fields on the planning application. To address this, we have designed our process so that updates to the new London Planning Data Standard questions made within Planning Portal will automatically update the GLA’s database. This will unlock the potential for applicants to own their own data on the Planning Portal and keep it up to date throughout negotiations with the LPA – something they are keen to do from our discussions with them!

We are eager for this process to be as streamlined as possible, keeping the case officer informed of changes when they happen but without overloading them with notifications. 

At a high level the process will work as follows:

  • A new application is submitted to the local authority via a submission portal
  • Data is passed to the GLA database including the Planning Portal Reference Number and version number.
  • At key stages of the application process, our system will check if any new versions of the application have been submitted and if so, update our system with the latest data.
  • Once a decision has been made, the application on the portal will be locked so that the applicant cannot submit further changes.

Our system will automatically look for changes on the Planning Portal to an application using the following criteria:

  1. When the application is valid (i.e. the first time we get the information from the back-office system)
  2. When we receive notification of any change to the back-office fields including:
    •       Amendment to the description
    •       Change to the valid date
    •       Change to the dates of consultation
  3. When a decision is made

This will ensure we have the latest information submitted by the applicant in our data feed, throughout the lifecycle of an application.  We will also provide the option to notify the LPA when a new version is received.

In addition, our system can flag potential issues in the data which may need further investigation.  For instance, if the description of the proposal has changed but none of the other fields have been updated soon after, this may indicate an issue with those fields that needs looking into.  As we learn more about how applications are amended, we will be looking to build further ways to flag potential issues.

From an applicant’s perspective, if they are logging in to access the same application and make small changes in Planning Portal, they will only have to adjust information in the relevant fields (in other words, the system will pull up their prior answers for them to edit).


What happens if an applicant submits an additional application (e.g. a reserve matters application) that supersedes the main application?

When submitting applications using the new questions on the Planning Portal, the user will be asked if the application supersedes (partially or completely) a previous application and the reference number(s) in question.  Our system will recognise these applications are linked and account for this accordingly.


What happens if a borough disposes of an application?

One of the most challenging parts of this projects has been dealing with 35 sets of different business processes and ways of defining the same meaning.  All of the LPAs have kindly been involved in mapping into a single ‘language’ or terms how and what different things mean. This will mean that there is a single way of saying that a planning application has been granted or refused.

A borough’s business process for recording that an application has been finally disposed of will dictate how it gets recorded in our database.  

E.g. If a borough’s business process entails removing the valid date, it will be as if the application has never been submitted and will disappear from our records. However if a borough uses a decision, such as ‘disposed of,’ it will appear on the register in that format.


What if an application crosses the boundaries of the GLA area?

We have not put in a place a way of recording only the elements of the applications that fall within the GLA area as yet, however we are looking to do this in later phases of the project.

In the interim, we will be able to spatially map the applications that cross the boundaries and we will have to undertake a manual review.


Is the system going to collect both approvals and refusals?

Yes. The new system will keep a register of all planning applications, their relevant data, and decisions (including appeal decisions).  

This new data set will enable planning authorities to have a clearer view on the outputs of the planning process, and a greater insight into the barriers of delivering housing and other objectives.


What wording do LPAs need to include on Local Validation Checklists?

We are looking for one additional requirement to be added to Local Validation Checklists:

‘Data required by the Greater London Authority Planning Data Standard’.


What safeguards are in place to protect borough systems and data? 

The systems that are being set up either use a scheduled report or a one-way API.  This means that at no time does the GLA, or any other body, have access to any data held in borough computer systems.

To ensure that the datasets submitted to us are protected, the GLA will be holding two copies of the database: one copy for us and LPAs to access, and a second copy that will be available to third parties, including the public. This public dataset will be refreshed nightly, so that it is up to date and remains uncorrupted.

We will be monitoring uploads daily, so that we can see quickly if anything in the automated system has failed. We are currently entering into agreements with back office system providers for maintenance of all of the connectors, meaning that we can all be satisfied that should they fail there is an efficient and robust mechanism to get them back online swiftly.


What will the ‘Go Live’ process entail, and will GLA be fully testing the system?

The ‘Go Live’ process is made up of two keys stages – enabling the new questions on the Planning Portal and enabling the APIs at each Local Planning Authority to export data from their back-office systems to our new Planning London DataHub.

We have been working with the Planning Portal and each of the four back office suppliers to plan these activities in detail.  The main highlights are:

  • Each supplier will install their APIs to export the data to us on a few pilot sites.
  • Once installed, our project team will create dummy applications on the Planning Portal’s test system to feed down to the pilot authorities’ test back office system.
  • The application will then be processed in the test back office system and exported to our system automatically.
  • Once verified that the supplier's API is working successfully, these will then be rolled out to all the authorities with the same back office supplier. This includes further testing and enabling ready for live.
  • On the agreed date the Planning Portal will enable each authority to receive new applications including our data standard questions. On the same date, the API at the relevant authority will be made live so we can receive the data.

We aim to have as many boroughs go live with the new questions on the Planning Portal at the same time; however, the solution has been built to enable each authority individually, so that we can adopt a phased approach to ‘Go Live’ if needed.  We are continuing to review our ‘Go Live’ plans and may adopt a phased ‘Go Live’ plan if this is deemed to be more suitable.


How will boroughs access and analyse the data?

Data will be stored in a search and analytics engine called Elasticsearch. Kibana is the tool used to interact with the data held in Elasticsearch. Kibana will be made available to local authorities and to the public, providing data visualisation and search capabilities. The GLA, with input from local authorities, will create a set of standard dashboards that provide answers to common queries. The GLA will also help local authorities set up their own, bespoke dashboards capturing questions specific to their areas of interest.

Kibana has a range of features to make analysis of data simple. Dashboards can be created in Kibana where visualisations and searches can be held on one page for side-by-side analysis. A dashboard can be tailored to meet a user’s requirements such as viewing and monitoring live annual monitoring report indicators.


What if I have further questions or concerns?

Please contact Simon Long with any additional queries and we will do our best to answer them.

Share this page