MD2652 London Learner Survey 2020-2022

1.1. The GLA has been actively developing a new approach to measuring the impact of participation in Adult Education Budget (AEB)-funded training courses on learners.

1.2. In November 2019, the Mayor agreed a set of seven priority impact areas for the AEB and endorsed the work underway to develop a new London Learner Survey to measure the impact of AEB provision across these areas. Under the cover of MD2555 (Scoping study for the London Learner Survey and revised structure of the AEB delivery and impact evaluation), the Mayor agreed to commission a feasibility study to explore the options around the design and the potential costs of the London Learner Survey.

1.3. Following a competitive procurement process, NatCen – in partnership with the Learning & Work Institute – were appointed to carry out the feasibility study. The study has assessed different survey design options, as well as identifying the costs and resource implications of those various options.

1.4. In developing a survey model for the London Learner Survey, some key criteria have been kept in mind. These are:

• ensuring a representative sample so that robust analysis can be undertaken. Without a representative sample of learners, basic analysis of important metrics may not be possible even at a London level;

• maximising the number of interviews conducted to ensure that as many providers as possible will have statistically robust data and to allow smaller subgroup analysis; and

• ensuring that the model is efficient and represents good value for money. There comes a point of diminishing returns in terms of analytical power of a survey once it gets to a certain size. It is important to ensure that the survey represents good value for public money.

1.5. Three principal design options for the survey were considered:

• Option 1: A census survey, using only online data collection;

• Option 2: A sample survey, using a mix of online, SMS, postal and telephone data collection; and

• Option 3: A hybrid survey model, using an online census survey of all learners, and then topping-up the data collected with additional data collection, for example through phone or postal surveys to ensure the sample is representative of all AEB learners.

1.6. Option 1 was excluded as it was unlikely to result in a representative sample. Whilst Option 2 would be the most efficient model for securing representative data at London, sub-regional and borough levels, it would provide only limited data at provider level. Many providers would receive limited or no data back, meaning they would have no understanding of their outcomes in some areas. Option 3 is considered to be the optimal survey design, as it maximises the number of interviews and amount of data for providers whilst remaining an efficient and good value model, and ensuring the sample is representative.

1.7. The preferred survey model (Option 3) will have the following components: (1) A baseline questionnaire completed by all learners at enrolment; (2) a follow up online questionnaire sent to all learners a set number of months after the end of their course (probably four or five); and (3) targeted follow up with key learner groups, following review of the data obtained, to maximise the number of providers for which detailed analysis is possible, and to ensure data is collected from under-represented and hard to reach groups.

1.8. The options for the survey were considered by the Outcomes for Londoners Advisory Group (a subordinate body of the Skills for Londoners Board) on 6 May 2020. A smaller sub-group of members met in the week beginning 11 May 2020 to discuss practical considerations pertaining to the recommended option – particularly around coverage of the survey, and operational implications for providers. The Skills for Londoners Board also considered the options and recommendation for the survey at its meeting on 20 May 2020. Members agreed that Option 3 was the preferred option for taking the London Learner Survey forward, subject to additional clarification on expected coverage and costs which has now been provided.

2.1. By introducing the London Learner Survey, the GLA hopes to gain valuable insight into the economic and social outcomes of AEB provision and thus build a compelling business case for more investment from central government in adult education. Better understanding of the impact AEB provision has on Londoners will also help providers to improve their future skills offer and respond better to the local demand.

2.2. The survey would allow for the collection of data on social impact and in-work progression that is not possible through the existing Individualised Learner Record (ILR) infrastructure and would also provide the GLA with more timely information on employment destinations. It would also give London greater control over data on the outcomes achieved by learners. There are currently significant restrictions on the use and publication of the ILR data, which is owned by the Department for Education (DfE).

2.3. It would provide an opportunity to better demonstrate the impact that the AEB has on London learners, employers and communities, and the impact that delegation of funding is having on the outcomes achieved by learners in both adult community learning and adult skills provision. This would build the evidence base on the impact of devolution of adult education and skills provision, boosting the arguments that the Mayor has made in “Skills for Londoners: A Call for Action” for further devolution of skills and employment services.

2.4. The survey will significantly enhance our understanding of the impact of learning of Londoners, providing far richer and more timely information across a wider range of outcome areas than current existing DfE datasets. Data collected through the ILR is partial, with GLA analysis demonstrating that no destination data was collected for 43% of learners completing learning in 2017/18. While Longitudinal Education Outcomes (LEO) data is much more accurate and complete, there is currently a significant time-lag, with at least a two-year gap between course completion and the data being made available. The LEO data however does not currently collect any information on measuring the social impact of training and education on a London wide basis.

2.5. The London Learner survey would give London greater control and ownership over data on the outcomes achieved by learners – there are currently significant restrictions on use and publication of the ILR data, which is owned by the DfE.

2.6. A number of AEB providers currently run their own local destinations surveys, which vary substantially across the provider base and can prove to be quite expensive. It is envisaged that running a Learner Survey at a London-wide scale will be a more efficient way of collecting large amounts of learner destination data than the existing current arrangements.

2.7. It is recommended that a pilot of the survey is carried out in the 2020/21 academic year, in order to test the operational model for the survey ahead of any decision to scale up the survey to the full AEB provider base in 2021/22. It is therefore recommended that the Mayor authorises expenditure of up to £160,000 funded from the AEB Management & Administration budget for the pilot survey.

2.8. Based on the results of the pilot survey, in summer 2021 the AEB Mayoral Board will be asked to consider whether to progress to full scale delivery of the survey in the 2021/22 Academic Year. Subject to the Board’s endorsement, a subsequent MD will be sought to approve the expenditure and continuation of the survey. For commercial and practical reasons (including the need to ensure continuity of delivery) it is proposed that a single provider is procured now that is able to deliver both the pilot survey in 2020/21, and full-scale survey delivery in 2021/22, 2022/23 and 2023/24.

2.9. It is therefore recommended that the scope of the forthcoming procurement exercise for the survey includes a contract for the pilot year with an option for the GLA to extend to cover the first three years of full-scale delivery. The procurement documentation will stipulate clearly that the commencement and continuation of full delivery of the survey in 2020/21 and beyond is contingent on the option(s) mentioned being exercised (should further funding be authorised by the Mayor for this purpose on an annual basis).

2.10. The feasibility study for the survey currently estimates that the full-scale delivery of the survey over the three years covered by the option above could cost up to an additional £375,000 per Academic Year. There are, however, some untested assumptions underpinning the cost estimates in the feasibility study report, which could increase or reduce the investment required. The most significant untested assumption concerns the initial survey response rate, which will be the principal driver of the number of telephone interviews required, and consequently, the cost of delivering the survey. These assumptions will be tested during the pilot stage of the survey.

3.1. In the Skills for Londoners Strategy and subsequent analyses, the GLA has highlighted that there are several groups that are disproportionately underrepresented in London’s labour market, including some groups with protected characteristics. This includes special educational needs and disabilities (SEND) learners, young people, disabled adults, Black, Asian and minority ethnic (BAME) groups and women.

3.2. The GLA will use any data collected through the London Learner Survey to measure and analyse trends in the participation of different groups of Londoners and the outcomes achieved by different groups of Londoners. This will give the GLA insight into how effectively providers are serving those groups, and will be used to inform future policy interventions to reduce gaps in participation and achievement among disadvantaged groups. Officers have considered the direct impact of the proposed work on groups with protected characteristics and do not anticipate that the collection of survey data, in and of itself, will disadvantage any particular group.

4.1. The proposed work is part of the implementation of the commitments made in the Mayor’s Skills for Londoners Strategy, to:

• empower all Londoners to access the education and skills to participate in society and progress in education and work;

• meet the needs of London’s economy and employers, now and in the future; and

• deliver a strategic city-wide technical skills and adult education offer.

4.2. It is intended that a pilot survey be run in the 2020/21 academic year in order to identify and mitigate any survey design complexities, or operational challenges, ahead of decision to scale up to the full provider base. There is a risk that carrying out the pilot in the next academic year – which due to the COVID-19 pandemic is likely to give rise to atypical results – may impact on its relevance to and return to “business as usual” in the following academic years. This risk will be mitigated through adaptation of any survey as the situation develops to account for changes in delivery.

4.3. During the development work undertaken by NatCen, the baseline survey has emerged as the most difficult and risky part of the London Learner Survey. Providers in the Outcomes for Londoners Advisory Group felt that the baseline survey was deliverable, but would be logistically challenging. Without a successful baseline survey phase, there is a risk that the survey will not be able to achieve a representative sample of learners and therefore its analytical power will be compromised. To mitigate this risk, work strand 2 has been developed to ensure all steps have been taken, working collaboratively with the sector, to ensure the success of the baseline survey. This work strand is also examining the timeliness of ILR submissions from providers, which is necessary for the successful delivery of the survey.

4.4. There are procurement and contracting risks associated with the complex nature of the service the GLA will be purchasing, for example the GLA may need to make changes to survey design and administration as a result of the process stress testing and pilot. It will be necessary to ensure that the survey service procured is sufficiently flexible to adapt to any required changes. This will add complexity to the development of the service specification. The GLA are working closely with colleagues in TfL Procurement to mitigate these risks.

4.5. There remains a residual risk that the response rate to the survey will be too low to generate meaningful representative results, for example as a result of poor quality or limited learner contact details or reluctance of learners to take part in the survey. This will be mitigated through the recommended survey design.

4.6. There is also a risk that Education and Skills Funding Agency (ESFA) will not provide contact details to the GLA and its supplier in a timely fashion. However, discussions with the ESFA have been positive in this regard so far.

4.7. There are no conflicts of interests to declare from those involved in the drafting or clearance of this decision.

5.1. Approval is being sought for expenditure of up to £160,000 for the development and delivery of a Pilot London Learner Survey to be carried out with providers during 2020/21 academic year.

5.2. This expenditure will be funded and contained within the 2020/21 Adult Education Budget (AEB) within the Communities and Skills Directorate.

5.3. Should the Pilot survey scheme converted into a full programme scheme this will be subject to approval via a future decision.

6.1. Section 39A of the Greater London Authority Act 1999 permits the delegation of ministerial functions to the Mayor, subject to certain limitations and conditions. This forms the basis of the delegation to the Mayor of AEB functions from the Secretary of State for Education. A particular limitation of the delegation is that the usual power of delegation by the Mayor is not available in respect of s39A delegated functions.

6.2. In taking the decisions requested, the Mayor must have due regard to the Public Sector Equality Duty - namely the need to eliminate discrimination, harassment, victimisation and any other conduct prohibited by the Equality Act 2010, and to advance equality of opportunity and foster good relations between persons who share a relevant protected characteristic (race, disability, sex, age, sexual orientation, religion or belief, pregnancy and maternity and gender reassignment) and persons who do not share it (section 149 of the Equality Act 2010). To this end, the Mayor should have particular regard to section 3 (above) of this report.

6.3 Should the Mayor make the decision sought officers must ensure that:

6.3.1 the services and/or supplies required are procured by Transport for London Procurement who will determine the detail of the procurement strategy to be adopted in accordance with the GLA’s Contracts and Funding Code; and

6.3.2 appropriate contract documentation is put in place and executed by the successful bidder(s) for such services and/supplies and the GLA before the commencement of provision of the same.