In this blog post, Lottie Norton from The Brilliant Club’s Research and Impact Department writes about our recent work with DataKind UK and how charities can implement data science approaches to increase their impact.
Charities and social enterprises exist to make a difference, so it’s important to understand what change they actually bring about. Many organisations commit to using evidence and data to better serve individuals and communities, and The Brilliant Club, where I work as the Data Analysis Officer, sees itself as one of them. From my two years at the charity I know that the journey from making this commitment to effectively using data is not an easy one. Especially when it comes to moving beyond analysing if what we do works, to using evaluation in a formative way that helps us improve what we do.
Two challenges stand out for me:
This year we teamed up with the great people at DataKind UK, a collaboration that has helped us to make a big step towards our vision for how we want to use data and evidence at The Brilliant Club.
From learning ‘what works’ to learning how it could work better
Getting to where we are now has been a process. For nearly five years, The Brilliant Club has had a team of evaluation specialists who work to understand the impact of the charity’s programmes on pupil outcomes. Before working with DataKind UK, we had already established that The Scholars Programme, which places PhD researchers in schools to deliver university-style learning to small groups of pupils, has a significant impact on its primary outcome: to support more pupils from underrepresented backgrounds to progress to highly-selective universities. Evidence for this has come from evaluations by UCAS STROBE over four consecutive years. Alongside this, the team has completed a whole array of other evaluation projects and we analyse and share the outcomes of the 13,000 pupils that take part in our programmes each year in bespoke impact reports for every one of our partner schools.
While we had made good progress in evidencing and reporting impact, we had yet to make much headway in using data insights to identify ways to improve what we do. We recognised that data science would be a helpful tool for understanding and planning our work, and the emerging field of educational data mining showed how large data sets could be used to improve educational outcomes. For example, could we identify patterns in the data that explain why some pupils engaged less well on our programmes than others? And if so, could we use this insight to improve outcomes for pupils who were struggling? But our experience in using this approach was limited. Whilst there was a clear desire to use data to improve our programmes we hadn’t yet developed some of the technical expertise needed to delve deeper.
Who you gonna call?
That was the situation when we first spoke to DataKind UK in 2019, about how their team of data scientist volunteers might be able to support The Brilliant Club. Fast forward to the end of March in 2020 and we are in the middle of our DataDive, working with 20 expert data scientists on data sets that usually just myself and a few colleagues get to analyse.
As we were at the start of a nationwide lockdown, the whole event was run online. This did little to dampen the mood and the atmosphere of the weekend was collaborative, intense, fun and productive. In preparation, we were lucky enough to still be able to meet up in person with our Data Ambassadors Katy, Emiliano and Daniel – leading volunteers who helped us to frame our questions and prepare our data so that by the time the DataDive weekend arrived, no time would be wasted with having to clean data or the clarify what we wanted to achieve.
We asked our team of DataKind UK volunteers to support us with two questions: how can we increase the number of pupils submitting their final assignments, and how can we better understand tutor and school availability geographically?
We wanted to better understand final assignment submission because completion of a final assignment is a big deal for pupils on The Scholars Programme. It is an academically challenging piece of work and, for many, it is the first time they have completed an assignment of this length. Pupils who submit a final assignment are invited to attend a graduation event to celebrate their achievement together with parents, schools, and our university partners. Attending a graduation event also gives pupils an opportunity to experience a different university to the one they visited at their launch trip. However, each year, a minority of pupils do not submit their final assignment, meaning they miss out on these important aspects of the programme.
With the help of our DataKind UK volunteers, we now have a better understanding of the factors that influence whether or not a pupil submits their final assignment. We found that older pupils have more difficulties submitting their final assignment. For these pupils, individual characteristics, such as being eligible for pupil premium, were associated with slightly lower submission. For younger pupils, school and tutor factors were associated with lower levels of submission. These include the number of academic terms both the tutor and school had worked with us before.
Our second question, how can we understand tutor and school availability geographically?, addresses a logistical challenge. We are a national charity, delivering The Scholars Programme in all four nations of the UK, working with 500 PhD researchers and over 750 schools each year – with many partner schools in rural areas or social mobility cold spots. Tutor availability and school needs change on a termly basis, which poses a planning challenge on a national and local level.
Thanks to the expertise of our DataKind UK volunteers, who found a multitude of ways to map our data, we now have some exciting new tools that will help us address these key planning questions. This includes a map that highlights areas of provision need – i.e. where we have a lack of tutors or schools – as well a dashboard that allows us to view all schools within 50km of a tutor.
The commitment and enthusiasm from our volunteers was fantastic – they spent their weekends immersing themselves in our data, completing complex analysis and sharing their expertise. The sense of community created by the DataKind UK team was extremely positive and the whole weekend was an inspiring experience.
Reflections and next steps
We came away from the DataDive with two key reflections. Firstly, we have a renewed understanding of the importance of making the time to explore our data fully and to innovate. In preparing for the dive we were forced to organise our data in a way that a data scientist, who had never encountered The Brilliant Club before, could take it and run – producing useful outcomes within a few hours. This has meant we had to think seriously about how we can further improve the data we collect, and how we organise our CRM systems. At the same time, we have committed to working with new software and upskilling staff. Secondly, the experience has really emphasised the importance of collaboration to us. Whilst collaboration is something that The Brilliant Club does on a daily basis – whether it is placing PhD researchers into schools or working with university widening participation departments to deliver trips – working with DataKind UK has been our first experience of working with a large team of data scientists. The expertise and collaborative spirit that were shared whilst working with DataKind UK have shown us that collaboration should play a role in everything we do, and we will seek further opportunities to do so.
For us, the DataDive was really the start of a much wider project. In light of the findings, we are developing materials that will provide tutors and schools with guidance that will help them to support all pupils on the programme complete their challenging final assignment. Meanwhile, armed with our new knowledge of geographic data mapping, we hope to improve our matching of schools and tutors by making more informed decisions, so we can ensure we are able to reach pupils in the most isolated communities. We are also developing a blueprint to do further analysis ourselves. We are looking forward to the next chapter!
If you’d like to talk to The Brilliant Club about their experience of DataKind UK or about our approach to research and evaluation you can contact firstname.lastname@example.org. To learn more about DataKind UK’s different projects, please visit their website.