Adam McMaster joined The Brilliant Club in October 2022 for a three-month internship as part of his doctoral training programme. Adam is in the third year of his PhD at the Open University and his research focuses on black holes and variable stars. Adam brought to The Brilliant Club specialist data science skills, and working with our Research and Impact team, he created an app to improve how we analyse qualitative data. In this blog, we share the insights from Adam’s project and how it has impacted the work we do at The Brilliant Club.
The Brilliant Club runs the UK’s largest university access programme, The Scholars Programme. In the academic year 2021-22, we supported over 22,000 students through our programmes; which, for our Research and Impact team, means a lot of data to process and analyse! Alongside reporting on the numbers, an integral part of our evaluation work is understanding how pupils and PhD tutors experience the programme – what do they want to tell us about the programme? Collecting qualitative feedback via surveys is one mechanism that we use to listen to the voices of the young people who take part in our programmes. But what do we do with the data, once we have it?
Until now, working with the qualitative responses to our surveys meant having to read every response. While reading the responses, the usual requirement is to code each response. That means labelling or categorising the responses in some way, to organise them and make it easier to refer back to subsets of responses later. Combined, our surveys receive thousands of responses per school term, so reading and coding everything is a lot of manual work. To make this process easier, we built an app that allows Brilliant Club staff to interactively browse and search the written answers to our surveys. It’s called Text Explorer.
In the next sections, we’ll detail the value of Text Explorer, features and capabilities in Dataiku that made the process easier and faster, and reasons other organisations might want to build their own version of Text Explorer.
Text Explorer is built in Dataiku, the platform for Everyday AI, which allows us to build data processing workflows that carry out data cleaning and analysis. It even includes plugins that can do a number of sophisticated things, such as natural language processing (NLP). Integrating these built-in features with Adam’s own Python code was simple, and as a result we were able to put together a workflow that takes a set of surveys as input and produces a set of outputs that tell us about the answers to those surveys.
Text Explorer has many useful features (click here for a diagram):
Thanks to Dataiku’s Text Explorer, it’s now possible to search, code, and export written responses from surveys far more easily than it was before. Not only has Text Explorer increased efficiency (i.e., saved hours of time processing qualitative data manually) but, even more importantly, it has enhanced how we use the data. For example, we can readily compare responses between different surveys, extrapolate different sentiments, and identify meta-themes based on feedback from pupils and PhD tutors.
The Brilliant Club’s Impact and Analysis Officer, Dr Meirin Evans said about the tool: “I was able to compare the positivity of survey responses across two of our programmes, which wouldn’t have been possible before Text Explorer“.
Adam McMaster said about his experiences of the internship: “I’m pleased I was able to create something that The Brilliant Club will find useful after my time here, while in the process I got to experience working on something very different to the data I usually use”.
Emilie Stojanowski, CSR & Ikig.AI Manager at Dataiku said “We are so glad to see how this partnership with The Brilliant Club embraces the potential of Everyday AI via concrete applications of how a data project can accelerate a non-profit’s mission in its day-to-day operations. I’m so glad to see how the team has been able to run this project successfully and can’t wait to see what’s next. I also hope it will inspire other non-profits to join the Ikig.AI program.”
And finally, from a Brilliant Club perspective “we hope that this blog shows that, with the right tools, qualitative data can be systematically analysed and used to understand the impact of large-scale programmes, such as The Scholars Programme. And that most importantly, qualitative analysis, just like quantitative analysis, needs investment – it takes time and needs the right tools and people behind it.”
For any questions about the blog, please contact meirin.evans@thebrilliantclub.org.
You can find out read more about The Brilliant Club’s evaluation work here and our programmes work here.
Our recent blog on attainment raising in schools can be found here.
This article was first published on Dataiku’s blog.