Learnings from Impact Evaluation of an Ed-tech Learning Application
Background
With rapid technological change and increasing penetration of smartphones in India, education technology (EdTech) has demonstrated significant potential for increasing learning outcomes for students globally and in India.
When it comes to adoption, schools have however, always considered educational apps or digital learning as a supplementary tool and may have had difficulty in mainstreaming it, mostly due to not having fully understood its efficacy. Moreover, the digital divide and inequalities in India highlight that most students need products that offer vernacular mediums of instruction, use different examples and references, target different learning and infrastructure gaps, and are sold at affordable price points.
With the context of the same, it becomes extremely essential to evaluate the technology enabled learning tools prevalent in the market and understand the efficacy challenges and implementation gaps for decision making and curriculum development.
Key Insights
To address this, Akshara Foundation partnered with Sattva Consulting to conduct a pre-and-post analysis for the cohort that was introduced to Akshara Foundation’s Building Blocks application and was conducted in urban and rural Bengaluru in Karnataka and in Bhubaneswar, Odisha covering 13 schools (both private and government aided), 1119 students and 479 parents. This study was conducted between Aug’19 and Feb’20.
Sattva looked at the study from three lenses – ‘activation’ which are the factors influencing the download of the application, ‘usage’ which are the factors leading to its consistent usage and ‘outcomes’ which are the critical factors influencing the learning of the end-user.
Some of the key insights from the study are highlighted below:
ACTIVATION
– Building Blocks application was downloaded by 67% of the students who were part of the study and 49% of them continued using the application till the time of the post-test.
– One of the key factors that the parents quoted as an influence for downloading Building Blocks was the fact that the application was free of cost, had the ability to operate in the offline mode and was available in 9 regional languages.
USAGE
– The trend of application usage was dependent on engagement activities planned by Akshara Foundation.
– Students found the Building Blocks application visually appealing and enjoyed solving problems and scoring stars while doing so. The application is based on competencies that match the school curriculum (Building Blocks as a tool conforms to the National Curriculum Framework 2005) and so becomes an ideal practice tool in the home environment, as highlighted by the mathematics teachers across schools.
OUTCOMES
– There was an increase in the learning outcomes of the students during the post-test who were not at their grade specific foundational competencies in mathematics during the pre-test.
*Excluding students who were already at the highest level in any competency during the pre-test
– There was an increase in the interest of students in mathematics after using the application. Between the pre-test and the post-test, the number of students who dislike mathematics as a subject had reduced (from a drastic 92% to 25% by the end of the post-test).
– When asked about their overall satisfaction with the application, the average rating given by 354 parents was 4 on a scale of 5. They had also started to recommend the Building Blocks application to their friends, neighbors and relatives.
Way Forward
– Sattva suggested a 4-step approach to design outreach and marketing efforts based on the target group including policy advocacy, building incentives for schools and teachers to get the students to use the application, demand creation for the product amongst the parents by establishing credibility of the application and linking learning outcomes of the application to school curriculum.
– Plan and execute concurrent engagement and monitoring activities for user support and nudges to drive engagement with parents and students.
– Launch periodic version updates of the Building Blocks application, strengthen the content of the application by adding more questions and increasing the difficulty levels, and enhancing adaptive nature to suit the user ability.
The full report can be accessed below.
————————————-
Sattva has been working with various corporate clients, foundations and social organisations to help them define the role and importance of effective Assessments. Our focus is to solve critical problems and find scalable solutions. We assist organisations in formulating their long-term social impact strategy by strategically aligning with business to provide meaningful solutions to social issues.
We’d love to hear your thoughts and feedback on this topic. Do write to us: impact@sattva.co.in
