Learnings from Impact Evaluation of an Ed-tech Learning Application

Learnings from Impact Evaluation of an Ed-tech Learning Application

Background

With rapid technological change and increasing penetration of smartphones in India, education technology (EdTech) has demonstrated significant potential for increasing learning outcomes for students globally and in India.

When it comes to adoption, schools have however, always considered educational apps or digital learning as a supplementary tool and may have had difficulty in mainstreaming it, mostly due to not having fully understood its efficacy. Moreover, the digital divide and inequalities in India highlight that most students need products that offer vernacular mediums of instruction, use different examples and references, target different learning and infrastructure gaps, and are sold at affordable price points.

With the context of the same, it becomes extremely essential to evaluate the technology enabled learning tools prevalent in the market and understand the efficacy challenges and implementation gaps for decision making and curriculum development.

Key Insights

To address this, Akshara Foundation partnered with Sattva Consulting to conduct a pre-and-post analysis for the cohort that was introduced to Akshara Foundation’s Building Blocks application and was conducted in urban and rural Bengaluru in Karnataka and in Bhubaneswar, Odisha covering 13 schools (both private and government aided), 1119 students and 479 parents. This study was conducted between Aug’19 and Feb’20.

Sattva looked at the study from three lenses – ‘activation’ which are the factors influencing the download of the application, ‘usage’ which are the factors leading to its consistent usage and ‘outcomes’ which are the critical factors influencing the learning of the end-user.

Some of the key insights from the study are highlighted below:

ACTIVATION

– Building Blocks application was downloaded by 67% of the students who were part of the study and 49% of them continued using the application till the time of the post-test.

– One of the key factors that the parents quoted as an influence for downloading Building Blocks was the fact that the application was free of cost, had the ability to operate in the offline mode and was available in 9 regional languages.

USAGE

– The trend of application usage was dependent on engagement activities planned by Akshara Foundation.

– Students found the Building Blocks application visually appealing and enjoyed solving problems and scoring stars while doing so. The application is based on competencies that match the school curriculum (Building Blocks as a tool conforms to the National Curriculum Framework 2005) and so becomes an ideal practice tool in the home environment, as highlighted by the mathematics teachers across schools.

OUTCOMES

– There was an increase in the learning outcomes of the students during the post-test who were not at their grade specific foundational competencies in mathematics during the pre-test.

*Excluding students who were already at the highest level in any competency during the pre-test

– There was an increase in the interest of students in mathematics after using the application. Between the pre-test and the post-test, the number of students who dislike mathematics as a subject had reduced (from a drastic 92% to 25% by the end of the post-test).

– When asked about their overall satisfaction with the application, the average rating given by 354 parents was 4 on a scale of 5. They had also started to recommend the Building Blocks application to their friends, neighbors and relatives.

Way Forward

– Sattva suggested a 4-step approach to design outreach and marketing efforts based on the target group including policy advocacy, building incentives for schools and teachers to get the students to use the application, demand creation for the product amongst the parents by establishing credibility of the application and linking learning outcomes of the application to school curriculum.

– Plan and execute concurrent engagement and monitoring activities for user support and nudges to drive engagement with parents and students.

– Launch periodic version updates of the Building Blocks application, strengthen the content of the application by adding more questions and increasing the difficulty levels, and enhancing adaptive nature to suit the user ability.

The full report can be accessed below.

————————————-

Sattva has been working with various corporate clients, foundations and social organisations to help them define the role and importance of effective Assessments. Our focus is to solve critical problems and find scalable solutions. We assist organisations in formulating their long-term social impact strategy by strategically aligning with business to provide meaningful solutions to social issues.

We’d love to hear your thoughts and feedback on this topic. Do write to us: impact@sattva.co.in

Technology-based models towards the improvement of Spoken English Skills

Technology-based models towards the improvement of Spoken English Skills

The time for EdTech is here.

Join Sattva and Michael & Susan Dell Foundation for a discussion on ways of scaling up EdTech in India for social good.

We will also present findings from our study on – Evaluating Effectiveness of Technology in Improving Spoken English: https://bit.ly/2Yrik9B

This two year, large-scale pilot assessment was conducted covering ~ 14,000 students across 9 states in India and the sample set included 18-22 year old students from the urban poor segment, in their pre-final or final year of study.

Date: Tuesday 23 June 2020
Time: 3 – 4:30 pm
Register: https://bit.ly/2YmkMhq

We hope to see you there!

Evaluating Effectiveness of Technology in Improving Spoken English

Evaluating Effectiveness of Technology in Improving Spoken English

Background

In India, out of 15 million who are employable each year, 75% aren’t job ready.

In the last decade, the importance of English has improved with an increase in the number of jobs that require fluency in spoken English. In a 2012 survey by the Economist Intelligence Unit, 70 % of executives said that their workforce will need to master English to realise corporate expansion plans, and a quarter said that more than 50 per cent of their total workforce would need English ability. Yet, only 4% men and 2% women in wage employment in India report speaking fluently in English.

Key Insights

To address this, the Michael & Susan Dell Foundation partnered with Sattva to evaluate the effectiveness of ed-tech leveraged models to improve spoken English across 14,000 students. This intervention ran from 2017 to 2019. The following were the key learnings from the ground:

1. Students who were trained showed a 2.1x improvement in spoken English over students who weren’t.
2. While pure online learning worked well for advanced students, blended models with offline content was most effective for beginner students.
3. ALL types of students improved, but beginners showed 6X improvement over advanced level students
4. Background factors like family income and parents’ education influenced starting levels but did NOT affect learning patterns and improvement.
5. Students who signed up on their own, voluntarily improved 36% more than students who were mandated by their colleges and schools
6. Specific mobile application features such as leaderboards can increase effectiveness and adoption among students.
7. Students with better English proficiencies earned 23% higher salaries
8. Factors such as semester of intervention, college support, type of cities were critical to the success of the intervention
9. Fully on-line models had the lowest cost of delivery and were most suitable for scale

What does this mean?

Ed-tech is an effective, affordable and scalable English-language learning tool that can improve employability for low-income, aspirational Indian youth at scale. The results of this study gains greater relevance in the light of the recent COVID-19 pandemic. Given the economic impact of the crisis, there will be a stronger need for students to improve their chances of employability and their readiness to the market. At the same time, the continued risk of the pandemic and the emerging reality of social distancing would mean the role of technology in education will continue to grow.
Hence, we hope our insights provide relevant answers when such technology solutions gain increased attention and adoption among colleges, skill development institutions and other social impact programmes.

Most schools around the world have been temporarily closed. These nationwide closures are impacting over 90% of the world’s student population, with 320 million children of 1.4 million schools impacted in India of which 70% of the schools are run by government bodies.

There is a huge need for customised, thoughtful and scalable programmes design to ensure learning continuity. The time for Ed-tech has arrived.

The full report can be accessed below.

————————————-

Sattva has been working with various corporate clients and social organisations to help them define their social impact goals. Our focus is to solve critical problems and find scalable solutions. We assist organisations in formulating their long-term social impact strategy by strategically aligning with business to provide meaningful solutions to social issues.

We’d love to hear your thoughts and feedback on this topic. Do write to us: impact@sattva.co.in

Enabling Equity in Classrooms in India

Enabling Equity in Classrooms in India

– By Farhan Shaikh

Equity as a construct propagates the core value of fairness and inclusion with a strong belief that all individuals deserve the available opportunities for development despite differences in background and personal abilities. Unlike the notion of ‘equality’ in education, where treatment of every child is expected to be the same before the learning process, ‘equity’ promotes redistribution of resources and teaching support for collective development within the classroom. Given the enormous diversity within a country like India, there have been remarkable initiatives like the Right to Education Act of 2009 and flagship schemes like Sarva Shiksha Abhiyan and mid-day meal which emphasises on education for all but does not necessarily solve for equity.

The latest Children in India Report by the Ministry of Statistics and Programme Implementation, Government of India, reveals alarming statistics on the dropout rates of girls and students belonging to other socially disadvantaged groups. There is a 30% reduction in enrolment of girls from grade 5 to grade 9.For public schools in rural and semi-urban areas, enrolment up to grade 8 remains high mainly due to the mid-day meal scheme and other government incentives for parents to send their children to school. With its high tribal population, Jharkhand has the highest dropout rate of close to 70% for school children.

You can read the full article, here.

—————–
Farhan Shaikh works with the Program Advisory and Management team at Sattva that largely engages with large scale non-profit Foundations. His work so far has primarily focused on organisational development of non-profits, data driven research studies and strategic philanthropy. Farhan has been associated with the Education circle of Sattva to develop content that can provide key actionable insights on specific problem areas. He completed his Bachelors in Statistics and followed it up with a Young India Fellowship at Ashoka University (Post graduate diploma in Liberal Arts).

Sattva has been working with various non-profits and social organisations as well as corporate clients to help them define their social impact goals. Our focus is to solve critical problems and find scalable solutions. We assist organisations in formulating their long-term social impact strategy by strategically aligning with business to provide meaningful solutions to social issues.

We’d love to hear your thoughts and feedback on this topic. Do write to us: impact@sattva.co.in

ECCE CSR Landscape in India and it’s Potential for Impact

ECCE CSR Landscape in India and it’s Potential for Impact

Background

Early Childhood Care and Education (ECCE), encompassing the inseparable elements of care, health, nutrition, play and early learning within a protective and enabling environment has long been underfunded by CSR programmes, owing to a lack of awareness on its importance in child development. The Draft National Education Policy (NEP) 2019, which defines the early learning needs in the age group 0 to 3, and the age group 3 to 8 as a single learning continuum called the “foundational phase”, has added to this with a lack of clarity on the modality of achieving the infrastructural and institutional changes required by the policy.

‘ECCE CSR Landscape in India and Potential for Impact’ is a study by Sattva and DHFL Changing Lives Foundation aimed at raising awareness on Early Childhood Care and Education (ECCE) in India and developing a guide to CSR funders to consider ECCE in their portfolio.

The study plots the current landscape of funding and solutions for ECCE enabled by CSR, explores trends and evolution of CSR in ECCE funding over the last three years and maps the solution landscape of ECCE interventions enabled by CSR funding to plot areas of interest, types of funding, gaps and challenges.

Key Findings

ECCE Needs and Trends in India:
1. Nutrition, health and early childhood education are deeply interlinked. A child’s development potential cannot be fully realised unless these interlinkages are incorporated in intervention design.
2. About one fourth of children in the age group 3 to 6 do not attend any form of pre-school in India. Amongst those who attend some form of preschool, almost 50% are not ready for formal schooling.

Policy Landscape:
1. The Draft National Education Policy (NEP) 2019 defines the early learning needs in the age group 0 to 3; and the age group 3 to 8 as a single learning continuum called the “foundational phase”. However, there is a lack of clarity on the modality of achieving the infrastructural and institutional changes required by the policy.
2. The Indian government spends about 0.3% of GDP on ECCE, which is much lesser than the OECD countries’ average of 0.8%.

Interventions by ECCE implementers:
1. ECCE implementers have been instrumental in executing innovative ECCE interventions through contextual approaches on the ground. However, these innovations remain largely localised, with very few translating to systemic change.
2. The implementer landscape has certain white spaces like early stimulation, responsive care, parental capacity building, children with disabilities. There is also a felt need by implementers to increase the focus on the 0 to 3 age group.

CSR funding for ECCE:
1. Despite education and health being top funded areas for CSR funders, only 17% of top education funders and 22% of top healthcare funders make some contribution to interventions related to ECCE.
2. There is little data available on CSR expenditure towards ECCE due to lack of standardised reporting practices. However, aligning schedule VII of the Companies Act to SDGs has the potential to change this
3. Interventions pertaining to health and nutrition are better represented than other components of ECCE in CSR funding.

Opportunities to unlock capital and promote collaboration:
1. Investing in ECCE has far reaching impacts ranging from improved economic growth, creating responsible citizenry, to low crime rates.
2. To enhance their ECCE impact, CSR funders can facilitate collaboration at three levels:

  • a. Collaborate with government authorities/institutions to complement the efforts
  • b. Collaborate with multiple non-profits towards comprehensive ECCE outcomes
  • c. Collaborate with other funders working on addressing ECCE or non-ECCE outcomes

 

The full report can be accessed below.

ECCE CSR Landscape in India and Potential for Impact – Full report

A factsheet for the report can be accessed below.

ECCE CSR Landscape in India and Potential for Impact – Factsheet

This is a first attempt at mapping the landscape of funding and solutions in ECCE in India. We deeply appreciate your feedback, comments, and suggestions. Write in to research.advisory@sattva.co.in.

EdTech – One Size Does Not Fit All

The Sattva View – One Size Does Not Fit All

In this column, Sulagna Datta lists how ed-tech in under-resourced communities differs from typical market-based products, and argues for thoughtful design and customisation of those products.

5 things to keep in mind while implementing ed-tech projects in the impact sector

Education Technology or Ed-tech is a buzz word in the Indian impact space today. There has been a flux of funding into this sector, with behemoths like Byju’s emerging as unicorns, crossing $1bn in estimated worth. The supply side is inundated with products that can be categorised in a multitude of ways: Subjects, target age group, in school/out of school, etc. As per Tracxn research, which is India’s leading data aggregation and analytics platform – there are ~4574 Ed-tech products in India today, and ~17,000 products globally.

Schools and colleges across the country are using these products for a multitude of reasons ranging from improving scores in specific subjects and preparing for competitive exams to practising for job interviews. However, if you work with implementation of Education technology programmes for the bottom of the pyramid, the question to ask is ‘What are the things that no one told you about Edtech projects for this context?’

1/ The number of Ed-tech products in the market that are actually built keeping the bottom of the pyramid in mind is shockingly low.
From a Sattva research, out of 566 school products catering to Hindi and Mathematics, only 19% had either already partnered with or shown interest in working with government schools. Which means a staggering 81% of products were meant for the private school context.

When products are built keeping private schools in mind, their data and infrastructure requirements are higher, and more often than not, their content is at levels not graspable by students in government schools. Implementation teams have the onerous task of spending time to customise these products for the BoP context.

2/ Even products that are meant for the BoP context cannot be utilised to their full potential Implementers need to be prepared that basic infrastructure varies drastically across government schools even in peri-urban areas in Bangalore and Delhi.
The biggest advantage of education technology over traditional pedagogical methods is the creation of personalised learning paths for students. Students can learn at their own pace with a curriculum adapted to their needs. For this, the ideal device to student ratio is 1:1, and almost all products are built keeping this ratio in mind. However, this fails in an Indian government school set-up. Even in schools that have labs, the device ratio is seldom 1:1, hampering engagement and consequently learning outcomes.

Most products are designed keeping in mind a certain number of modules to be completed at home as practice. However, most children in government schools come from households with an annual income of <1,00,000 Rs. They don’t own devices at home, and hence are not able to complete most of the self-learning that is meant to happen on the product. Additionally, another constraint in the government school context is access to internet. Since the maintenance budget of all government schools in India is ~8,000-10,000 rupees annually [often going into maintenance of buildings, etc.]- covering bills like the internet becomes cumbersome and is ignored. This leads to further interruption of technology-based learning. 3/ From ages 17-23, ~90% of BoP college youth have smart phones. However, they are extremely data conscious and tend to delete any application/product that takes more than 20 MB of space. While choosing products for the vocational context, practitioners have to be very conscious of the product they recommend.
From a pilot to learn English for interviews through applications through 5 top applications in India, 2 applications stood out in performance owing to the following reasons:

i. They functioned fully offline. After the initial download, they didn’t require any data to run
ii. They were between 15-20 MB in size
iii. They were available across all playstores: Android, Jio, etc.
The other three failed on at least one of the above parameters

4/ From qualitative interviews with about ~2000 college students across India, there is a clear set of features that makes an application more successful than another
i. Leaderboards were a clear favourite amongst students. Students were motivated to use applications when they could see their peers use it. They liked to see where they stood in their comparable cohort
ii. Applications that had short modules and progress bars/gamification were favoured. Students used it like a game to finish the stipulated target defined by the product for the day
iii. For a pan India context, the application that was most successful had an 18 language interface. Students preferred to learn in their vernacular language.

5/ The optimum learning time on an edtech product is about 20 minutes a day
A critical element to keep in mind while designing an edtech initiative is to set a target for content consumption a day. Applications that stipulated more than half an hour a day, saw declining engagement and drop-outs. A 15-20 minute engagement/day was seen in about 80% students who completed the entire course.

While private enterprise products are pushed to the BoP context without considering its nuances, the learning experience is less than optimum, and that typically discourages the learner, further jeopardizing the quality of education. It’s important to address this demographic thoughtfully, with an eye on specific needs and access.

——————————————————————-
This article was originally published in Impact Magazine.

Sattva has been working with various nonprofits and social organisations as well as corporate clients to help them define their social impact goals. Our focus is to solve critical problems and find scalable solutions. We assist organisations in formulating their long-term social impact strategy by strategically aligning with business to provide meaningful solutions to social issues.

● Talk to us: impact@sattva.co.in

Connecting to the Idea of Impact – a report from the field

Connecting to the Idea of Impact – a report from the field

Hugh Lupson is from London and studies History and Geography at the University of Leeds in the UK. His recent university projects whetted his appetite for the social sector and he spent some time as an intern at Sattva. This was his first time in India.

Akshaya is currently pursuing M.Sc. (Hons) Economics at BITS Pilani. She has actively volunteered in the programmes of ‘Education’ and ‘Rural Women Empowerment’ undertaken by the Nirmaan Organization at BITS. Her inclination for community work led her to intern at Sattva where she hopes to learn more about how organisations specialise in social service and gain insight into social entrepreneurship.

As part of their internship Hugh and Akshaya visited a school for marginalised communities in Bangalore. Read about their experience here:

This Foundation’s vision is simple: to provide the poorest children from local slums with world-class opportunities, the key is education. However, this Foundation differs from other projects in providing what they call a ‘360-degree development model’, a more holistic approach to education. Besides lessons, the ‘360-degree development model’ focuses on healthcare, nutrition, emotional support and community development.

We visited them on 27th September to observe the model in action. We wanted to experience the influence that Sattva’s programmes have on their beneficiaries. The exploration into the lives of beneficiaries would also help us connect to the idea of impact and visualise it first-hand. The insights gained from this visit could even allow us to perceive the ways in which our new product, Shift 2.0 could give an enhanced picture of impact to all programmes undertaken by Sattva. We spent only a couple of hours at the Foundation, a 4-floor building with a multipurpose terrace. So while our analysis may be far from comprehensive, the visit gave us a valuable opportunity for a qualitative appraisal, shining a light in a way that statistics simply cannot and adding a human element to project evaluation.

Education is the primary pillar of the Foundation’s approach. Their school follows the I.C.S.E, an intense yet balanced secondary-schooling curriculum. We observed several lessons including English and Mathematics. In the Mathematics class the children were using blocks representing groups of ten to form number bonds to 100. Their numerical ability was impressive. Adapting to different styles of question, the children showed an understanding of the relationships between different numbers and functions rather than simply rote learning of the bonds. This speaks highly of the teaching style here. Unfortunately – as the school’s principal mentioned – the quality of the teachers here attracts the attention of fee-paying schools, who are able to lure some of them with higher wages each year.

In the English classes the students struggled slightly when not following memorised sentences. Nevertheless, they articulated to us their impressive ambitions and dreams; from becoming doctors and English teachers, to travelling the world. Through ideas like naming classrooms after planets and asteroids, it seemed to us that the Foundation’s ethos was to encourage the children not to put limits on themselves or the ways they think.

Their focus on emotional development was also clear to see. The happiness of the children is perhaps our most lasting impression of the visit. We were met in each classroom by beaming young faces, excited to speak to us and clearly proud of what they were learning. The school has a ‘friendship corner’ for any child who is feeling unhappy. The pupils are encouraged to sit in the ‘friendship corner’ whenever they are feeling unhappy and other pupils will join them to cheer them up. While we didn’t see this initiative in action, it suggests that developing empathy in children was important to the school.

Unfortunately, according to the school’s principal, the children’s happiness doesn’t always follow them home each day. Many children return home to difficult lives and carry a sizable emotional burden due to past or ongoing traumatic experiences. In response to this, the school has an in-house therapist who will see pupils on demand. However, acknowledging that a therapist will not be able to tackle this issue at its root, the school also invests in efforts to make sure children are happier at home. The community development programme aims to forge a stronger community for children through collaboration with other local schools, for example discussing a book the children had recently read via Skype. An initiative for fathers suffering with alcohol problems was also mentioned as well as teaching parents how to make soap using vegetable peel.

The children also face challenges when they graduate from the school. The strong community spirit at the Foundation’s schools contrasts with normal life as a young adult. We heard that not all graduates have been able to make the necessary emotional adjustments. One solution to this issue has been to extend the school’s structured mentoring system to include alumni. Access to this wider network of the Foundation’s alumni serves as a useful tool for pupils striving to achieve their career goals.

Some areas of the 360-degree model were harder to gain an appreciation of during our visit. We narrowly missed the children’s lunch, which they had clearly been eagerly anticipating. Therefore we had little opportunity to observe the school’s nutrition programme. However, the children spoke about their food with enthusiasm, especially the eggs they get twice weekly. At the risk of making an inference, it would be hard to imagine hungry children being as happy and animated as the ones we met.

Healthcare and extra-curricular activities were also difficult to gain an understanding of during our visit. While some older children had an inter school arts competition, there was a noticeable lack of outdoor space for the children to play sports and little mention was made of activities outside of lessons. With regards to healthcare, we were given only a brief look at the infirmary, which two children were using to revise for a test. The teachers didn’t mention the healthcare programme. However, we noticed that the children’s ID cards were lacking basic details such as their blood group. As such, for the next visit: nutrition, extra-curricular activities and healthcare should be prioritised for a deeper understanding of the Foundation and its impact.

In its 16th year, the Foundation and its pupils seem to be thriving. The school has received several awards for innovation from institutions including the British Council and Tata Communications. Going forward, the principal mentioned that a key objective for them will be to secure a more reliable funding system. Currently, with funding only being guaranteed for one year at a time, it is difficult for the school to plan for future growth. Perhaps with a 5-year funding guarantee, the Foundation could scale-up and reach its true potential.
————————————————————-

Sattva has been working with various nonprofits and social organisations as well as corporate clients to help them define their social impact goals. Our focus is to solve critical problems and find scalable solutions. We assist organisations in formulating their long-term social impact strategy by strategically aligning with business to provide meaningful solutions to social issues.

● Talk to us: impact@sattva.co.in

Parvathy Ramanathan

Parvathy leads Sattva’s technology CSR Programme Management platform – SHIFT.

Parvathy has worked extensively in both the US and India at the intersection of systemic transformation and technology, in sectors including Government, Healthcare and Education.

Parvathy is an entrepreneurial leader, now focused on solving urgent problems in the development sector, leveraging over 18 years of global experiences in strategy, marketing, product innovation, services delivery and business development. She has launched, generated and managed global revenue streams across products and services. Her leadership roles range from Accel-Partners funded Big Data Analytics start- up to Fortune-500 firms like Amazon, IQVIA and McGraw-Hill. Her experience spans sectors including Government, Healthcare, Financial Services, Education and Retail.

Parvathy has an MBA from the Kellogg School of Management and a Bachelors in Engineering from RAIT, Mumbai University.

Rahul Shah

Rahul leads Non-Profit Advisory at Sattva, leveraging his deep experience working on organisational development with social organisations across domain areas, size, scale, challenges and aspirations.

His diverse experience in the development sector has evolved from his time working at the grassroots level in Ahmedabad, India, to community organising in his hometown of Washington, DC, consulting with social organisations across domains and managing multi-year development projects. Prior to joining Sattva, Rahul worked with TechnoServe India where he managed a CSR funded accelerator programme for women-led social enterprises and NGOs, and a USAID funded project transferring frugal agricultural innovations from India to Africa. In addition to his development sector work, he has five years of progressive experience in corporate finance with industry leading, Fortune 500 corporations in the United States.

Rahul has an MBA and an MS Finance from the University of Maryland’s Smith School of Business and an Executive Certificate in Non-Profit Management from Georgetown University.

Chaitanya Pathak

Chaitanya is a part of the Consulting Services (Implementation) team in Mumbai, working on design and implementation of CSR projects.

Before Sattva he has worked in the IT services industry with a business intelligence firm. He went on to a Gandhi Fellowship with Kaivalya Education Foundation, working with government systems and stakeholders in rural Rajasthan. Through all these years he continued to volunteer at Make A Difference (MAD), an organisation that works in improving life outcomes for underprivileged children. He is committed to using his knowledge and skills to make lives better, while being a catalyst for social change.

Chaitanya is a Gandhi Fellow and has a B.E. in Computer Technology from Yeshwantrao Chavan College of Engineering, Nagpur.