Can mobile applications improve children’s learning outcomes? What data do we need to collect when evaluating an application? These were just some of the questions we hoped to answer when we launched our first mobile application “Karim & Jana” (K&J).
K&J was developed by QRF to improve numeracy skills among children between the ages of 3-5 years across the Arab world[1]. The first version of the application was launched in May 2017 and it has reached more than 250 thousand downloads to date.
To evaluate the application, a randomized control trial (RCT) was conducted in June 2017. The RCT ran for 6 months until the end of 2017, and included 360 children who were randomly assigned to either one of three treatment groups or the control group. Each treatment group received a different early numeracy application[2]. To evaluate the outcomes of the study, the International Development and Early Learning Assessment (IDELA)[3] was used. The tool consists of two assessments: a child assessment and a parent survey. The IDELA assessment was administered twice, once at baseline and once three months later at end-line after exposure to the treatment.
Karim and Jana's mobile application was found to improve children’s overall IDELA scores by 5 percentage points. Specifically, numeracy skills improved by 7 percentage points, and motor skills improved by 9 percentage points.
The findings emphasized the importance of parents’ perception toward education, and the availability of educational material at home. Children of parents who completed higher education certificates were found to score above average. We learned from the IDELA parents’ survey that parents were aware of the importance of early childhood education, but the financial costs of enrolling their kids were the main barrier for them to enroll their children in a kindergarten.
Our evaluation brought to light valuable lessons that we will use for future iterations of such studies. Specifically, for this study, we didn’t have a significant sample size. The sample size limited our ability to attribute the detected impact to the usage of the Karim and Jana. A larger sample size would help us in distinguishing whether the changes in children’s skills can be attributed to using the application or if the improvement observed was caused by the natural progression of children due to early years’ development.
For future studies, we’ve learned it would be more optimal to gather data on variables such as parents’ perception towards applications and children’s engagement while using the application, including usage data.
Additionally, conducting focus groups with parents from different governorates would provide more insight on parental perception of mobile applications in general, including the perceived potential impact of the application on children’s school readiness. We believe that this additional information will allow us to better cater the application to the needs of mothers and children.
We’ve also realized the importance of conducting extensive application testing with the target audience prior to launch. Observing the children while they play can help direct the application design and development process.
After launching the application, usage data including the number of downloads, retention rate, and capturing the most engaging components should be collected to inform a more rigorous analysis and evaluation of children mobile applications.
The launch of K&J is a strong step towards creating accessible mobile applications with the aim of improving children’s learning outcomes. Lessons learned from this trial will further inform future application development and program evaluations for QRF and beyond.
[1] Details on the content of the application can be found in this blog post
[2] We studied the impact of three mobile application, Karim and Jana, Lamsa, and Little thinking mind. The three applications aim to improve children numeracy skills through games and activities.
[3]IDELA was developed by Save the Children. For more info: https://idela-network.org/