By Anna Bergstrom and Bogdan Yamkovenko
A randomized controlled trial (RCT) run by University of Toronto researcher Phil Oreopolous in partnership with Khan Academy showed positive learning gains for students whose teachers devoted time to practice on Khan Academy.
Results from randomized control trial
Results from a randomized controlled trial (RCT) run by University of Toronto researcher Dr. Phil Oreopolous in partnership with Khan Academy showed positive learning gains for students whose teachers devoted time to practice on Khan Academy. Students who demonstrated learning gains were elementary school students in grades 3-6 whose teachers instructed them to use Khan Academy. These students’ scores were .12 to .17 standard deviations higher on the end-of-year mathematics assessment after using Khan Academy for ~35 minutes per week compared to students of the teachers who were in the control group.
The study also included students in grades 7-8; however, these students did not experience the same results as students in grades 3-6. According to researchers, the varying results are likely due to different average usage by grade: whereas elementary school students in grades 3-6 used Khan Academy an average of 35 minutes per week, middle school students in grades 7-8 used Khan Academy an average of 10 minutes per week.
Higher usage in elementary school was likely due to longer class periods (70m), the fact that students were taught by the same teacher during the day, and that they had additional enrichment time during the week to allocate to Khan Academy. This reinforces the idea, demonstrated in other efficacy results, that the optimal benefit of Khan Academy is achieved with regular and ample time spent on the platform (30+ minutes per week over the course of the school year).
Why the RCT results are impactful
Studies that use RCTs to investigate educational interventions rarely yield compelling results: a review of 77 RCT studies conducted between 2002-2015 from the Coalition for Evidence-Based Policy found that over 90% of RCTs showed null or marginal effects. Unlike the majority of RCT studies, the Arlington RCT demonstrated a meaningful, positive effect between students whose teachers instructed them to use Khan Academy and student test results. Furthermore, the Arlington RCT results were observed despite inconsistent teacher engagement and student usage. The fact that these results were generated under less-than-ideal conditions, using the most stringent of evaluation methods, suggests that use of Khan Academy produces measurable, substantive learning outcomes.
Study design
At the end of the 2020-21 school year, the researchers contacted all math teachers for grades 3-8 in Arlington Independent School District (TX) about whether they were interested in using a computer-assisted learning program in their class the following year. Three hundred twelve math teachers expressed interest and were randomized by grade level into treatment and control groups. Teachers in the treatment group received access to professional learning (PL) “Khoaching” in addition to access to Khan Academy Districts, rostered access to Khan Academy content and product features like teacher and district level data and dashboards. Teachers in the control group did not receive Khoaching but received the same access to Khan Academy Districts, rostered access to Khan Academy content and the same product features. Note that control teachers receive delayed access to the treatment one year later. The final sample resulted in 112 treatment teachers and 112 control teachers, with a total of 10,979 students.
The research team developed professional learning (PL) Khoaching with guidance from Khan Academy’s PL team but localized to the district. The research team delivered all of the professional learning to the teachers in the treatment group before the start of the 2021-22 school year. Most teachers (79%) attended the professional learning in person. As part of the program, a total of 33 “khoaches” were hired to support teachers throughout the year. Khoaches were trained to provide technical support and keep teachers accountable toward goals. At the start of the year, khoaches met virtually with teachers each week, but as the school year progressed, the meeting cadence naturally reduced to monthly or less often. Nearly all teachers (90%) attended their first meeting with the khoach, and the majority (77%) scheduled follow-up meetings for ongoing support.
Conclusion
For students to realize the full benefits of Khan Academy, Dr. Oreopoulos notes several success factors for districts to consider, including:
leadership support
the teacher’s engagement with the program
consistent monitoring of student progress
whether usage is considered integral rather than optional
According to Dr. Oreopoulous, teachers should maintain a regular time for practice on the Khan Academy platform and motivate students to remain focused during learning sessions and engaged during practice exercises. By doing so, students can make substantial gains as a result of the time they spend practicing on Khan Academy.
**
Read Dr. Oreopoulous’ working paper with the findings here.
The post University of Toronto Randomized Controlled Trial of 11K Students Demonstrates a Meaningful, Positive Effect of Khan Academy on Student Learning appeared first on Khan Academy Blog.
By Anna Bergstrom and Bogdan Yamkovenko A randomized controlled trial (RCT) run by University of Toronto researcher Phil Oreopolous in partnership with Khan Academy showed positive learning gains for students whose teachers devoted time to practice on Khan Academy. Results from randomized control trial Results from a randomized controlled trial (RCT) run by University of
The post University of Toronto Randomized Controlled Trial of 11K Students Demonstrates a Meaningful, Positive Effect of Khan Academy on Student Learning appeared first on Khan Academy Blog. News Khan Academy Blog
