By Kristen DiCerbo, Khan Academy’s chief learning officer
Many U.S. students struggle with math. It’s a trend that starts early in elementary school, and snowballs over time: as students progress through grade levels, math concepts tend to build on one another from year to year. For students who struggle with math in earlier grades, that means math only gets more challenging with time.
The Decline in Math Proficiency
This idea is echoed in the latest data from The Nation’s Report Card, which shows the number of U.S. students who are proficient in math drops as students move through grade levels: According to the Nation’s Report Card, 64% of grade-4 students are below proficient in math. At this point, grade 4 students may struggle with adding and subtracting multi-digit numbers. By grade 8, 74% of students are below proficient in math, meaning students may struggle to solve real-world problems with fractions, like converting cents to dollars. By grade 12, 76% of students are below proficient in math. At this point, grade 12 students may struggle to solve single-step percentages in real world problems like calculating sales tax or adding a tip to a check at a restaurant. Making matters worse, these learning gaps are unevenly distributed and disproportionately affect students in historically under-resourced communities.
Khan Academy’s Commitment to Math Learning
Helping more students succeed in math has been a long-time goal for us at Khan Academy. From Khan Academy’s earliest years since our founding in 2008, we’ve offered a deep library of mastery-enabled math content designed to help learners fill knowledge gaps and gain foundational math skills. More recently in 2023, we introduced Khanmigo, an AI student tutor designed to help guide students through structured problem-solving and provide encouragement and motivation along the way.
The Potential of AI in Math Tutoring
AI holds immense potential to realize real, tangible aspects of learning science that no previous technology has been able to deliver. AI enables students to receive both tailored support and feedback in real-time, as learning is taking place. Both of these are critical, according to learning science. Yet before the advent of AI, the only way for students to receive both was working with a good human tutor. Now, AI can approximate what a good human tutor can do, which we believe will help millions of students unlock more of their potential.
There’s just one catch: generative AI was not designed to handle mathematical reasoning. Rather, genAI was designed to focus on language. It appears to solve math when asked, but it is doing so in a way that leads to inaccuracies. Rather than calculating an answer, it uses all of the language it is trained on to predict the most probable numbers to come next in the sequence. The most probable number in the training data is not always the correct answer.
How Khan Academy is Closing the Math AI Gap
While math wasn’t part of genAI’s original design, it doesn’t mean it’s not possible. In fact, it’s very possible, though not without additional engineering effort to ensure the AI access to additional tools and information that it can access to obtain correct answers to math problems and effectively tutor students. This is where Khan Academy comes in – the very same mastery-enabled math content that Khan Academy has used to empower millions of learners to succeed in math is the very same content that can be used help AI-based systems to improve in math. Of course this is just a starting point, but it’s an enormous leap forward.
Enhancing Khanmigo’s Math Capabilities
Because we hold everything we produce to high standards, including Khanmigo our AI tutor, we’ve taken a significant number of additional steps to ensure Khanmigo’s math computation and tutoring abilities surpass those of its underlying large language model (LLM):
- We built a calculator for Khanmigo to solve numerical problems instead of relying on AI’s predictive capabilities. Read more on the Khan Academy blog
- We re-engineered Khanmigo’s responses so that Khanmigo more closely mimics how a live tutor works with a student.
- We consistently assess various large language models (LLMs) to find the latest best fit for application in math tutoring. Most recently, we moved Khanmigo math tutoring from GPT-4 Turbo to GPT-4 Omni, which we found leads to better tutoring performance.
- We’ve improved the way AI “thinks” during a tutoring session before responding to a student. We have instructed the AI to write out all the ways in which the student may have arrived at their answer behind the scenes. This allows Khanmigo to follow students’ steps more accurately regardless of which solution path they choose. We’ve found it significantly improves the quality of math interactions.
- We created a benchmark dataset of math tutoring conversations to help us evaluate how AI models can act like a tutor. Since then, the dataset has led us to understand that, compared to raw GPT4o (the LLM powering ChatGPT), Khanmigo is much better at catching and pointing out mistakes. This dataset also led us to a set of Khanmigo’s most common math tutoring mistakes, which we have since made improvements to address. Read more on the Khan Academy blog
- We conducted a human-driven, manual review of a large set of Khanmigo’s conversations with students to identify whether Khanmigo provided accurate responses. Through this exercise, we learned that when Khanmigo has access to human-generated exercises, steps, hints, and solutions prior to making a calculation or evaluation, Khanmigo’s accuracy improves. This finding led us to update Khanmigo’s architecture to ensure Khanmigo consistently takes the additional step to gather context from these sources prior to responding back to the student. The manual review exercise also revealed that Khanmigo occasionally struggles to interpret graphics. This finding led us to generate textual representations of all graphics so that Khanmigo can easily “see” what a learner is looking at by reading the descriptive text for the graphic.
- We improved Khanmigo’s ability to interpret the different types of exercises on Khan Academy. While Khanmigo has always been good at comprehending certain exercise types (ex. multiple choice questions) it had trouble coaching learners on exercises that contained graphs, number lines, or other visual content. We analyzed the different types of exercises across Khan Academy and spent time closing gaps in Khanmigo’s ability to understand those different types. Now Khanmigo is able to comprehend nearly all exercise content on Khan Academy and coach learners on it effectively.
- We have also engineered new advanced math capabilities for Khanmigo that improve Khanmigo’s ability to work with symbols, which resulted in improved accuracy for geometry, calculus, and trigonometry.
- Finally, we have created a new Khanmigo tutoring quality dashboard and metrics, and implemented monitoring that enables us to track the impact of our updates on math error rates.
Looking Ahead: The Future of Khanmigo
As is the work with any novel technology, our work is ongoing. We consistently monitor Khanmigo’s math computation and tutoring performance, consistently evaluate available generative AI models to ensure we are running the best possible model for Khanmigo’s use case, and regularly share our findings with the field. Though we are still learning alongside Khanmigo, we are confident in the strides that we’ve made to bolster Khanmigo’s capabilities in a short time. As a result of this work, we believe Khanmigo’s capabilities are very strong and will only become stronger over time. We look forward to continuing to share updates with you, our readers, as these capabilities evolve.
Onward!
The post Khanmigo Math Computation and Tutoring Updates appeared first on Khan Academy Blog.
By Kristen DiCerbo, Khan Academy’s chief learning officer Many U.S. students struggle with math. It’s a trend that starts early in elementary school, and snowballs over time: as students progress through grade levels, math concepts tend to build on one another from year to year. For students who struggle with math in earlier grades, that
The post Khanmigo Math Computation and Tutoring Updates appeared first on Khan Academy Blog. Math, News Khan Academy Blog
