The Myth That AI Will Fix Learning Gaps

Well, yeah. Isn't that the whole story?

That's the story people tell. The Stanford SCALE Initiative ran two randomized trials, 355 students, grades one to five, two districts. Access alone changed almost nothing.

Two minutes? That's nothing.

In the independent-use group, average use was two to five minutes a week. Two minutes. That's, like, tying one shoelace.

Okay, so what changed the picture?

Human support. Add a human tutor alongside the platform and engagement jumped seventy-one to eighty percent. Still low overall, but the direction mattered.

So it's not AI versus teachers. It's AI with teachers.

That's what the implementation story suggests. And UNESCO's 2023 guidance lands in the same place — a human-centred approach, policy and teacher training before software.

And that's kids' data. Were countries even ready?

They recommend a minimum age of thirteen for classroom AI tools, for example. Plus data protection, because using generative AI usually means sharing user data with the provider.

assess first, don't just adopt. Which means... they shouldn't assume it works?

Right — UNESCO says build evidence rather than assume educational value. Which brings me to the UK trial. Want the hopeful version?

Yes. Give me the hopeful one.

There's this interview I keep replaying. Here's Alfred Burgesson on the AI From The Ground Up podcast, on the questions that actually matter for students.

"At what age, for what purpose, under whose supervision." He's reframing it — not whether, but how.

Exactly. And the UK trial speaks to the how. 165 students, five secondary schools, generative AI tutoring in mathematics on the Eedi platform, using LearnLM. [[CLIP:4dcb0351-eff1-435d-962c-fcb6ea962b87:AI From The Ground Up — Alfred Burgesson:33750]]

Five. That's, that's remarkably clean.

They supervised it and kept responsibility for every interaction. And when researchers reviewed the drafted messages — five factual errors out of three thousand six hundred seventeen, no harmful content.

So is that the proof point? Should every school copy this?

Careful. The authors themselves call it exploratory. Small trial, qualified findings. UNESCO would say that's a reason to keep gathering evidence, not a mandate.

So both things are true. Stanford says kids won't use it alone. UNESCO says it could widen gaps if we rush.

It is. Generative AI could deepen existing inequalities. And there's the content question — material used without consent.

So the debate isn't AI replacing teachers — it's what stays human.

Access isn't enough. Humans make the tool matter. That's where I'd plant the flag.

And that's the fight worth having. Because if the Stanford kids taught us anything, it's that the technology doesn't show up on its own.

No. Someone has to sit beside the kid and say — open it.

Picture a science teacher walking out of school on a Friday with twenty-five extra minutes she didn't used to have. That's my optimistic opening. Where's the harm?

Nowhere yet — and that trial's solid. Two hundred fifty‑nine teachers, sixty‑eight schools in England. ChatGPT plus a usage guide, about twenty‑five fewer minutes of prep per week, no drop in material quality.

So AI already changed school. Done. Debate over?

That's the optimist's move, and I get it. But the skeptic says you just saved the teacher's evening — you didn't touch the kid's brain.

Right. Okay, so the skeptic needs evidence too. Not just vibes.

The skeptic's best card is the OECD's 2026 review. General‑purpose generative AI can improve performance on a task without producing learning gains.

Wait, how can you perform better but not learn anything?

Because the OECD defines learning as lasting change in knowledge and skills — retention, transfer. A polished AI‑assisted answer doesn't demonstrate any of that.

And it gets worse?

Some studies found the performance advantage disappeared — or reversed — when students later worked without AI.

And here's a concrete number for that reversal — listen to Dan Fitzpatrick on the AI for Educators Daily with Dan Fitzpatrick podcast. [[CLIP:90905e8b-663e-4342-9319-17dfac008719:AI for Educators Daily with Dan Fitzpatrick — Dan Fitzpatrick:32888]]

Sixty-seven down to fifty-seven — that's the reversal in the flesh, and not just a simple 'AI is bad' story either.

Reversed.

Yeah. So the two visions both look at the same tool and see different things.

So let's actually test them. My optimist says: tutoring works. Bloom, back in 1984 — tutored students scored two standard deviations above the class average.

Careful — that was tutoring, human tutoring, not AI tutoring. The optimist borrows Bloom's number and hopes the machine inherits it.

Fine, but there's AI‑specific evidence. A review of twenty‑eight studies, four and a half thousand students—

Hold on. What did it actually find when the comparison was fair?

...generally positive effects for AI tutoring. But the effects were smaller—

Than against what? That's the pressure point.

Smaller than against non‑intelligent tutoring systems. So plain software did almost as well.

Which the optimist reads as 'AI helps' and the skeptic reads as 'the intelligence isn't doing the work.' Same number, opposite conclusions.

Give me the concrete case where that split plays out. Like when—

Think of the Stanford review finding. Students preferred an AI chatbot, rated it more helpful — and the chatbot group had the lowest comprehension and retention among the conditions.

They liked it more and learned less. That's, that's brutal for my side.

It's brutal for both. The kid who loves the tool is the kid least equipped to judge it.

motivation. A 2022 review of thirty‑two studies — AI education boosted motivation and engagement in K‑12.

True finding. But that same review flagged a shortage of objective measures of knowledge. Engagement is easy to count; learning wasn't measured well.

So both sides keep citing the same weak spot — we can't cleanly measure learning.

Exactly. And there's a third reading. Microsoft Research found generative AI shifts where critical‑thinking effort goes — from gathering information to checking it.

So the skill doesn't vanish, it, it relocates?

That's one way to put it. The OECD's version of hope: AI supports learning when it has a clear pedagogical purpose, or when teaching is redesigned around it.

So if I concede the skeptic's point — performance isn't learning — does the optimist survive?

AI changes school when someone redesigns the schoolwork. Not the software alone.

I'll take that. Because my twenty‑five saved minutes — that's a win if the teacher spends them on the kid.

And that's the choice. It survives your best objection, because even the reversal studies leave that door open.

Okay, so a school district buys every kid an AI tutor and the achievement gap just... closes itself. That's the pitch, right?

In its strongest form, yes. 'The best teachers, personalized, in every pocket — inequality solved.' And if a superintendent believes that version and spends the budget on it, the kids it was meant to help get nothing.

Because it's false advertising?

Because it's backwards. The districts with the biggest gaps are the least ready to use the tool at all.

Wait — least ready? They need it most.

generative AI won't address fundamental education challenges unless it's inclusively accessible and designed to advance equity. It's not a standalone remedy.

That's, that's the polite version. What does the impolite data say?

RAND surveyed U.S. teachers in 2023. Teachers in higher‑poverty schools were less likely to use AI tools than teachers in lower‑poverty schools.

So the tool arrives where the gaps are smaller.

And it deepens them — UNESCO says exactly that: AI can widen existing disparities when learners differ in access to technology and resources. The principals part is worse.

Principals?

RAND found principals in higher‑poverty schools gave AI‑use guidance less often. So the kids with the least support at home also get the least guidance at school. Nobody says that part out loud.

That's the thing the sales pitch leaves out, sitting right there. Okay — but has technology ever closed a gap? Any case?

One, and it's my favorite. China, 2004. Rural students got recorded lessons from highly qualified urban teachers.

And that worked?

Worked is an understatement. Nicola Bianchi, Yi Lu, and Hong Song studied it. Exposed students completed 0.85 additional years of schooling—

Point‑eight‑five years. From watching recordings?

And later earned fifty‑nine percent more than people in the same county who weren't exposed. Fifty‑nine percent more, on average.

So the myth is... half right? Technology did bridge a gap.

No. Look at what actually ran. Students watched at school, with local teachers supervising the classroom. It wasn't the recordings alone — it was great teaching delivered to underserved kids, with a human holding the room.

So the delivery system mattered more than the—

Than the screen, yes. And UNESCO's report shows the flip side. Distance learning could reach a billion students — it missed at least five hundred million. Seventy‑two percent of the poorest.

Same technology. One closes a gap, one widens it.

Same pattern everywhere. Distributing computers alone doesn't improve learning if teachers aren't in the pedagogical experience. Benefits can disappear when tech is used excessively or without a qualified teacher.

So what would I actually wager on? If someone forced me to bet that AI closes gaps anywhere—

That bet would be at long odds, but with conditions. Access — identify kids without affordable internet and reduce that barrier. Second, a teacher redesigning the work around it. Third, judged by learning outcomes, not by how many devices arrived.

Because schools count inputs. iPads deployed, licenses purchased.

And UNESCO's GEM report says sound, impartial evidence on ed tech's learning impact is limited. Adoption isn't proof of learning. Count what was learned, not what was bought.

One more pressure point. Could the rollout actively hurt the kids it claims to help?

Yes. The U.S. Department of Education warns that patterns in training data can make AI detect patterns in biased ways and automate unfair decisions — like tracking. It wants human alternatives and recourse when automated recommendations go wrong.

So a struggling kid gets an algorithm's low expectation, locked in.

With no teacher in the loop to override it. That's the harm — not the tool, the missing human.

So the strong myth doesn't weaken. It collapses — because its two pillars, access and teacher support, are precisely what's missing where the gaps are.

Which is why it collapses instead of bending. The claim that survives is more modest: AI can help narrow gaps when it carries quality teaching to underserved learners, with access and equity built in.

And there's fresh evidence on exactly how that modest claim plays out. Here's Jon Krohn on the Super Data Science: ML & AI Podcast with Jon Krohn, on what happened when students actually got AI access. [[CLIP:8f3c3f66-d5ea-4949-8315-c860f3c2fcf6:Super Data Science- ML & AI Podcast with Jon Krohn — Jon Krohn:26567]]

So even the good news comes with strings attached — the gains only stick when the AI explains rather than writes it for you. Which brings us right back to the teacher redesigning the work.

The China story, but with a teacher in the room.

That's the whole wager I'd defend. Nothing more.

Picture a seventeen-year-old in Mendoza, stuck at home in medical treatment, quietly disappearing from school.

That's Francisco, from a UNESCO account. Nobody at school had noticed him slipping — until an early-warning alert fired on his file number.

The machine found him?

The machine flagged him. Then headmistress Eliana Moreira and her team actually called him. The alert didn't rescue anyone — it pointed, a human acted.

So that's today's question. Should teachers trust an alert like that?

Wrong framing, honestly. The better question is: what should a teacher do in the ninety seconds after the alert pops up?

Okay, so walk me through one teacher's day. The alert arrives — what does it look like?

Think of a dashboard, teacher-facing, never sent to the student. A 2026 study modeled risk from engagement patterns, learning difficulties, time allocation — and showed faculty the results as interpretive cues.

Wait. The kids never see their own risk score?

No. It lands on the teacher's screen, as a prompt to look closer — the study suggested clarifying a topic, extra examples, adjusting an assignment.

And when the teacher looks closer — does the alert hold up?

Sometimes not. The same study says false positives occur — alerts that reflect no real problem. Faculty are told to review them alongside qualitative information.

So the obvious suspect is the data. The model's just built on the wrong—

No. That's the lead that fails. Even good data can't settle it, because the alert is a signal, not proof. The case lives or dies on the teacher's judgment.

Hmm. Because the data can look clean and still be unfair?

Documented, yes. A 2021 review found bias in dropout prediction, essay scoring, emotion detection. And removing race from a model doesn't scrub it — correlated variables carry the information anyway.

If you want a feel for how lopsided this can get, listen to Mary Coogan on the Kidlaw podcast. [[CLIP:3482b946-e5ff-4c7c-92b1-b8dc99123a42:Kidlaw — Mary Coogan:22805]]

So the flags themselves can discriminate before any teacher even looks at them. Which makes that ninety seconds of judgment matter even more.

So... what's a teacher supposed to actually do? Give me the protocol.

Review — check the alert against what she knows. Validate — talk to the kid, look at work. Act — only then intervene. UNESCO puts pedagogical responsibility with the teacher, and demands schools prepare her for it.

And the override? When does she ignore the suggested micro-lesson?

When she knows something the dashboard can't — say the pattern is a language barrier, not a math one. Human oversight, transparency, accountability: that's UNESCO's line.

But does the alert even help? I want evidence it changes a kid's day.

Inq-Blotter. Eight middle-school teachers, 280 students — dashboard users gave feedback to roughly twice as many students. And the helped kids improved more on the next lab, interpreting data and warranting claims.

Twice as many. That's, that's a real day changed.

One teacher noticing one more kid per class, multiplied. Though a review of twenty-one studies warns the evaluations are small-scale, and evidence linking predictions to effective interventions is thin.

So the honest verdict on Francisco's story?

The alert worked because Moreira picked up the phone. And there's a caution under it — privacy. A Dallas Morning News report described student files exposed with names unredacted. The risk isn't only the model; it's the handling.

So trust isn't the question. Process is.

Right. And what would move me is findable: studies tracking whether alert-prompted help produces learning, at scale, across subgroups. That evidence, and I change my mind.

So a kid sits down at a screen, and the test is different from the kid next to him. That still feels like magic to me.

Think of baking a cake. The ingredients are the question pool, the mixing is the selection, and the cooling is the scoring. The timer, well, that's the part nobody designs.

Since we're leaning on the cake metaphor, let's hear how Dietmar Fischer on the A Beginner's Guide to AI podcast explains the way a machine actually learns about cakes. [[CLIP:4db3831a-dcb5-40aa-9b45-6af07f98fa61:A Beginner's Guide to AI — Dietmar Fischer:39158]]

So it learns from thousands of cakes, not one recipe. Which brings us back to our kitchen: where do the ingredients come from?

Ingredients, okay. So the pool of questions.

Right. And in adaptive testing, the student's answers to earlier questions help determine which questions appear next. The test changes as the student's performance becomes clearer.

So after every answer the system, what, updates its estimate?

Exactly. It re-estimates the student's level after each response and picks the next item using that estimate. It avoids questions far too easy or too difficult, and aims for items that reveal where the student actually is.

And the kid may not see the whole pool?

No. A student may see a subset of the available items. The test can stop after a preset number of questions, or when a score-precision target is reached.

Stop early.

Mm-hm. Which means two kids can sit the same test, same timer, and answer completely different questions at different difficulty levels.

Wait, that's the contradiction, isn't it? Everything inside is personalized, but the clock on the wall is uniform.

Same minutes for every kid, different test for every kid – and research shows that variable‑time adaptive exams can adjust the clock itself, giving more time to harder items and less to easier ones. The timer is designed for fairness, and it collides with the adaptation at every other stage.

Okay, so now the mixing stage. What is it balancing?

Difficulty density. Too dense, the kid drowns and the test keeps probing weaknesses she already knows she has. Too light, she coasts and the score tells nobody anything.

And the pinch of salt? You mentioned—

The bias filter. A brief caution: automatic question filters can unintentionally exclude culturally relevant items, so designers should audit the pool for hidden bias.

So the cake looks fine and tastes wrong.

And nobody tastes it until the scores come out. The U.S. Department of Education’s “audit‑as‑you‑go” guideline advises continuous checks of AI‑driven assessments for bias, ensuring diverse data representation throughout the testing pipeline.

So then the cooling stage, the scoring. That's the fast part?

Fastest by far, and that's the trap. Automated scoring arrives in seconds, so teachers scrutinize it least. But the guidance says scores should be judged not just by how closely they match human scores, but whether they relate to other measures of writing ability.

Do they? Match up, I mean?

An Educational Testing Service study of about 1,700 prospective graduate students found that combining one human rater with the e‑rater tool correlated with external writing criteria almost as much as two human raters.

Almost as much as two humans. So why not just let the machine score alone?

Because that same comparison suggests automated scoring is more informative when paired with human ratings than as a stand‑alone measure. One human plus one machine, not machine alone.

And the machine can be unfair in ways nobody checks. Any documented case?

A study found that seven AI‑writing detectors labeled many TOEFL essays written by non‑native English speakers as AI‑generated.

Sixty‑one percent.

At least one detector flagged 89 of the 91 essays. James Zou's explanation was that detectors rely on measures like perplexity, which penalize writing patterns common among non‑native English speakers.

So the kid who writes in a second language gets called a cheater.

By a tool nobody audited. Which is why the U.S. Department of Education wants teachers able to inspect AI systems, exercise professional judgment, and override AI recommendations when necessary.

So where's the single moment that matters most? The whole pipeline, where does the human have to stand?

Between mixing and serving. The Department of Education says AI decisions need protections against discrimination, notice, explanation, and a way to seek human recourse. Teachers should audit that question pool, and keep auditing it, before and while it's served.

And UNESCO's view on all this? Beyond cheating?

They say generative AI's implications for assessment extend beyond cheating, because these tools can produce essays and pass some knowledge‑based exams. So systems need to reconsider what learners should learn, and how learning is assessed and validated.

So the cake survives, if the baker stays in the kitchen.

I'd put it differently. The cake survives, and the baker's job changes. But I still don't think we've answered whether the clock should change too.

Meaning you'd give every kid a different amount of time?

Meaning I honestly don't know. The adaptation says yes, fairness says no, and I, I can't make those two sit comfortably together yet.

AI is going to revolutionize learning for children. That's what I genuinely believe — put a tutor on each screen and schools are transformed.

I know you believe it. And there's a real case where somebody actually ran that experiment, start to finish. Two years, real schools.

Who ran it? Can you provide the source? That claim is often heard from vendors.

Stanford researchers, in eighteen Tennessee middle schools. A cluster-randomized experiment — the real thing, not a vendor's brochure. It tested Khan Academy's Khanmigo tutor during remedial math sessions.

Two full years. That's, that's longer than most pilots survive.

And it's exactly the timeline a superintendent dreams about. So this is your school of tomorrow, actually built.

Okay, so walk me through it like a place. The grant lands, the devices arrive — what happens next?

Think of it in three beats. Access, engagement, outcomes. Access: ninety‑six percent of students tried the tutor at least once. Nearly everybody walked through the door.

See? Adoption. That's the revolution starting.

Wait. Second beat. The median student messaged the tutor on about a third of practice days.

A third.

And it gets sharper. When students made mistakes — the moment a tutor matters most — they messaged it in about seventeen percent of those sessions.

So the kid who's stuck just... sits there stuck.

limited engagement, not access, was the constraint. The door was open and most kids didn't walk through it.

But what were they doing when they did use it? Actual math questions?

Mostly bare answers, or clicks on suggested prompts. Not the substantive mathematical dialogue the dream promises.

Fine — grant my own side its strongest form. Some kids did use it. Did they learn?

Some did. Assignment to the program raised math achievement by about 1.3 national percentile ranks per term. That's a real gain. I'll give you that much.

There! It worked! The revolution, measured!

Now the part that hurts your side. The researchers noted those gains resembled what students got from Khan Academy practice without AI assistance.

Without... the AI. So the intelligence added —

Barely moved the needle beyond plain practice software. Same conclusion an earlier review reached: effects were smaller than against non‑intelligent systems.

Hold on. I want to check that claim before I concede. Source on that review?

A review of twenty‑eight studies, roughly four and a half thousand students. Same pattern: AI helps, but the AI part does the least of the work.

Okay. I concede the strongest version of my side fails. Tutoring didn't transform all children.

It fails as a replacement. But there's a version that survives — AI as a partner. Same Stanford group studied Tutor CoPilot: AI supporting human tutors, nine hundred tutors, eighteen hundred students.

And that worked better? Why would helping the adult work when helping the kid doesn't?

Because the adult already shows up. Students whose tutors had the AI support were four percentage points more likely to progress through math assessments.

Four points. Modest.

But look where it landed. Kids working with lower‑rated tutors gained up to nine points. The tool lifted the weakest teaching toward the middle.

So it fills gaps instead of replacing anyone.

That's the whole finding. Which means the school of tomorrow still needs its teachers trained — and that's where it usually falls apart.

What would real training even look? My superintendent would just schedule one workshop.

UNESCO built an actual teacher competency framework — fifteen competencies across five areas, including AI pedagogy. And it stages them: acquire, deepen, create.

So not one workshop. A, a progression over years.

And students get their own framework — twelve competencies, four areas, from a human‑centered mindset to AI ethics, staged as understand, apply, create.

It even calls students potential co‑creators, not just users?

That's the framing, yes. And before any of it runs, UNESCO says institutions should assess whether the tools are pedagogically and ethically suitable. Suitability before purchase.

And the data? All this tutoring runs on children's information.

The U.S. Department of Education flags privacy as a core concern and wants clear limits on collecting, using, transferring, retaining personal data. UNESCO adds age limits — a minimum of thirteen for classroom AI tools.

So a school doing this honestly — it, it has to build the guardrails before the grant arrives?

child safety, privacy, fairness, transparency, accountability, inclusion, and children's well‑being. Well‑being, notably, is on the list.

So — final honesty between us. Was my revolution wrong?

Wrong about the mechanism, right about the direction. AI can change school — when it carries good teaching to more kids, with trained adults holding the room.

Ninety‑six percent tried it. A third of the days, they used it.

That's the cost of the dream, in the unit it opened with. The screen reached everyone. The learning reached whoever a human sat beside.