The AI Evidence Gap Is Real. Leading Through It Is the Job.

Two research organizations on two different continents said the same thing this week: the evidence on how AI affects student learning is thin. That is the AI evidence gap, and it is real. But school leaders don’t get to wait for it to close.

Students are already using these tools. Teachers are already experimenting. Boards are already asking for a plan. The research will come. The decisions are due now.

So the question isn’t whether leaders have enough evidence. They don’t. The question is how they lead well without it.

What’s happening

On October 15, Education Week reported on a new set of federal policy recommendations from the Alliance for Learning Innovation, a coalition of researchers, practitioners, funders and nonprofits. In Education Week’s coverage of the call for federal AI research funding, Lauraine Langreo lays out the problem plainly. Stanford’s SCALE Initiative reviewed more than 800 academic papers on AI in K-12 education, and only 20 produced strong causal evidence. Meanwhile, districts are writing policies, training teachers and making purchasing decisions with little evidence to go on.

The coalition is asking Congress and the administration to build a stronger evidence base, improve AI tools, build research infrastructure and coordinate a unified AI-in-education research agenda. Its executive director, Sara Schapiro, put it this way: “No state, district, or school should have to answer these questions alone.”

The same article notes that a majority of school leaders surveyed by the National Principals Association are at least somewhat confident about leading AI implementation. For those who are less confident, CEO Ronn Nozoe said, “the absence of clear evidence and practical guidance is a real barrier.”

A day earlier, on October 14, the Australian Education Research Organisation (AERO) released advice for school leaders and policymakers. EducationHQ’s summary of AERO’s new AI explainers reports that robust research on AI and learning outcomes is limited, that most studies are recent, short-term and exploratory, and that little research looks at whether learning lasts once the AI is taken away. Many studies measure engagement or perceptions rather than learning itself.

AERO names a “performance paradox”: students may perform better with AI but learn less. As ABC News reported on AERO’s warning about AI and long-term learning, CEO Dr. Jenny Donovan pointed to overseas studies where students who used AI could not reproduce their results in an exam without it. AERO’s advice is to use AI sparingly, to avoid it in the first three years of primary school, and, where it is used, to make sure it scaffolds learning rather than replacing the cognitive work students need to do themselves.

Two organizations. Two countries. One message. We don’t know as much as the sales pitches suggest.

Why the AI evidence gap is a leadership test

Here is what I want leaders to notice. Neither report says “stop.” Neither says “go.” Both say, in effect: decide carefully, be clear about why, and keep watching.

That is judgment work. It is exactly what I describe in Chapter 7 of Human Still Required. Information has scaled. Responsibility has not. A study, a vendor dashboard or a federal report can inform a decision. None of them can carry it for you.

AI didn’t make judgment harder. It made it unavoidable.

Human Still Required

When the evidence is strong, leaders can lean on it. When it is thin, they have to lean on something else: a clear understanding of how students learn, honest knowledge of their own context, and the willingness to say, “Here’s what we know, here’s what we don’t, and here’s what we’re doing about it.”

That is why AERO’s guidance stands out to me. It doesn’t pretend to have the answers on any specific tool. Instead, it anchors decisions in something that hasn’t changed: students build knowledge by doing the thinking. That is a principle you can lead from even when the studies aren’t in yet. It is the same line I draw in Chapter 5. AI should extend thinking, not finish it.

The performance paradox is also a warning about how we measure success. If a pilot shows higher scores on AI-supported work, that may tell you about the work, not the learning. Leaders who judge an AI rollout by how polished the output looks are measuring the wrong thing.

Uncertainty is not a reason to freeze

In Chapter 15, I write about what schools must embrace, and uncertainty is near the top of the list. Many of us were trained to lead from certainty. Have the plan. Know the answer. Project confidence.

This moment doesn’t allow that. And pretending otherwise is how districts end up swinging between extremes: a ban one year, an all-in rollout the next, with no clear thinking connecting the two.

Embracing uncertainty is not weakness—it is leadership.

Human Still Required

Embracing uncertainty does not mean doing nothing. It means making decisions you can explain, setting a date to revisit them, and being honest with staff and families that you will adjust as you learn. Restraint is part of this, too. Choosing to go slower in the early grades, or to limit AI to a few well-defined uses, is not falling behind. It can be the most responsible call on the table.

What it can’t be is avoidance. Saying “we’re waiting for the research” while students use these tools every night at home is not caution. It is leaving teachers and families to figure it out alone.

What leaders can do now

• Write down your learning principle before you write down your tool list. AERO’s anchor is a good starting point: if AI is used, it should support students to acquire, retrieve, practice and apply knowledge, not do that work for them. Every AI decision should be tested against that sentence.
• Be specific about where students work without AI. AERO recommends clarifying when students must work independently, when AI use is appropriate and how teachers will keep oversight. Make that visible by course and grade band, not buried in a policy appendix.
• Measure learning, not output. If you run a pilot, check what students can do when the tool is gone. A short, AI-free check a few weeks later tells you more than a stack of polished assignments.
• Put a review date on every AI decision. Thin evidence means today’s best call may need to change. Saying so up front builds trust. Changing course quietly erodes it.
• Ask vendors the evidence question, and expect an honest answer. What does the research say about this tool’s effect on learning, and how long did students use it? “We don’t know yet” is an acceptable answer. Pretending to know is not.

If your leadership team wants to work through questions like these together, the 5-Week Book Study is built for exactly that kind of conversation.

Frequently asked questions

Is there good research on AI and student learning in K-12?

Not much yet. Education Week reports that a Stanford SCALE review found only 20 of more than 800 papers on AI in K-12 produced strong causal evidence. AERO in Australia similarly found that most studies are recent, short-term and exploratory, and few check whether learning lasts once AI is removed.

What is the “performance paradox” with AI in schools?

It is AERO’s term for the risk that students perform better while using AI but actually learn less. The work looks stronger, but the knowledge isn’t built. For leaders, it means AI-assisted output alone is not good evidence that a tool is working.

Should our district wait for more research before adopting AI?

Waiting for perfect evidence isn’t realistic when students already use these tools. A better approach is to make limited, well-reasoned decisions anchored in how students learn, measure what students can do without the tool, and set a clear date to revisit. That is judgment, not guesswork.

The work in the gap

The call for more federal research is right. I hope it gets funded. I hope the studies get longer, bigger and closer to real classrooms.

But leaders can’t outsource this moment to a future research agenda. The evidence gap is where leadership happens right now. It is where we show our staff and families what we value, how we think and whether we are willing to change our minds when we learn more.

So here is my question for you: if someone asked you today why your school uses AI the way it does, could you explain your reasoning without pointing to a vendor or a headline?

Sources

• We Need Research on AI in Schools. The Feds Should Fund It, Report Says — Education Week, October 15, 2026
• Education research body urges schools to use AI sparingly due to learning risks — ABC News (Australia), October 14, 2026
• Little robust evidence to show AI’s impact on student learning: AERO — EducationHQ, October 14, 2026