The Best AI Assessment Tool Is Still a Good Question

Two stories landed in the same week, and on the surface they point in opposite directions. National History Day just banned generative AI from its student contest. Meanwhile, assessment researchers are using AI to build new kinds of tests. Read them side by side, though, and they agree on the thing that matters most in AI assessment: the strongest evidence of learning is a student explaining their own thinking to someone who asks.

One story is about keeping AI out. The other is about bringing AI in. Both end up at the same place. A conversation.

That should get every school leader’s attention. Not because of what it says about tools. Because of what it says about where we look for proof that learning happened.

What’s happening: one ban, one build

On October 8, Education Week’s Sarah Schwartz reported that National History Day changed its contest rules to ban generative AI for brainstorming topics and creating entries. Until now, the organization had allowed students to use AI while brainstorming topic ideas or searching for sources. More than 500,000 middle and high school students take part each year.

The reason is not abstract. Lynne O’Hara, National History Day’s deputy director for education programs, told Education Week that over the last two contest cycles, at least two projects that reached the final rounds were flagged for citing sources that AI had hallucinated. “We want to make sure that AI isn’t taking away student voice and student thinking,” she said. “We value their ideas, their analysis, and their conclusions.”

Here is the detail I keep coming back to. According to O’Hara, judges want to see what students can do on their own, even if it is imperfect and less polished, and students who didn’t do the research themselves would struggle to answer the questions judges ask about how the project was developed. She also said that when a problem is identified, the organization most of the time treats it as a learning moment.

The same article shows teachers working through the other side. Amy Allen of the Virginia Tech School of Education described ways students could first generate their own research questions and then compare them with what AI produces, or ask AI for feedback on their historical analysis, with the teacher playing “a pretty significant role.” Lindsay Miller, a 7th grade social studies teacher in Wayne, N.J., described AI making up sources and handing struggling students full paragraphs of answers. She now uses it behind the scenes to set up lessons, while students do the learning.

The same day, K-12 Dive’s Ed Finkel published an interview on what role AI could play in reimagining assessment. Temple Lovelace, executive director of the Assessment for Good program at the Advanced Education Research and Development Fund, described using AI for “inventive item formats,” including Socratic-style conversations in which a learner talks and the system assesses their thinking. She also pointed to faster scoring and to measuring durable skills such as critical thinking, collaboration and cognitive flexibility. “AI has allowed us to change what it means to assess,” Lovelace said. In her view, formative assessment is “where AI starts to shine,” because it has to be responsive, quick and nimble.

What this reveals about AI assessment

Notice what both stories have in common. Neither one trusts the finished product on its own.

National History Day could have relied on the ban alone. It didn’t. It relies on judges asking questions. The researchers could have built a faster multiple-choice machine. Instead, the most interesting thing Lovelace described is a conversation. When the polished artifact can be produced in seconds, everyone, from contest organizers to assessment designers, starts looking for the thinking behind it.

This is the heart of Chapter 6 of Human Still Required, “What Still Counts (And What No Longer Does).” I argue there that output is cheap now and insight is not, and I offer the D.E.C.I.D.E.R. framework as a way to name what still counts as evidence of learning: Decision-making, Explanation, Critique, Inference, Development, Evaluation and Relocation, meaning transfer to new contexts. A hallucinated source in a finalist’s bibliography is exactly the kind of thing that a product-only system misses and a conversation catches.

A student can generate an answer. They cannot fake understanding in a conversation.

Human Still Required, Chapter 6

Chapter 10 makes the leadership version of the same point. When answers get cheap, the questions become the differentiator. The judges at National History Day are not running detection software. They are asking good questions about process. That is a skill, and it is one we can build in every teacher and every leader.

Automation makes answers cheap. That’s not a threat—it’s a test.

Human Still Required, Chapter 10

Why it matters for school leaders

Most K-12 assessment systems were built to collect products: the essay, the worksheet, the project board, the test. Conversations happen, but they rarely count. They live in the margins of a busy week. If the strongest evidence of learning is now the conversation, our systems are weighted the wrong way.

There is also a caution in the K-12 Dive interview that leaders should hold onto. AI that scores a student’s conversation is still a tool making an inference about a child. Lovelace sees its best fit in formative assessment, where quick, rich feedback guides the next lesson. That fits what I argue in Chapter 4: decision-support is not decision-making. A system can surface patterns in student talk. A teacher still has to decide what that talk means for this student, today.

Let AI prepare you to decide. Do not let AI replace the act of deciding.

Human Still Required, Chapter 4

And there is a quieter lesson in the National History Day story. The organization drew a clear line, and it explained why. It also chose to treat most problems as learning moments. Clear parameters and a learning response are not opposites. Students need both.

What leaders can do now

• Audit where evidence comes from. Pick one course or grade level and list its major assessments. How many count a product only? How many include a moment where a student explains, defends or revises their thinking out loud?
• Build the “process question” habit. Give teachers a short bank of process questions, like the ones contest judges ask: How did you choose this source? What did you change after your first draft? What would you argue differently now? Practice them in PLCs before using them with students.
• Make conversations count. If conferences and oral defenses happen but don’t show up in grades or reports, they will always be the first thing cut when time gets tight. Decide where they belong in your grading and reporting practices.
• Set clear parameters, then explain them. Whatever your AI rules are for a given task, say when AI can be used, when it can’t, and why. National History Day’s lesson is that clarity helps students more than ambiguity does.
• Keep a human in the loop on AI-scored assessment. If you pilot conversational or AI-scored assessment, start with formative uses, and be clear about who reviews the results and who makes decisions about students.

If your leadership team wants to work through Chapter 6 and the D.E.C.I.D.E.R. framework together, the 5-Week Book Study is built for exactly that kind of conversation.

Frequently asked questions

Should schools ban AI on major projects and assessments?

Some tasks do need AI-free evidence, and it is reasonable to say so clearly. But a ban alone does not show you what a student understands. National History Day pairs its rule with judges who ask students about their process, and that questioning is what reveals whether the thinking was the student’s own.

How can AI be used in assessment without replacing teacher judgment?

Start with formative assessment, where speed and rich feedback help teachers plan the next lesson. Treat AI-generated scores or insights as input for the teacher, not as the final word. The teacher still decides what the evidence means for each student.

What counts as evidence of learning when AI can produce the work?

Evidence that is hard to fake: a student’s ability to explain choices, critique sources, defend a conclusion, revise in response to feedback and transfer ideas to a new situation. The D.E.C.I.D.E.R. framework in Chapter 6 of Human Still Required names these as Decision-making, Explanation, Critique, Inference, Development, Evaluation and Relocation.

The question is the assessment

One organization kept AI out. One program built AI in. Both ended up valuing the same thing: a student who can talk about their own thinking.

That is not a technology insight. It is a teaching insight we have always had, and AI just made it impossible to ignore.

So here is my question for you: in your school, when was the last time a student’s explanation counted more than the thing they turned in?

Sources

• In Social Studies, Does AI Shortcut Historical Thinking? Or Deepen It?, Education Week, October 8, 2026
• What role could AI play in reimagining assessment?, K-12 Dive, October 8, 2026