AI Is a Rater, Not an Exception: The Reliability Evidence AI Marking Now Needs

AI marking has moved from the pilot to the procurement conversation. Assessment organisations are being offered systems that mark faster than any team of examiners and agree with them most of the time. The question that matters is not whether those claims are true. It is what evidence would show that they are.

The answer is less novel than it sounds. When an AI marker enters an assessment pipeline, it joins the rater pool. It does not replace it. Ofqual’s January 2026 working paper on AI in marking makes the regulatory position plain: using AI as the sole mechanism for determining a mark does not comply with its regulations, and agreement with human marks is not, on its own, enough to assure validity. This article sets out the evidence that fills that gap, with the statistics, the thresholds, and what belongs in the evidence pack.

AI is a rater, not an exception

The temptation when AI marking arrives is to treat it as either better or worse than human marking, and to skip the work of measuring it the way human markers are measured. Both instincts are mistakes. The defensible position is to evaluate the AI marker with the reliability framework you already apply to examiners, with three adjustments.

• The AI marker is one entity, not a team, so the statistics that compare examiners with each other now compare examiners with the AI as well.

• It drifts differently. Human markers drift with fatigue and familiarity. AI markers drift when the model, the prompt, or the surrounding pipeline changes.

• Its behaviour across candidate groups is the question that matters most, because studies have found that AI marking    performance can vary across demographic groups, a point Ofqual’s paper makes directly.

Once those adjustments are made, the rest of the discipline carries over directly.

Agreement is the foundation, not the finish

Agreement statistics measure whether a different qualified marker would have reached the same mark. No marking decision can be defended without that evidence.

For two markers on an ordinal mark scale, Cohen’s kappa measures agreement beyond chance, and its weighted form, quadratic weighted kappa, is the working standard in automated scoring. The framework published by Williamson, Xi and Breyer at ETS in 2012 remains the most widely cited benchmark. AI and human agreement should reach a quadratic weighted kappa of at least 0.70, it should be no more than 0.10 below the agreement your examiners achieve with each other, and the standardised mean difference between AI and human marks should sit within 0.15 either side of zero.

The second of those criteria deserves emphasis. The benchmark is not a fixed number. It is your own examiners. An AI marker that agrees with humans at 0.72 on a task where examiners agree with each other at 0.90 has a problem that a headline figure would hide.

For continuous mark scales, the intraclass correlation coefficient does the same job. The usual interpretation treats values below 0.50 as poor, 0.50 to 0.75 as moderate, 0.75 to 0.90 as good, and above 0.90 as excellent. The form of ICC matters, so state which one you used.

Standard 2.7 of the Standards for Educational and Psychological Testing asks for evidence of interrater consistency wherever judgement enters scoring, and its commentary warns that agreement between raters does not by itself guarantee reliable scores. Everything else in the evidence pack follows from that caution. High agreement is necessary. It is nowhere near sufficient.

Harsh, lenient, or simply different?

Agreement tells you whether markers converge. It does not tell you whether they converge in the right place. A simple comparison of mean marks, expressed as a standardised difference, catches an AI marker that is systematically half a grade generous or harsh against the human consensus.

For performance and constructed-response assessment, Many-Facet Rasch Measurement goes further. It estimates candidate ability, task difficulty, and marker severity together, which places the AI on the same severity scale as every examiner on the panel. If the AI sits inside the spread of your human markers, that is supportive evidence. If it sits at the extreme, the validity argument has to explain why the cut score is still defensible. Fit statistics in the region of 0.5 to 1.5 indicate a marker whose behaviour is consistent enough to measure. Outside that range, the marker is either too predictable or too erratic to trust.

Fairness lives at the criterion level

Consistency is not fairness. For fairness you need differential functioning analysis, which asks whether candidates of equal underlying ability receive systematically different marks depending on their group. The Mantel-Haenszel procedure, with the ETS classification of negligible, moderate, and large differences, is the established starting point, and Williamson and colleagues recommend a tighter flag of 0.10 for subgroup mean differences between AI and human marks.

Two points are specific to AI marking. First, run the analysis at the level of individual rubric criteria, not only total marks. An AI marker can look even-handed on totals while treating one group differently on a single criterion, and that criterion can decide outcomes for candidates near the pass mark. Second, include the groups your candidate population and the law require: sex, ethnicity, age, disability and access arrangements, and assessment centre. In the UK, the Equality Act 2010 makes this a legal obligation, not good practice.

The quiet failure: marks that hug the middle

A subtler problem is range restriction. A marker who only ever awards the middle of the scale differentiates candidates poorly, even when agreement figures look respectable.

AI markers are prone to exactly this. Models tuned to avoid large errors can drift towards safe, central marks that fail to separate strong candidates from weak ones. Compare the spread of AI and human marks directly. A common benchmark keeps the ratio of their standard deviations between two thirds and one and a half. This matters most where the cut score sits away from the median, because compression pushes more candidates towards the boundary, which is precisely where classification errors happen.

Drift belongs to versions, not Mondays

Human markers drift across a long marking session or a season. AI markers drift when their underlying model is updated, and Ofqual notes that providers may not always explain or announce those updates.

The discipline is to anchor monitoring to versions. Log every change to the model, the prompt, the thresholds, or the post-processing, and re-run your agreement statistics on a held-out set of examiner-marked scripts before the new version marks a live candidate. If the new version marks differently from the old one, the change is material and needs governance approval. In live operation, statistical process control does the rest: control chart points beyond two standard deviations are a warning, beyond three are a signal to act.

The decision is what gets appealed

Candidates do not appeal coefficients. They appeal outcomes. The standard error of measurement converts reliability into something an appeals panel can use, and Standard 2.14 expects conditional standard errors to be reported near every cut score. Standard 2.16 goes further for pass and fail decisions, asking what proportion of candidates would be classified the same way on a replication.

In practice, this gives a clean operating rule. Where the confidence interval around a candidate’s mark crosses the cut score, the decision is statistically marginal and goes to additional human review, regardless of whether a human, an AI, or both produced the mark. That rule moves the conversation from whether the AI got it right to how confident anyone can be at that mark, which is the question regulators and appeals panels actually ask.

Where the design supports it, generalisability theory completes the picture. A decision study shows whether adding the AI as a second marker improves reliability more than adding another examiner or another task, which is the business case for AI marking expressed in measurement terms.

What belongs in the evidence pack

For each component where AI influences a mark, refreshed on a defined cycle and tied to the model version in use at the time:

• agreement between AI and human markers, benchmarked against human-to-human agreement on the same scripts

• severity diagnostics for the AI marker, including a Many-Facet Rasch analysis where the design allows

• differential functioning analysis at both total and criterion level for every relevant candidate group

• spread diagnostics that would expose compression towards the middle of the scale

•  version-anchored drift monitoring, with statistical process control in live operation

•  standard errors and classification consistency at the cut score, with the rule that routes marginal decisions to human review

This is a substantial body of evidence. It is also the body of evidence that defends a credential when a decision is challenged.

One evidence base, many questions

Readers following this series will recognise the destination. The EU AI Act brings high-risk obligations for specified AI uses in assessment from December 2027, including accuracy, robustness, and human oversight. ISO/IEC 17024:2026 expects certification bodies to monitor and validate AI-generated outcomes. Ofqual expects marks to be explainable and decisions to rest on human judgement. Each asks, in its own vocabulary, for numbers an auditor can verify. Reliability evidence is those numbers.

None of this is new to psychometricians. The Testing Standards have asked for it for decades. AI does not change the obligation. It changes the work, and it raises the cost of not having done it.

Start with one component

The organisations that build this evidence are not doing extra work for its own sake. The diagnostics that catch an AI marker drifting also catch a human examiner drifting, so the work strengthens the credential whether or not the AI stays.

If you are piloting AI marking, start small and start now. Double-mark a few hundred scripts from one component with your examiners and the AI, then run the agreement, severity, subgroup, and spread analyses before anyone asks for them. The results will tell you more about the system you are buying than any demonstration will.

By Martin Melconian, Founder and CEO of Globebyte, an AI Systems Integration firm specialising in regulated industries, credentialing bodies and associations. Follow Martin at https://www.linkedin.com/in/melconian or for more insights at https://globebyte.com