The AI Benchmark Series: Can Machines Label Show, Don't Tell?

Conflict of interest. This benchmark evaluates large language models, and two of the systems evaluated in round three were Claude and ChatGPT. The protocol originally excluded Claude as a rater for exactly this circularity; that exclusion was deliberately reversed in round three and is reported as a protocol deviation. Judge the numbers, not the authority of any model — or of the author, who also designed the rule-based detector being tested.

Short answer: This is an ongoing series testing a narrow question: can automated systems apply the six rules of Objective Projection to unseen prose as consistently as a human can? Three rounds have run, covering 220 scenes and five automated labellers. The headline result is that the inference-heavy rules are not reliably labelled by anything tested so far — but the more useful result, and the one this article spends most of its space on, is that the benchmark itself has a design flaw serious enough to limit what any of it proves.

What the series measures — and what it does not

The distinction matters more than any number below. This benchmark measures labelability: whether a rule is defined sharply enough that different raters, given the same definition and the same text, mark the same scenes. It does not measure whether the writing techniques work on readers. That is a separate question requiring a preregistered reader experiment, which has not been run.

So a rule failing here does not mean the technique is useless. It means nobody can yet agree on where it occurs — which is a problem for teaching it, testing it, or building tools around it, but not, by itself, evidence about its effect.

Round 1 — the detector against its designer

120 Turkish scenes, rule-based detector, single human rater (the author).

Results were mixed and were reported as such. The prohibition on simile reached perfect agreement — a rule with a clear surface signature is straightforwardly checkable. The emotion-label rule produced precision of 0.17, meaning most of what the detector flagged was not what a human would flag. Micro focus reached recall around 0.80. Atmospheric contradiction landed at κ 0.26. Materialized metaphor came out at approximately κ 0.00 — chance.

Two honest weaknesses: the only human rater was the detector's own designer, and a criterion shift occurred mid-study (the treatment of emotion statements in dialogue was clarified partway through, and reported openly rather than quietly retrofitted).

Round 2 — an independent human, and two models

100 fresh scenes, zero overlap with round 1, sampled with a fixed seed. Human rater: an independent non-expert, blind to detector output. Labellers: the detector, Gemini 2.5 Flash, Grok.

The human's positive counts across the six rules were: emotion labels 0, simile 1, materialized metaphor 9, micro focus 96, temporal anchor 99, atmospheric contradiction 44.

On materialized metaphor the detector marked 72 scenes positive against the human's 9 — seven true positives, sixty-five false. Gemini found essentially the opposite, catching one of the nine. Grok marked none at all. On atmospheric contradiction, where the human marked 44, the detector caught 0, Gemini 2, Grok 3.

Overall raw agreement with the human: detector 74.7%, Gemini 75.7%, Grok 86.3%. The two models agreed with each other 85.7% — no better than either agreed with a person.

Round 3 — two more models, same scenes, same locked labels

Same 100 scenes, same locked human reference, same prompt blocks. Labellers: Claude Fable 5 and ChatGPT 5.5, fresh sessions.

On materialized metaphor, Claude marked 78 scenes positive and ChatGPT 40, against the human's 9. Claude's κ was 0.03, ChatGPT's 0.02 — both at chance, despite behaving completely differently: Claude caught eight of the nine the human found, at the cost of seventy false positives.

Put the five labellers side by side on that one rule, against a human who marked nine:

Five systems, the same definition, the same hundred scenes, and a range covering almost the entire spectrum. Whatever this rule is measuring, nothing tested measures it the same way twice.

Round 3 also produced a correction to an earlier claim. After round 2 it looked as though atmospheric contradiction was simply invisible to machines — the three labellers had caught 0, 2 and 3 of the human's 44. The newer models did substantially better: Claude marked 55 scenes and caught 31 of the 44 (κ 0.27); ChatGPT marked 42 and caught 23 (κ 0.18). Still weak, but not blindness. The earlier statement was too strong and applied only to the labellers in that round.

Overall raw agreement with the human: ChatGPT 84.5%, Claude 81.0%.

What is wrong with this benchmark

This is the part that matters most, and it argues against the benchmark rather than for it.

Five of six rules have a broken class balance. In the round 2 corpus, the human marked emotion labels in 0 scenes, simile in 1, micro focus in 96, and temporal anchors in 99. When 99 of 100 scenes are positive, a labeller that says "yes" to everything scores 99% and has demonstrated nothing. Only atmospheric contradiction, at 44 positive against 56 negative, was tested with a usable class distribution. Most of the κ values in this series therefore carry very little information, and the high raw-agreement percentages mostly reflect skew.

The temporal anchor rule was never really tested. Measurement showed that 94 of the 100 scenes open with an explicit clock reading in the first thirty characters. That is a property of how the corpus was written, not a difficulty any labeller had to overcome.

One human rater. With a single human, there is no way to separate the two live explanations for the failures: that these rules require inference current systems cannot perform, or that the definitions are not sharp enough for anyone to apply consistently. A second independent human rater would separate them — high human-human agreement points to the first, low agreement to the second.

Further limits. Turkish only. Models were run through web interfaces in blocks rather than independent API calls. The detector's designer is also the benchmark's author. And one accidental run on a different model produced degenerate output — thirty identical lines — which was discarded before analysis rather than being quietly folded in.

What round 4 has to fix

1. A second independent human rater on the same 100 scenes under the same blind protocol. Without this, the series cannot say why the failures happen.
2. A balanced corpus, with each rule represented on both sides. The design warning here is circularity: scenes must not be built to defeat the detector, or the benchmark measures the corpus rather than the rule. Balance has to come from a narrative rationale, not from the measurement target.
3. API-level runs with recorded parameters, replacing block-pasted web sessions.
4. A second language, to separate rule behaviour from Turkish-specific surface features.

Until at least the first two are done, the accurate summary of this series is narrow: on a skewed Turkish corpus, five automated labellers failed to match a single independent human on the inference-heavy rules, and disagreed wildly with each other. Everything beyond that sentence is currently speculation.

All labels, scripts and results from rounds 2 and 3 are public, so that anyone can recompute the numbers or contradict them.

Frequently asked questions

Which model won? None, and the question is malformed. On five of six rules the class distribution is too skewed for a ranking to mean anything. On the one balanced rule the newer models did better than the older ones, but "better" here means weak agreement instead of almost none.

Doesn't this show AI cannot understand narrative? No, and that claim is not made here. The benchmark shows that automated systems cannot apply these particular definitions consistently. It says nothing about understanding, and part of the explanation may be that the definitions are underspecified — which is a criticism of the framework, not of the models.

Why include a model that is also involved in producing the work? Because excluding it would leave a gap in the comparison, and because the deviation can be disclosed. The original protocol excluded it for circularity; round 3 reversed that deliberately and reported the reversal. Readers should weight results accordingly.

Why do the models disagree with each other so much? That is the most informative finding in the series. If a rule were operationally sharp, independent labellers would converge regardless of architecture. Wide divergence between systems is evidence that the definition, not just the labeller, is doing something unstable.

Can I run this myself? Yes — that is the point of publishing the labels, the scenes and the scoring script. Independent replication, including replication that contradicts these results, is the only outcome that would make the series worth anything.

Cite this article

@misc{bulut2026benchmark,
  author = {Bulut, Levent},
  title  = {The AI Benchmark Series: Can Machines Label "Show, Don't Tell"?},
  year   = {2026},
  url    = {https://leventbulut.com/ai-benchmark-series/},
  note   = {Independent Researcher. ORCID: 0009-0007-7500-2261}
}