How Reliable Are LLM Annotations? A Three-Study Benchmark

Would a human agree with your dataset's machine labels?

Published corpora increasingly ship with machine-generated annotation layers: sentiment scores, stylistic flags, discourse categories, applied by a heuristic or a language model and distributed alongside the text. Downstream users inherit them as ground truth. The question that is rarely put to the test is whether an independent human would agree with them at all.

This report answers that question for a corpus I publish myself. The Objective Projection dataset carries a per-scene applied_rules field, generated by a bilingual rule-based detector over six craft features. Those flags had been used to describe the corpus and to select examples. Nobody had checked them.

I have now checked them three times. The full paper is at 10.5281/zenodo.21740239. This page summarises what it found.

An earlier report covered the second study alone: Can a machine detect "show, don't tell"? Two of its conclusions are corrected below.

1. The six features

The scheme comes from the Objective Projection methodology and distinguishes two prohibitions from four positive techniques.

• Emotion embargo (prohibition) — the narrator's voice must not name an emotion directly. Physical action a reader may interpret emotionally is not a violation; dialogue is exempt.
• Simile prohibition (prohibition) — no overt simile markers (Turkish gibi, sanki, adeta).
• Materialized metaphor — an abstract inner state rendered as a concrete, measurable physical detail instead of being named.
• Micro-focus — narration concentrated on a narrow, small, concrete object or detail.
• Temporal anchor — a concrete time, duration or measurement.
• Atmosphere contradiction — an unexpected detail running against the scene's prevailing situation.

The first two are surface features that a string search can approximate. The last four require judgement, and materialized metaphor requires the most: it asks whether a physical detail is doing representational work, which is not a property of the string.

2. Three studies

The Study 1 and Study 2 scene sets are disjoint — zero overlap. The Study 2 scenes were drawn from a 180-scene pool untouched by Study 1, then shuffled and stripped of their identifiers. The independent rater's labels were locked before any machine ran.

Two procedural rules were fixed in advance. First-run-counts: the first response to each prompt block is the datum. Degenerate-output rejection: an output showing a mechanical repeating pattern is discarded rather than scored — one model, Gemini 3.6, returned thirty identical label rows and was rejected under this rule, so it does not appear in the results. Block re-runs measured within-model stability: ChatGPT reproduced 58 of 60 re-run labels, Claude 60 of 60.

A note on the word held-out: the 100 scenes are held out from rater and detector development, not from the corpus. They are corpus scenes, and the identifier mapping is published. The term is not being used to imply more separation than exists.

3. The finding: one rule, five machines, five different answers

Against a locked human count of 9 positives out of 100, the five machine labellers returned:

In Study 1, against a different human reference on a different set of scenes, the detector's kappa on the same rule was 0.004. That makes five near-zero results, obtained from different labellers against different humans on different scenes.

Gemini's 0.185 is the single exception, and it rests on one positive judgement out of a hundred.

The spread covers essentially the entire available range. This is not a ranking of accuracy. A definition that yields 0 from one competent reader and 78 from another is not yet functioning as a definition.

The models disagree with each other, not only with the human

This is the part I find hardest to explain away. On micro-focus, a rule that asks only whether the narration concentrates on a small concrete detail, Gemini marked 9 scenes and Grok marked 82 — against a human count of 96. Same written definition, same hundred scenes, an order of magnitude apart.

If the constructs were crisp, independent competent raters would converge. They did not converge with the human and they did not converge with each other.

Why the raw agreement figures are misleading

ChatGPT posted the highest overall raw agreement of any system in any of the three studies — 84.5% — while its kappa on the inference rule was 0.019. Claude reached 81.0%; Grok reached 86.3%, achieved partly by saying "absent" to every scene on two rules.

Both facts are compatible because five of the six rules have severely lopsided human distributions. In Study 2 the human's positive counts were 0, 1, 9, 96, 99 and 44 out of 100. When one rater marks 99 of 100 scenes positive, agreement statistics have no variance to work with; kappa carries almost no information and high raw agreement reflects the imbalance rather than any skill.

Four of the six rules were, in effect, not tested at all. This is a defect of my evaluation set, not of the systems evaluated. A corpus constructed so that nearly every scene has a temporal anchor cannot be used to measure whether anything detects temporal anchors. Only atmosphere contradiction (44 positives, 56 negatives) sits in a usable distribution — which is exactly why it is the only rule where I am willing to say two models did better than chance.

4. A correction to the earlier report

The earlier write-up concluded that the machines missed atmosphere contradiction almost entirely. That was true of the three Study 2 raters. It does not generalise.

Against the human's 44 positives, Claude marked 55 scenes with 31 correct (kappa 0.269) and ChatGPT marked 42 with 23 correct (kappa 0.184). Both are far from reliable agreement. Both are clearly above chance, on the one rule whose class distribution can support that judgement. Newer models recovered partial signal on a feature I had written off.

A second, smaller correction: that report gave Grok's atmosphere positives as 3. Grok marked 6 scenes, 3 of them correctly.

5. Two readings, and why neither is chosen

Reading (a) — the feature is genuinely inferential. Deciding whether a physical detail carries an unstated inner state requires reconstructing what the text withholds. If that is a reader-side act rather than a text-side property, no surface detector should find it, and models that approach the task by pattern rather than by reconstruction should not either. Under this reading the low agreement is a finding about machines.

Reading (b) — the definition is not operational. In a corpus built entirely from the instruction "render states physically", nearly every scene qualifies under a permissive reading and few under a strict one. The human's 9-out-of-100 is itself a strict-reading artefact. And in Study 1 my own criterion for this rule drifted mid-pass: I had to re-label the entire rule in a separate corrected pass because earlier and later scenes were being judged against different implicit standards. That is direct evidence that the criterion is unstable even within a single rater — evidence against my own rule, recorded in the published label file.

The two readings make one distinguishing prediction. Under (a), two independent human raters should agree with each other substantially better than any machine agrees with either. Under (b), two independent humans should also disagree.

I have one human rater per study, so I do not decide. A reader who wants to use the corpus's applied_rules field should note that under both readings the field is unreliable for inferential features; the readings differ on where the fault lies, not on the practical conclusion.

Worth adding: if two humans do disagree, that would not automatically condemn the rule. There is a serious line of argument in the annotation literature that human label variation on interpretive tasks is genuine signal about the task rather than noise to be minimised. Distinguishing an unstable definition from a legitimately variable judgement would then become the next question rather than the closing one.

6. What had to be reconstructed

Two provenance caveats affect what a reader can independently verify, and they are stated here rather than buried in the paper.

The Gemini and Grok per-scene label files were lost. Their per-rule kappa and agreement values survived. That turns out to be sufficient: for a binary rule over 100 scenes with a known human positive count and a known agreement count, the entire confusion table is a function of a single unknown, and kappa fixes it. Enumeration returned a unique solution for eleven of the twelve rule-by-labeller cells; the twelfth was settled by a sentence in the original write-up. Those counts are derived, not raw. Scene-level quantities — including the reported 85.7% Gemini–Grok agreement — cannot be recovered this way.

The detector's Study 2 labels were reconstructed from the applied_rules field already published in the corpus, mapped through the published identifier mapping. The reconstruction reproduces the original Study 2 detector column exactly. It is still a reconstruction.

Both derivation scripts are published, so the arithmetic can be checked rather than trusted.

7. What this means if you use annotated corpora

• Do not treat shipped machine annotations as ground truth. Unvalidated applied_rules-style fields are heuristic markers. Validate on a task-by-task basis against human labels; general model capability does not transfer.
• Report class distributions alongside agreement. A kappa reported without prevalence and bias indices can mislead. High raw agreement on an imbalanced feature is scientifically uninformative.
• Record the conditions. Exact model version, mode, interface, session state, date. All runs here used public web interfaces in fresh sessions without project context or memory, so temperature and system-prompt conditions were not controlled — a limitation, and one that has to be stated rather than implied.

8. What would settle it

1. A second independent human rater on the same 100 scenes, blind to everything above. This is the one measurement that separates readings (a) and (b). No further model run substitutes for it.
2. A balanced evaluation set for the four working rules — scenes sampled so each feature is present in roughly half. Without this, four of six rules remain untestable no matter how many systems are run.
3. An operational rewrite of the materialized-metaphor rule, with worked positive and negative examples at the boundary, followed by re-labelling.

Everything needed for step 1 is open: the 100 scene texts, the six definitions, the locked human labels, every model's label file, the ten prompt blocks, the identifier mapping and the scoring scripts, in the evaluation/ directory of the dataset. If you label those scenes and disagree with my volunteer, that disagreement is more useful to this work than any additional model I could run.

Data and further reading

• Paper (all three studies): 10.5281/zenodo.21740239
• Dataset: Objective Projection Dataset v7.2 — DOI 10.57967/hf/8960, Zenodo archive 10.5281/zenodo.19511369
• Earlier report (Study 2 alone): Can a machine detect "show, don't tell"?
• Summarization Bias — a conjecture that language models substitute abstract summary labels for a shown layer: 10.5281/zenodo.20783465. These data are consistent with it but do not test it; the spread here runs in both directions at once, and its registered experiment has not been run.
• Glossary: terms used across this work

Frequently asked questions

What is the central finding?

On materialized metaphor — the rule requiring the most inference — five machine labellers marked between 0 and 78 of 100 scenes positive, against a human count of 9. Cohen's kappa was at or indistinguishable from chance for five of the six labellers, across two human references and two disjoint scene sets.

Why was raw agreement high when kappa was near zero?

Class imbalance. When a feature is present in 96 or 99 of 100 scenes, a rater can reach very high raw agreement by predicting the majority class every time, while its chance-corrected agreement stays at zero. Five of the six rules in this evaluation have that shape, so raw agreement mostly measures a rater's willingness to output the dominant class.

Which systems were evaluated?

Five machine labellers: a rule-based bilingual heuristic detector, Gemini 2.5 Flash, Grok, ChatGPT 5.5, and Claude Fable 5 (High). A sixth, Gemini 3.6, was rejected before scoring for producing degenerate repeated output.

Can language models detect atmospheric contradiction or narrative subtext?

Partially, on this evidence. Claude Fable 5 reached kappa 0.269 and ChatGPT 5.5 reached 0.184 on atmosphere contradiction — above chance, well short of reliable agreement, and measured on the only rule in this set whose class distribution supports the judgement. Earlier models in Study 2 recovered almost no signal on the same rule.

Does this show that machines cannot understand storytelling?

No, and the report explicitly declines that conclusion. Two readings survive the data — that the features genuinely require inference beyond current automatic raters, or that the definitions are not operational enough for any rater to apply consistently. Separating them requires a second independent human rater, which this work does not have.

How can the results be reproduced?

All locked human reference labels, model label files, reconstructed detector labels, the ten prompt blocks, the identifier mapping and the Python scoring and derivation scripts are published under CC BY-NC-ND 4.0 in the evaluation/ directory of the Objective Projection dataset on Hugging Face.

Citation

@misc{bulut2026reliability,
  author       = {Bulut, Levent},
  title        = {Inter-Rater Reliability of LLM and Rule-Based Annotation for
                  Inferential Narrative Features: Three Studies on a Turkish Corpus},
  year         = {2026},
  month        = {August},
  howpublished = {Zenodo},
  doi          = {10.5281/zenodo.21740239},
  url          = {https://doi.org/10.5281/zenodo.21740239},
  note         = {Version 1.0. Independent Researcher.
                  ORCID: 0009-0007-7500-2261}
}

Levent Bulut — Independent Researcher. ORCID 0009-0007-7500-2261. Data and code released under CC BY-NC-ND 4.0.