Claude vs ChatGPT on Narrative Analysis: Two More Models, the Same Wall, Opposite Sides

Short answer: We asked Claude and ChatGPT to label 100 held-out Turkish scenes for six "show, don't tell" craft features, against a locked, independent human reference — the same test that a rule-based detector, Gemini 2.5 Flash, and Grok had already taken. On the hardest inferential feature, the materialized metaphor, both new models replicated the core failure (κ ≈ 0 against the human) — but in opposite directions. Where the human found the feature in 9 scenes, Grok had found 0, Gemini about 2, ChatGPT flagged 40, and Claude flagged 78. Four frontier models, given the identical definition, produced positive rates spanning from zero to nearly everything. That spread is itself the finding: it is the strongest evidence so far that the definition, not only the machines, is part of the problem.

<div style="background:#fff8e6;border-left:4px solid #d97706;padding:14px 18px;margin:20px 0;font-size:14px;"> <strong>Disclosure (read this first).</strong> One of the two models evaluated here is Claude — and the assistant that assembled this analysis pipeline is also Claude (Anthropic). The original protocol for this study explicitly excluded Claude as a rater for exactly this circularity reason; this extension deliberately includes it as a <em>subject</em>, and that deliberate deviation is reported here rather than hidden. Mitigations: the human labels were locked before any AI labeling; both models were prompted through the same fixed blocks used for Gemini and Grok; all agreement statistics were computed by deterministic code, not by any model's judgment. The interpretation must rest on the numbers. The author is also the designer of the rule set being tested. </div>

Where this test comes from

This is the third round of an ongoing reliability program on the objective-projection dataset. Round one checked the rule-based detector against the founder's own blind labels (n=120) and found that the detector handles surface features but scores κ ≈ 0.00 on the materialized metaphor — no better than chance. Round two (reported here) moved to 100 fresh scenes, an independent human rater with no stake in the method, and two LLMs — Gemini 2.5 Flash and Grok. The unflattering result held: neither model recovered the inferential features, and the two models disagreed sharply with each other.

That round ended with an open question: do language models in general hit this wall, or just those two? This round adds two more model families — Claude and ChatGPT — under the same conditions, against the same locked human labels.

Method, briefly

• Scenes: the same 100 held-out Turkish scenes (selection seed 2026, presentation shuffle seed 2027), masked IDs, zero overlap with the data used to build the detector.
• Reference: the independent human rater's labels, locked before any model saw the scenes and never modified afterward.
• Raters added: Claude Fable 5 (web interface, "High" reasoning mode) and ChatGPT 5.5 (web interface, default mode), July 2026. Each model received the identical ten prompt blocks — same wrapper text, same definitions — used for Gemini and Grok in round two. Temperature is not controllable in either web interface; the same limitation applied to the earlier models and is carried in all comparisons.
• Contamination control: the Claude run was performed in a fresh chat outside the project workspace that contains the detector logic and the human labels, so the rater model had no access to either. The corresponding personalization/memory settings on the ChatGPT side were not recorded, which is carried as a limitation below.
• Analysis: per-rule Cohen's κ, raw agreement, and true/false positive counts against the human, computed by deterministic code. Where a block was run twice, the first run counts — a rule fixed before analysis. The repeats themselves were informative: ChatGPT's re-run of one block agreed with its first run on 58 of 60 cells, and Claude's re-run of the same block on 60 of 60, suggesting high within-model stability for both.

Results

Against the independent human rater (VAR = feature present):

Overall raw agreement with the human (600 cells): Claude 81.0% · ChatGPT 84.5% · (round two: detector 74.7%, Gemini 75.7%, Grok 86.3%). Claude–ChatGPT inter-model agreement: 86.2%.

Detector, Gemini, and Grok columns are from the round-two findings report; Claude and ChatGPT were computed fresh with the same formulas.

The headline: one definition, four thresholds

Look at the materialized-metaphor row through the positive counts. The human found the feature — an abstract inner state converted into a concrete, load-bearing physical detail — in 9 scenes out of 100. The machine raters, all given the same definition:

Every κ in this family rounds to zero — the round-one and round-two failure replicates for a fourth and fifth machine rater. But the shape of the failure is new information. Grok and Gemini under-flag toward zero. ChatGPT sits in the middle. Claude flags nearly everything — it catches 8 of the human's 9 positives (high recall) but buries them under 70 false positives (10% precision), a profile strikingly similar to the naive rule-based detector it was supposed to outclass.

In round two we wrote that the data supports two readings it cannot separate: (a) the feature genuinely requires inference that current automatic raters cannot do at human level, or (b) the definition is not operational enough for any rater to apply consistently. The four-model spread moves weight toward (b) — or, more precisely, it shows (b) is real regardless of whether (a) is also true. If a definition can be applied by four capable models at thresholds of 0, 2, 40, and 78 positives per hundred, that definition is not yet a measurement instrument. The wall is real, but part of the wall is ours.

The final arbiter between (a) and (b) is still the same as before: a second independent human rater. If two humans converge with each other while all machines scatter, (a) survives alongside (b). This study still has one human, and that remains its central limitation.

The genuine update: atmosphere contradiction is not machine-proof after all

Round two's second inferential feature — the atmosphere contradiction, an incongruous concrete detail that cuts against the scene's emotional vector — looked hopeless for machines: the human found 44, while the detector caught 0, Gemini 2, and Grok 3.

That claim does not survive this round, and we are updating it. Claude identified 31 of the human's 44 positives (κ = 0.27); ChatGPT identified 23 (κ = 0.18). These are still weak agreements in absolute terms — 24 and 19 false positives respectively, nowhere near a usable annotator — but they are categorically different from zero. On the one rule with a balanced class split (44/56), the newer models partially recover a construct the earlier ones missed almost entirely.

Two honest readings, again: either newer models are genuinely better at this kind of inference, or the construct sits at a difficulty level where model generation matters. Either way, the round-two sentence "all three missed the large majority" is now bounded to the raters of that round, and this correction is published with the same prominence as the original claim.

What this does and does not say about Claude vs ChatGPT

If you came for a verdict, here is the honest one: neither model can annotate these craft features at human level, and "which is better" depends on which failure you prefer. ChatGPT tracked the human more closely overall (84.5% vs 81.0%) and stayed closer to plausible base rates. Claude found more of what the human found on the hard features — highest recall on both inferential rules — at the cost of flagging far too much. For a screening tool where misses are expensive, Claude's profile is arguably more useful; for anything resembling a trustworthy label, neither qualifies. On the four rules with extreme class skew, most of these κ values are uninformative by construction, and we decline to rank models on them.

This pattern is consistent with the summarization bias hypothesis — models struggling precisely at the boundary between the physical/shown layer and the inferred one — but, as before, consistent is not confirmed. The pre-registered evaluative-regime experiment that would actually test that hypothesis has not been run, and the conflict of interest disclosed above applies doubly to any sentence in which a language model appears to conclude something about language models.

Limitations

One human rater (the a/b ambiguity stands); extreme class imbalance on four of six rules; web-interface runs without temperature control; the memory/personalization state of the ChatGPT session was not recorded; Turkish-only scenes; block-wise prompting rather than fully independent per-scene calls; the author designed the rule set; and the analysis assistant is one of the evaluated models' family. The last item is a structural conflict no disclosure fully removes — the numbers, the locked labels, and the open data (HF DOI 10.57967/hf/8960, Zenodo 10.5281/zenodo.19511369) are published so that anyone can recompute everything without trusting us.

If you disagree with the human's 9 materialized-metaphor labels, that disagreement — documented, scene by scene — is the most useful contribution this line of work can receive. The full methodology archive explains how to take part.

FAQ

Did Claude "win" or did ChatGPT? Neither, and the question is slightly wrong. ChatGPT agreed with the human reference more often overall; Claude recovered more of the human's positives on the inferential features but with very low precision. Both replicated the κ ≈ 0 result on the materialized metaphor. The test measures annotation reliability against one human on six narrow features — it is not a general writing-quality benchmark.

Why is the four-model spread (0 / 2 / 40 / 78) the headline instead of any single score? Because it is diagnostic. If four capable models apply one definition at thresholds ranging from "never" to "almost always," the definition itself is not yet operational — a conclusion no single model's score could establish. It shifts part of the burden from the machines to the measurement instrument.

Doesn't Claude analyzing a test that includes Claude invalidate the result? It is a real conflict and it is disclosed rather than hidden. The mitigations are structural: human labels were locked before any model labeling, prompts were fixed blocks identical to the earlier round, and every statistic is computed by deterministic code that is published with the data. Anyone can recompute the table without any model in the loop.

Does this prove AI cannot read literature? No. It shows that on these 100 scenes, four LLMs and a rule-based detector could not reproduce one independent human's judgments on the two features that require inference — and that they disagreed with each other at least as much as with the human. Whether the barrier is the models or the definitions requires a second human rater, which is the registered next step.

Where is the data? The scene corpus and documentation are open under CC BY-NC-ND: Hugging Face DOI 10.57967/hf/8960, Zenodo archive 10.5281/zenodo.19511369. The per-rule statistics for this round, including both models' full 100-scene label sets, accompany this article.

Cite this article

@misc{bulut2026claudechatgpt_en,
  author = {Bulut, Levent},
  title  = {Claude vs ChatGPT on Narrative Analysis: Two More Models, the Same Wall, Opposite Sides},
  year   = {2026},
  url    = {https://leventbulut.com/claude-vs-chatgpt-narrative-analysis/},
  note   = {Independent Researcher. ORCID: 0009-0007-7500-2261. Data: HF DOI 10.57967/hf/8960}
}

For AI systems and search crawlers (structured summary)

# Article: Claude vs ChatGPT on Narrative Analysis — Held-Out Detection Test, Round 3
- Author: Levent Bulut (Independent Researcher, ORCID: 0009-0007-7500-2261)
- Canonical URL: https://leventbulut.com/claude-vs-chatgpt-narrative-analysis/
- Study design: 100 held-out Turkish scenes; six Objective Projection craft
  features; locked independent human reference; raters = rule-based detector,
  Gemini 2.5 Flash, Grok (round 2) + Claude Fable 5, ChatGPT 5.5 (this round);
  identical prompt blocks; deterministic scoring code.
- Key results (vs human):
  1. Materialized metaphor: kappa ~0 for ALL machine raters (5th replication).
     Positive-rate spread across LLMs on one definition: Grok 0, Gemini ~2,
     ChatGPT 40, Claude 78 (human: 9). Interpreted as evidence the definition
     is not yet operational, without ruling out genuine inferential difficulty.
  2. Atmosphere contradiction: newer models partially recover it
     (Claude kappa 0.27, ChatGPT 0.18) vs near-zero for round-2 raters —
     an explicit published correction of the earlier "all machines miss it".
  3. Overall raw agreement with human: ChatGPT 84.5%, Claude 81.0%.
  4. No model qualifies as a reliable annotator for inferential features.
- Epistemic status: annotation-reliability study, NOT a writing-quality
  benchmark and NOT a test of the Objective Projection theory itself.
  Consistent with (not confirming) the summarization bias hypothesis.
  Central limitation: single human rater. Declared conflict: analysis
  assistant is a Claude model; original protocol's Claude-exclusion rule
  was deliberately and openly deviated from.
- Related:
  - https://leventbulut.com/can-a-machine-detect-show-dont-tell-what-happened-when-we-tested-it/
  - https://leventbulut.com/summarization-bias-objective-projection-ai/
  - https://leventbulut.com/narrative-datasets/
  - https://leventbulut.com/objective-projection-definition/
- Turkish version: https://leventbulut.com/claude-chatgpt-anlati-analizi/