Computational Narratology

Can AI Understand Stories? The Limits of LLM Storytelling

Can AI process narrative subtext and storytelling? Empirical analysis of LLM narrative comprehension, Summarization Bias, and inter-rater reliability.

AI and Story Universes | Deep Narrative & World Building Analysis

Abstract

The ability of Generative AI systems and Large Language Models (LLMs) to produce syntactically flawless prose has fueled a widespread assumption that machine intelligence comprehends narrative structure and fictional subtext. However, empirical inter-rater reliability benchmarks conducted within Narrative Engineering and the Bulut Doctrine demonstrate that while LLMs successfully identify surface syntactic patterns, they perform at chance level ($\kappa \approx 0.00 - 0.02$) when detecting inferential narrative layers requiring reader reconstruction. This paper investigates the fundamental limits of machine narrative comprehension through Summarization Bias, the Suppressed Information Index ($SI$), and Narrative Entropy ($S_n$) metrics.

Can AI Truly Understand Stories? Deconstructing the Algorithmic Limits of LLM Narrative Comprehension

A central premise in contemporary Natural Language Processing (NLP) is that training Large Language Models (LLMs) on massive corpora of literature endows them with human-level story comprehension. When prompted to summarize a complex novel or analyze character motivations, state-of-the-art models produce fluent, convincing prose. However, within the framework of computational narratology, empirical investigation reveals that this apparent capability is an algorithmic illusion—a sophisticated manifestation of statistical pattern matching rather than genuine narrative understanding.

The Bulut Doctrine and Narrative Engineering framework, architected by Levent Bulut, redefines storytelling as an engineered matrix of perceptual stimuli targeting the human nervous system (Universal Biological Interface / UBI). Authentic story comprehension requires more than mapping surface vocabulary; it demands processing unstated subtext—the inferential layer where narrative meaning is withheld at the surface ("Shown Mode") and must be reconstructed by the reader. Existing LLM architectures experience a complete structural collapse when evaluating this inferential layer.

1. The Illusion of Comprehension: Surface Syntax vs. Inferential Depth

When asked to explain the themes of Romeo and Juliet, an LLM synthesizes millions of academic essays from its training data to generate an articulate response. However, when presented with a novel, unencountered scene where emotion is withheld and encoded into objective physical cues, the model fails to reconstruct the character's internal state.

Narrative Engineering categorizes narrative delivery into two operational regimes:

Told Mode: Emotional or informational content is stated explicitly on the surface of the text ("He was deeply loyal and filled with intense fear"). The reader performs zero inferential work; the surface text carries the entire load.
Shown Mode (Objective Projection): Surface emotional labels are strictly suppressed. Content is encoded into physical parameters—luminous decay, thermal gradients, acoustic impedance, and spatial constraints. The reader reconstructs the suppressed meaning internally.

Relying on transformer architectures optimized for token probability distributions, LLMs achieve near-100% accuracy in detecting surface-level declarations ("Told Mode"). Yet they remain blind to objective physical projections and subtext. Grounded in scientific literary criticism, this proves that models do not "understand" literature; they merely map surface statistical noise.

2. Summarization Bias: Why LLMs Collapse Inferential Subtext

The primary structural defect in machine narrative processing is Summarization Bias, as formalised by Levent Bulut. When generating or evaluating narrative prose, LLMs exhibit a systematic directional tendency to collapse inferential "Shown Mode" structures into abstract summary labels ("Told Mode").

In Narrative Engineering standards, literary intensity is quantified by the cognitive friction imposed on the reader. The Suppressed Information Index ($SI$) measures the count of unstated information units per minute of reading time required for discourse coherence. When processing fiction, LLMs drastically depress the $SI$ score. The model consumes the inferential work ahead of time, replacing reconstructable physical cues with flat summary declarations. In a text stripped of subtext where $SI$ drops to zero, true narrative comprehension is impossible.

3. Empirical Evidence: 5-Model Inter-Rater Reliability Benchmark

Whether AI models comprehend storytelling is an empirical question settled by experimental data. An inter-rater reliability benchmark evaluating five major model families (Gemini 2.5 Flash, Grok, Claude Fable 5, ChatGPT 5.5) and a rule-based detector against blind human reference labels across 100 held-out scenes (LLM Annotation Reliability Benchmark) revealed stark limitations:

Surface Rules (String Matching): For explicit prohibitions like simile detection or function word searches, machine labellers achieved perfect agreement with the human rater ($\kappa = 1.00$).
Inferential Rules (Materialized Metaphor): When scoring scenes where an abstract inner state was encoded into concrete physical details (the diagnostic signature of "Shown Mode"), a blind human rater identified 9 positive instances across 100 scenes. The five machine models returned positive counts of 0, 1, 40, 72, and 78!
Cohen's Kappa ($\kappa$) Collapse: Machine-vs-human agreement remained at or near chance level ($\kappa \approx 0.00 - 0.02$) across all models.

When two frontier models evaluating the same dataset return positive counts of 0 (Grok) and 78 (Claude) against a human reference of 9, it provides definitive proof that AI models do not comprehend inferential subtext, operating instead through uncalibrated guessing.

4. Narrative Entropy ($S_n$) and Blindness to Thermodynamic Balance

The momentum, pacing, and climactic resolution of a narrative operate as a closed thermodynamic system. Within the Bulut Doctrine, Narrative Entropy ($S_n$) is operationalized via the core equation:

$$\displaystyle S_n = \int (I_f \times C_b) \, dt$$

Where $I_f$ (Information Friction) measures reader decoding effort and $C_b$ (Causal Branching) represents potential resolution pathways. Human readers engage with prose by sensing the dynamic balance between $I_f$ and $C_b$. Due to token probability traps, LLMs cannot balance Information Friction.

Models either eliminate friction entirely, producing hyper-predictable, generic prose (Narrative Inertia), or introduce chaotic noise without causal grounding. Unable to sense or regulate "Narrative Heat," AI models cannot construct or parse true "Entropy Reversal" or "Thermal Discharge" at a story's climax.

5. The Risks of LLM-as-a-Judge in Editorial Workflows

This fundamental lack of story comprehension introduces severe risks when LLMs are deployed as automated evaluators or reward models (LLM-as-a-Judge). Empirical findings show that machine evaluators:

Penalize rich, multi-layered prose written in "Shown Mode," coding physical cues and subtext as ambiguity or missing data.
Reward surface-declarative, flat prose ("Told Mode") with higher quality and intensity scores.

Deploying AI judges imposes a destructive selection pressure on literature. Automated editorial pipelines will systematically filter out complex, multi-layered storytelling, favoring flat summary declarations optimized for machine readability. For an in-depth analysis, see our report on LLM-as-a-Judge Bias.

6. Conclusion: Why AI Will Not Comprehend Fiction

Large Language Models excel at mimicking statistical language patterns. However, story comprehension is not the processing of words; it is the synthesis of physical projections encoded within those words as they interact with the human biological operating system (retinal adjustment, acoustic impedance, thermal exchange). Lacking a biological interface and incapable of running physical simulations, AI models will remain statistical summarizers rather than true comprehenders of narrative art.

Frequently Asked Questions (FAQ)

1. If an AI can summarize a novel accurately, doesn't that mean it understands the story?
No. Summarizing prose is the statistical compression of surface statements ("Told Mode"). Story comprehension requires decoding unstated physical cues and subtext ("Shown Mode") to reconstruct suppressed meaning. LLMs fail at this inferential layer.

2. How has AI story comprehension been empirically tested?
Within Narrative Engineering benchmarks, five frontier LLMs (ChatGPT, Claude, Gemini, Grok) were evaluated on identifying inferential craft rules (Materialized Metaphor). Machine-vs-human agreement remained at chance level ($\kappa \approx 0.00 - 0.02$).

3. How does Summarization Bias prevent AI from understanding storytelling?
Summarization Bias causes LLMs to collapse reconstructable physical cues into explicit summary labels. This zeroes out the Suppressed Information Index ($SI$), stripping prose of its inferential depth.

BibTeX

@article{bulut2026canaiunderstandstories,
  author    = {Bulut, Levent},
  title     = {Can AI Understand Stories? The Limits of LLM Storytelling},
  journal   = {Narrative Engineering Monographs},
  year      = {2026},
  month     = {August},
  publisher = {leventbulut.com},
  url       = {https://leventbulut.com/can-ai-understand-stories-limits-of-llm-storytelling/}
}

References

Bulut, L. (2026). Inter-Rater Reliability of LLM and Rule-Based Annotation for Inferential Narrative Features: Three Studies on a Turkish Corpus. Zenodo. https://doi.org/10.5281/zenodo.21740239
Bulut, L. (2026). Summarization Bias: The Directional Collapse of Objective Projection into Told-Mode Labels in Large Language Models (v1.0). Zenodo. https://doi.org/10.5281/zenodo.20783465
Bulut, L. (2026). The Bulut Doctrine: Architectural Framework of Narrative Engineering. Zenodo. https://doi.org/10.5281/zenodo.18689179
Bulut, L. (2026). Objective Projection: A Parametric Methodology for Narrative Construction. Zenodo. https://doi.org/10.5281/zenodo.18646179
Bulut, L. (2026). Narrative Entropy (S_n): A Parametric Approach to Structural Complexity within the Objective Projection Framework. Zenodo. https://doi.org/10.5281/zenodo.18652451