Objective Projection Research Agenda: 13 Designs That Invite Refutation

The Objective Projection research agenda is a working paper that gathers, in one citable document, 13 study designs prepared so that people other than the author can test the method's claims. It was published on Zenodo on 10 November 2026 under a CC BY 4.0 licence (doi:10.5281/zenodo.24115699). The paper contains no new data. Its aim is not to confirm the method but to make it as easy as possible for others to find where the method may be wrong.

This article summarises what the paper contains and makes a call for its first design: can someone who reads Turkish, and who had no part in preparing the dataset, label 100 short scenes independently?

Why a research agenda?

Objective Projection has a public dataset, a reliability study, a pilot study and a conceptual paper on a directional bias in language models. It has no independent replication. Every reliability comparison so far rests on a single human reference, and none of the reader-response hypotheses associated with the method has been tested.

Most of the articles published on this site in September and October 2026 applied the method to a narrow question and ended with a proposed test: screenplay action lines, the situation models readers build, space and sound in Severance, fair play in detective fiction, physical cues in translation. Those proposals were written for general readers, scattered across pages and not citable as research designs. The agenda paper gathered them and rewrote each in a form someone else could preregister.

What does the existing data show, and what does it not?

The dataset contains 500 scene pairs (300 Turkish, 200 English). Of its six features, two are prohibitions (no named emotion, no simile) and four are techniques: materialized metaphor, micro-focus, temporal anchor and atmosphere contradiction.

In the second test of the reliability study (n=100), an independent and blind human annotator labelled the scenes without seeing any machine output; four large language models and a rule-based detector were compared with these locked labels. The human marked emotion label in 0 scenes, simile in 1, materialized metaphor in 9, micro-focus in 96, temporal anchor in 99 and atmosphere contradiction in 44. On materialized metaphor, the kappa values of the machine annotators stayed between 0.000 and 0.185. The highest value on atmosphere contradiction was κ 0.269, and on micro-focus κ 0.296. The confusion matrices of two machines are derived from published statistics, not raw.

Interpretation: The two prohibitions can largely be traced to a surface pattern; the four techniques require inference and cannot yet be labelled reliably. But every study had a single human reference. With the current data it cannot be told whether low agreement comes from the feature itself, from a loose definition, or from that one reader's reading. The article Inter-annotator agreement: what one annotator can't tell you explains this problem in detail.

One scene, two readings

To show why the problem cannot be solved with machines alone, here is a short scene written for this article (it is not in the dataset):

When Kemal opened the door, there were two cups on the kitchen table. Coffee grounds had dried at the bottom of one; the other had never been used. Kemal put the unused one back on the shelf, left the one with the grounds in the sink and did not turn on the tap.

One reader may see a materialized metaphor here: the unused cup turns someone who did not come into a concrete object. Another reader may see only a kitchen detail and leave the feature unmarked. Both readings are defensible. If a definition does not lead two careful people to the same answer, the fact that machines fail to match one of those people says nothing about the machines on its own. That is why a second independent human is needed.

The thirteen designs at a glance

Sample sizes are for 80% power under the stated assumption. No effect size rests on pilot data; the tables are sensitivity analyses, not forecasts.

D8 is the rewritten protocol of the Summarization Bias hypothesis described in what gets lost when AI agents summarize for each other. D10 grew out of the limits to the Emotion Embargo accepted on this site article by article; the cases in horror where telling is legitimate was the first of them. The flagging tool in D12 is the site's Scene Checker; it is not used to score outcomes, because it generates the flags.

Four rules that hold in every design

• Preregistration first. Every design is registered with its target effect and primary analysis before any data are collected (for example on OSF).
• Results are published whatever they are. Null and negative results are published in the same place and in the same form as positive ones.
• The author does not annotate. In no design does the author act as an annotator, a rater or the person who selects materials after random assignment.
• Effect sizes are assumptions. There are no pilot data; the power tables are not forecasts.

The first step: a call for a second independent annotator (D1)

D1 is the cheapest design in the agenda and the one whose result would change the most. It needs one person and a set of scenes that already exists. With the definitions unchanged, the same 100 scenes are labelled by a second independent human. If two humans also agree at chance level on materialized metaphor, the problem most likely lies in the definition; if the humans agree and the machines do not, the feature may require an inference the machines do not make.

Who can take part? Anyone who reads Turkish fluently and had no part in writing or labelling the dataset. The participant does not open the locked label file before the comparison step and uses no AI system to label, translate or summarise the scenes.

How long does it take? 3–5 hours in total; it can be split across several sessions.

How does it work?

1. Pass A: for each scene you first write one sentence on what you noticed, then mark five features as VAR (present) or YOK (absent) and give each label a confidence rating from 1 to 3.
2. You take a break of at least 24 hours.
3. Pass B: you go through the 100 scenes again for materialized metaphor only. The first annotator also labelled this feature in a separate pass.
4. You post the form's hash (SHA-256), with the date, in the relevant thread on the Hugging Face forum. The comparison script will not run without this hash.
5. You run the comparison yourself and share the result and the form in the same thread. You choose how you are credited: by name, by username or anonymously.

All materials are in the dataset's Hugging Face repository, in the evaluation/second_annotator folder: the 100 scene texts, the unchanged definitions, a blank form, guides in Turkish and English, and an analysis script that runs on the Python standard library alone. The script needs no internet connection.

The analysis is fixed in advance. The primary features are materialized metaphor and atmosphere contradiction. The other four are reported descriptively only, because of their extreme prevalence. Kappa is reported with a percentile bootstrap interval. No threshold decides success; the result is published whatever it is.

No payment is offered. The dataset's author is an independent researcher with no institutional or grant funding. Participation is voluntary.

What the doctrine cannot say here

• The agenda cannot say that Objective Projection is correct. It only lists how that could be tested.
• It cannot say whether the assumed effect sizes are realistic; none rests on pilot data.
• It cannot remove the influence of the author having written the designs. Independent researchers may rebuild any of them.
• No design measures the reader's body, brain activity or the reader's own emotion. The Universal Biological Interface (UBI) hypothesis cannot be tested directly by this agenda.
• Even if two humans agree well in D1, this says nothing about the method's effect on readers; it only shows that the definition works consistently for two readers.

Limitations

• The agenda paper has not been published in a peer-reviewed journal; it is a working paper on Zenodo and contains no new data.
• Blinding in D1 rests on a declaration. The first annotator's locked labels sit in the same public repository. The hash step shows that the form did not change after it was posted; it does not show that the locked file was never opened.
• Some of the first annotator's totals (for example 44 of 100 for atmosphere) appear in the published paper and in this article. This is a weaker leak than knowing the labels themselves, but it is still a leak.
• The first annotator is not an expert. Agreement with them is agreement with one reader, not with a gold standard.
• The scenes were written inside the project; they are short and stylised. Results may not carry over to published fiction.
• The site uses more than one definition of materialized metaphor. D1 uses the version 3.1 definition given to the first annotator, unchanged.
• Earlier records of the method are listed in Levent Bulut's knowledge graph and on the Levent Bulut corpus page. The agenda treats none of these records as confirmed.

What the site's free text tools can and cannot do is set out separately on the Objective Projection Tools page.

Frequently Asked Questions

Does the Objective Projection research agenda contain a new finding?

No. The paper reports no new data. It gathers, in one citable document, 13 study designs that would test the method or its limits, each with a hypothesis, a primary outcome, a power or precision analysis and threats to validity. None of the designs has been run.

Why is a second independent annotator the first step?

Because the reliability study had a single human reference. On materialized metaphor the machines agreed with that human at chance level; with one reference it cannot be told whether this comes from the machines failing at an inference or from a loose definition. A second human is the cheapest test that can separate these two readings.

What does it take to be the second annotator, and is it paid?

Reading Turkish fluently, having had no part in writing or labelling the dataset, not opening the locked labels before the comparison step and not using AI are enough. The total time is 3–5 hours. No payment is offered; participation is voluntary and the participant chooses how to be credited.

What does the SHA-256 hash do, and what does it not show?

Before the comparison, the annotator posts the hash of the completed form on the forum with the date. If the form is changed after it was posted, this can be seen. The hash does not show that the locked label file was never opened; on this point the design rests on the participant's declaration.

Do I need permission to run a design from the agenda?

No. The agenda paper is licensed CC BY 4.0 and anyone can run its designs. The author does not annotate; he only asks that a design be registered before data collection and that its result be published whatever it is. Materials for designs other than D1 are shared on request.

References

• Bulut, L. (2026). A Research Agenda for Objective Projection: Testable Hypotheses, Power Analyses, and Open Designs (version 0.2). Zenodo. doi:10.5281/zenodo.24115699
• Bulut, L. (2026). Reliability study of the Objective Projection annotation scheme. arXiv:2609.13936; doi:10.5281/zenodo.21740239
• Bulut, L. (2026). Objective Projection dataset. Hugging Face. doi:10.57967/hf/8960
• Bulut, L. (2026). D1 second-annotator package. Objective Projection dataset, evaluation/second_annotator/. Hugging Face.
• Byrt, T., Bishop, J. and Carlin, J. B. (1993). Bias, prevalence and kappa. Journal of Clinical Epidemiology, 46(5), 423–429.
• Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. doi:10.1177/001316446002000104
• Feinstein, A. R. and Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543–549.
• Plank, B. (2022). The "problem" of human label variation: On ground truth in data, modeling and evaluation. Proceedings of EMNLP 2022.

Citation

To cite this article:

@misc{bulut2026agendaannouncement,
  author       = {Bulut, Levent},
  title        = {Objective Projection Research Agenda: 13 Designs That Invite Refutation},
  year         = {2026},
  howpublished = {\url{https://leventbulut.com/objective-projection-research-agenda-open-designs/}},
  note         = {leventbulut.com}
}

To cite the agenda paper itself:

@misc{bulut2026agenda,
  author       = {Bulut, Levent},
  title        = {A Research Agenda for Objective Projection: Testable Hypotheses, Power Analyses, and Open Designs},
  year         = {2026},
  version      = {0.2},
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.24115699},
  howpublished = {\url{https://doi.org/10.5281/zenodo.24115699}}
}