> ## Content Index > Fetch the complete content index at: https://aworkoflove.ghost.io/llms.txt > Use this file to discover other available public pages before exploring further. # The Summaries Looked Good. Then I Read One Properly. - URL: https://aworkoflove.ghost.io/the-summaries-looked-good-then-i-read-one-properly/ - Published: 2026-07-25T04:38:04.000Z - Updated: 2026-07-25T04:38:24.000Z - Description: I was about to call the AI summaries a success. Then I read one properly, and found a mistake I'd only caught because it happened to be in a field I know. - Author: Aworkoflove - Tags: Side Notes, zotero, obsidian, AI --- Second in a side series. The first is [here](https://aworkoflove.ghost.io/automating-my-paper-backlog/). In the first post I built a system that summarizes papers and turns them into infographics, and I made two predictions about where it would fall short. Nine papers later, I was ready to write that both predictions were wrong and the thing just worked. Then I read one of the summaries carefully, and had to write something more complicated instead. ### The two things I expected to break The prompt I borrowed was written by a gastroenterologist for clinical trials. Its structure asks for study design, population, exclusion criteria, primary outcome, statistical methods — the anatomy of a randomized controlled trial. My library is not made of trials. It is systematic reviews, consensus reports, and — because I do not only read dentistry — miscellaneous topics such as basic science, foreign affairs, and sometimes literature. So I predicted two failures. First, that the trial-shaped template would leave large gaps for anything that was not a trial. Second, that the image model would invent charts it had no numbers for. ### What it looked like Neither seemed to happen. Across nine papers — periodontal reviews, a consensus document, cancer research from Nature — the summaries read cleanly. Nothing was left as an awkward labelled blank. The infographics did not fabricate graphs. A basic science paper got a summary that read like it was written for a basic science paper. I had braced for a template fighting against the wrong material, and instead it seemed to flex around whatever I fed it. I was, at this point, about to declare success. ### Then I read one properly The mistake I had been making was reading the way you skim: top to bottom, checking the sentences made sense and the facts matched the abstract. On that standard everything passed. So I slowed down on one — a Nature paper that re-ran early tumour evolution in inbred mouse strains, nothing to do with a clinical trial. The facts were right. But under "methods" it had written: *primary outcome: selection of driver mutations; secondary outcome: cancer susceptibility, tumour growth rate.* A basic science experiment does not have a primary outcome. "Primary outcome" and "exclusion criteria" belong to clinical trials — studies that give an intervention to people and measure a pre-specified effect. A mouse tumour-evolution study has an experimental design, controls, all of it, but describing them in trial language is a category error. Nobody who does this research would frame it that way. So the summary was not inventing facts. It was doing something quieter. The template asked for a primary outcome, and rather than say "this does not apply here," the model found something in the paper to label as one. It filled a box that did not belong on the form. My first prediction had it backwards: the danger was not empty fields, it was fields filled too willingly. ### Why I nearly missed it This is the part worth being honest about. I nearly missed it for exactly the reason I could not have caught it in most of the nine papers: I do not do cancer genomics. When a summary of a periodontal review misstates something, my own knowledge flags it. When a summary of a Nature oncology paper quietly forces a trial frame onto basic research, I have no reflex that says *that word does not belong there.*I caught this one only because the mismatch happened to be blatant enough to see from outside the field. Which means the standard I was about to publish — "these read naturally to me" — was measuring the wrong thing with the wrong instrument. A summary reads naturally to me precisely when it does not trip my expertise. Outside my field I have no expertise to trip. The smooth ones are the ones I am least equipped to check, and most of my library is outside my field. ### What I actually learned Not that the system failed. The facts were accurate every time I could check them, and no charts were invented — my second prediction really was wrong. What failed was my test. "It reads naturally" tells me a summary has not made a mistake I can recognize. It says nothing about mistakes I can't. That is a more useful thing to know than "it worked," and I would not have known it if I had published a day earlier. The next post is about the fix — rewriting the prompt so it stops assuming every paper is a trial — and about the part of this that a rewrite cannot fix.