A UC Santa Cruz researcher published a paper in June that got almost no attention outside a narrow slice of the machine learning conference circuit. It deserves more, because it describes a failure mode that's already partly visible in tools people use every day — and it comes with actual math showing how bad it could get.
The paper is called "Dead Science Walking." The title sounds like clickbait. The argument underneath it is not.
The problem starts in 2005, not 2026
To understand what Kargi Chauhan is warning about, you have to start two decades before anyone was talking about AI scientists at all. In 2005, epidemiologist John Ioannidis published a paper arguing that, under realistic assumptions about statistical power and researcher flexibility, most published research findings are false. It read as provocative at the time. It has aged into something closer to an operating assumption.
The evidence piled up over the following fifteen years. Bayer HealthCare tried to internally reproduce 67 of its own landmark oncology and cardiovascular studies and could only validate about a quarter of them. Amgen scientists attempted to reproduce 53 "landmark" preclinical cancer papers and succeeded with 6. In 2015, the Open Science Collaboration ran the most systematic test yet — resampling 100 published psychology studies — and found that only about 36% replicated. A year later, a 24-lab preregistered study took direct aim at "ego depletion," one of psychology's most cited and most taught effects, and found essentially nothing there.
None of this is new information to anyone who follows science journalism. What's new is what happens when you hand this literature to a machine that reads faster than any human ever could, and doesn't know which parts of it were already wrong.
What an "AI scientist" actually does
Over the last three years, a new category of system has moved from research demo to actual infrastructure. Sakana AI's "AI Scientist" runs the full loop — idea generation, code, experiments, a written paper, even automated peer review — with no human in it. ChemCrow and similar systems let language models plan and execute real chemistry. Google DeepMind's AI co-scientist, built on Gemini, generates hypotheses and experiment designs for human labs to test. Anthropic has launched its own AI for Science program and, in 2025, ran a "1,000 Scientist AI Jam" with U.S. National Laboratories.
These aren't toys. Google's AI co-scientist proposed drug-repurposing candidates for acute myeloid leukemia that were later confirmed in the lab to inhibit tumor viability. It suggested liver-fibrosis drug candidates that a Stanford team, led by Gary Peltz, validated experimentally. It proposed a new gene-transfer mechanism tied to antimicrobial resistance that researchers at Imperial College London independently confirmed after the fact. Separately, DeepMind's materials-discovery system GNoME predicted 2.2 million candidate crystal structures; at Berkeley Lab's fully autonomous A-Lab, researchers actually synthesized 41 of 58 attempted GNoME predictions — a 71% real-world hit rate. Outside labs have independently synthesized 736 more GNoME materials on their own.
That's the case for the defense: AI-assisted science is already producing real, externally checked discoveries, not just plausible-sounding text.
Chauhan's paper isn't arguing against any of that. It's arguing about a specific, narrower mechanism — one that applies most directly to systems that work the way most "AI scientists" currently do: by retrieving published literature, generating a synthesis, and having an automated evaluator decide what's promising enough to pursue next.
The mechanism, in plain terms
Here's the chain. The published scientific literature is not a neutral sample of everything that's been tried — it's a filtered record of what was positive, legible, and publishable. Franco and colleagues quantified this directly in 2014: a strong result is roughly 40 percentage points more likely to get published at all, and 60 points more likely to even get written up in the first place. Failed experiments mostly vanish into what researchers call the file drawer.
An AI system trained and grounded in that literature doesn't just have an incomplete picture. It inherits a distorted prior about how often ideas actually pan out. Chauhan formalizes this as the "null result gap" — the difference between how often a corpus presents an idea as successful and how often it actually holds up under replication. Using the numbers above, that gap comes out to roughly 0.56 in psychology, 0.60 in drug discovery, and 0.35 in cancer biology.
The part that makes this a systems problem rather than just a familiar complaint about bad science is what happens next. A retrieval step pulls up the literature most similar to a given hypothesis — and papers reporting positive results tend to be easier to find than null results, which often get filed under different, less obvious language. A generation step then turns that retrieved evidence into a fluent, confident synthesis — and language models are documented to do this in a specific, lopsided way: one study found LLM summaries of scientific findings overgeneralized the underlying claims in 26–73% of cases, nearly five times more often than human-written summaries. An evaluation step, often another AI model acting as judge, then decides which of these fluent syntheses is worth pursuing further — and LLM judges are separately documented to have real biases toward confident, novel-sounding, verbose answers regardless of whether the underlying evidence actually supports them.
Each of those three biases is modest and defensible on its own. Chauhan's contribution is showing what happens when you stack them: using deliberately conservative numbers for each stage — a 40% retrieval bias, a 30% generation bias, a 20% evaluation bias — the three stages compound into roughly a 2.18x amplification of the original gap. Push the assumptions down as far as still seems reasonable and it's still around 1.3x. The direction doesn't change. Only the size of the effect does.
What this looks like when it actually happens
The paper's clearest illustration is deliberately mundane. Imagine an AI scientist asked in 2026 to propose interventions that improve self-control. A retrieval pass over the pre-2016 literature would return a coherent, well-cited, entirely wrong story: ego depletion is real, well-established, and ripe for new applications. Unless the retrieval system has also indexed the 2016 multi-lab replication that found nothing, there is no mechanism inside the pipeline that would ever surface the correction. Nothing in that chain has to hallucinate. The system can be operating in good faith at every step and still confidently propose a research program built on an idea the field already spent a decade debunking.
Chauhan calls this "confident rediscovery," and it's the mildest of the four failure modes the paper names. The others get harder to catch. "Ghost evidence accumulation" is what happens when multiple AI systems draw on the same biased corpus, partially validate the same shaky idea, and then start citing each other's outputs as though they were independent confirmations — no single paper needs to be fraudulent for the aggregate literature to become misleading. "Replication laundering" is the same idea one step further: an AI-generated claim gets cited by another AI system as prior evidence, then comes back around dressed up as a confirmation, without any actual independent experiment ever happening in between. "Confidence miscalibration" is simply a system reporting high certainty about a finding with little or no real replication support — indistinguishable, on the surface, from ordinary confident science.
None of these are hypothetical anymore in their precursor form. GPTZero's citation audits found 100 confirmed hallucinated citations spread across 51 accepted NeurIPS 2025 papers, and more than 50 in ICLR 2026 submissions. A 2026 audit of AI-generated surgical reference material found that the worst-performing systems fabricated or failed to verify roughly a third of their cited sources. Separately, researchers testing ChatGPT-4o-mini fed it 217 papers that had been retracted or flagged for serious concerns; across 6,510 quality assessments, it never once mentioned the retraction. Another team found that ChatGPT-4o, DeepSeek, and Grok collectively cited 84 of 93 retracted stem-cell papers in their answers, with no warning attached. These are today's ordinary chatbots, not tomorrow's autonomous research loops — and they're already exhibiting the exact blindness Chauhan's model predicts would get worse, not better, as these systems get folded into full research pipelines.
Why this doesn't apply everywhere equally
The honest complication — and Chauhan's paper says this plainly, which is more than most papers making a provocative claim bother to do — is that this mechanism has a specific shape. It targets AI systems that work primarily by retrieving and narrating text literature. It has much less to grab onto in systems like GNoME, which generates candidate materials from structural and thermodynamic data rather than searching prose written by humans about what they found. That's a meaningful part of why GNoME's real-world hit rate holds up: there's no equivalent "file drawer" of unpublished crystal structures biasing what the model has seen, because it isn't reading papers about crystals — it's predicting stability from physics.
There's a second complication, which is that today's AI-scientist agents are still frequently bottlenecked by more basic problems than corpus bias. A separate evaluation of Sakana's AI Scientist and similar systems found they often fail simply at implementation — configuring an experiment correctly, running code without errors, avoiding fabrication in the final write-up. If an autonomous system can't reliably execute a clean research cycle in the first place, it can't yet run the thousands of cycles per week that would be needed to compound a corpus bias into the kind of "self-reinforcing stream of machine-generated claims" Chauhan's paper warns about. The risk is real and the mechanism is sound; the timeline for it becoming a dominant failure mode, rather than an occasional one, is genuinely uncertain.
Chauhan doesn't ask anyone to slow down AI-assisted science, and is explicit about that. The paper's own framing is that acceleration and reliability aren't in tension if the right infrastructure exists — it just doesn't exist yet, in most places it would need to.
What the paper actually proposes
Three fixes, all aimed at making the evidence substrate an AI system works from auditable rather than just larger:
• Null-result databases as first-class training infrastructure — a structured, machine-readable registry of failed replications and negative trials, extending the logic that ClinicalTrials.gov already applies to human researchers to the corpora AI systems retrieve from. A related 43-author consensus call published in PLOS Biology earlier this year argues for the same thing on the human side, for what it's worth — this isn't a fringe position.
• Retraction-aware evaluation — a simple scoring formula that penalizes a system for using retracted work as unflagged support, computable today from Retraction Watch and Crossref metadata. Chauhan is careful to note this metric would look reassuring almost everywhere by default, since retractions are rare relative to the total literature — it only becomes genuinely informative on exactly the contested, fraud-adjacent queries where retracted work clusters.
• Training corpus disclosure — a "corpus card," modeled on the model cards and datasheets already standard practice in machine learning, that would require anyone publishing AI-generated or AI-assisted research to state what their system's evidence base actually contained: which sources, what fraction of indexed claims were null or negative, whether retracted work was filtered or merely flagged, and whether a system's own outputs are allowed to re-enter its future training or retrieval data.
What's actually worth checking
None of these fixes require anyone to stop building AI-science tools, and none of them are difficult to imagine implementing at a single lab or journal this year. The gap between "difficult to imagine" and "actually built" is where this sits right now — none of the three proposals exist as standard practice anywhere yet.
If you're using an AI tool for literature review, hypothesis generation, or anything adjacent to research right now, there's a concrete, two-minute test worth running: pick a claim the tool gives you real confidence about, and ask it directly whether the source has ever been challenged, retracted, or failed to replicate. Based on the studies above, there's a real chance the honest answer is that it was never asked to check.
No comments:
Post a Comment