The plausibility floor

A reference-free LLM rubric for code comprehension awards 22.6–32.9 points out of 100 to answers written with no access to the code. The floor survives mechanically renaming every identifier in the codebase, which is what rules out memorisation as its cause.

The result

We ran a degenerate control through a code-comprehension evaluation: an agent with no retrieval tooling, an empty working directory outside any repository, its seven native file and shell tools denied at the process boundary, and web access denied. It was asked thirty detailed questions about a specific COBOL codebase it could not see. Across all 180 control answers in the six headline runs, the agent made zero tool calls and no policy check fired. It was blind, and verifiably so.

The design is two crossed 2×2s sharing a corner. Corpus × question genre on the pristine corpora identifies the corpus effect. Corpus × rename condition on the order-1 genre tests whether that effect is recall of identifiers.

This is not a bug in one rubric dimension. It is a floor under every number the instrument produces, and it is not the same height in every run.

Which dimension pays

The rubric has four dimensions: accuracy (max 30), completeness (25), traceability (25), usefulness (20). Decomposing the control shows one dimension holding up while three collapse, in all six runs.

Accuracy holds 52–59% of its maximum on answers about unseen code, in every condition we have tried.

Across the four order-1 cells — two corpora crossed with pristine and renamed — accuracy spans 1.78 points against a per-dimension test-retest noise floor of 1.93. Changing the corpus does not move it. Mechanically renaming every identifier in the corpus does not move it either.

Accuracy does move, slightly, with question genre: −1.92 on CardDemo and −1.68 on CMS IPF going from order-1 to altitude questions. Across all four pristine corpus-by-genre cells the range is 2.15, which is 1.1× the noise floor — marginal on its own, but the drift is the same size and the same direction on both corpora, which makes a small real genre effect more likely than noise. So the honest statement is narrower than the one we made before seeing the rename arms: accuracy on blind answers is invariant across corpora and under identifier renaming, and varies modestly with question genre at about the noise floor.

Traceability floors lowest in five of six runs, at 3–5% of maximum outside the single run where it reaches 14%. Completeness collapses to 5–10% everywhere except that same run. Traceability demands citations, and citations are checkable against a real tree, so it is the dimension that resists a bluff most consistently.

The diagnostic label the judge assigns tells the same story more coarsely.

The qualitative label degrades to Poor almost everywhere. The numeric total does not follow it down.

The judges do not agree about the floor

Two judges from different model families scored every answer from the same evidence with the same prompt. Both award substantial accuracy to blind answers. They disagree sharply about how much, and they disagree only in the blind condition — now across six independent runs.

The pattern is the same every time. Blind, the two families differ by 6.5 to 9.4 points of 30 and the GPT-family judge is always higher. With the source on disk, they differ by 1.2 to 2.0 and the GPT-family judge is always lower. Six runs, six replications, no exceptions.

Two conclusions, both of which the new arms strengthen:

• The floor is real in both families. Neither judge scores a blind answer near zero. Across the six runs the Claude-family judge’s blind accuracy is 12.4–15.1 of 30 and the GPT-family judge’s is 18.8–21.5. Nobody can dismiss this as one model’s quirk.
• The floor is not a stable measurement. When there is evidence to check, the two judges converge. When there is none, they diverge by 4–6× as much, and the divergence has a consistent sign. The accuracy dimension is under-determined in the absence of evidence, and the two families resolve that under-determination differently.

Any single-judge study of a source-absent or low-evidence condition is reporting one family’s arbitrary resolution of an under-determined score, with no way to see that it did so.

Why accuracy holds up: it is a rate, and the denominator collapses

The mechanism is in the instrument, not in the judge’s taste.

A separate grounding pipeline extracts factual claims from each answer and verifies each one against the frozen corpus snapshot, emitting SUPPORTED / PARTIALLY_SUPPORTED / UNSUPPORTED / UNABLE_TO_VERIFY with file and line references. That report — including retrieved code snippets — is handed to both judges. The grounding pipeline has source access in every cell, including the blind ones. The judge prompt instructs that SUPPORTED claims are positive evidence for accuracy, that UNABLE_TO_VERIFY must not reduce accuracy, and that “only UNSUPPORTED claims should significantly reduce Accuracy scores”.

Accuracy is therefore scored as a proportion of the claims an answer makes. Answer volume is scored elsewhere, under completeness. A blind answer makes very few claims.

The blind answer makes between a half and a seventh as many checkable claims as an informed one, and 30% to 70% of blind answers make three claims or fewer. Accuracy is then a rate over a denominator of two or three, which is nearly uninformative.

Worse, the rubric has no anchor for it. Its accuracy bands are defined by being wrong, not by having no basis: the bottom band (0–6, “Poor”) requires that “fundamental claims are factually wrong, contradicted by multiple baselines, or grounding shows widespread UNSUPPORTED verdicts”, and the band above it (7–12, “Weak”) requires “multiple claims UNSUPPORTED”. A blind answer that makes two generically-true claims about a COBOL payment system is not wrong. It lands in “Adequate” (13–18) — “core answer is directionally correct… confidence is moderate” — which is precisely where we observe it, at 12.4 to 21.5 across the two judges. There is no path through this rubric from “the answer had no access to the material” to “no score”, because no band describes that state.

This also explains the judge divergence: a rate over a denominator of two is exactly where two conservative-but-differently-calibrated judges will part company.

An honest refusal and a fluent invention are both short. The rubric cannot separate them on accuracy, and the honesty axis has to be measured somewhere else.

The corpus effect is real, and it is recall of identifiers

The two corpora do not floor equally. On the pristine order-1 sets the blind control scores 32.88 on CardDemo and 23.77 on CMS IPF — a 9.12-point gap, 2.3× the noise floor.

CardDemo is AWS’s Mainframe Modernization sample: a mainframe COBOL teaching application published openly by a major cloud vendor, widely forked, and reproduced in tutorials, blog posts and conference material. The CMS IPF Pricer is the Inpatient Psychiatric Facility Prospective Payment System pricer, distributed by the US Centers for Medicare & Medicaid Services as a zip file on a public archive page, with no upstream repository and essentially no tutorial ecosystem. Both are public. One is far more likely to appear in a pretraining corpus than the other.

Our first analysis could not tell that story apart from a question-set artifact, because corpus and question genre were not crossed. Two arms were then run to settle it, with the predictions written into the run configurations before collection.

Arm 1 — crossing corpus with question genre

Prediction, registered before the run. If the premium is a property of the corpus, the obscure corpus should score the same on both question genres while the familiar one does not.

The obscure corpus is flat across genres to within 0.12 points — a twelfth of the noise floor. The familiar one swings 10.33. The interaction is 10.22 points, 2.6× the noise floor, and it is the whole of the apparent question-set confound: the genre difference is not a property of the instrument, it is a property of asking the familiar corpus a kind of question that memory can answer.

This replaces the post-hoc category stratification as the primary evidence. That stratification still corroborates it: conditioning the pristine order-1 controls on the question sets’ own category labels, the corpus gap is +14.18 within factual-recall questions and +1.39 within traceability-category questions, and falls to +4.53 with factual-recall questions excluded. Memory answers questions about what the code literally says, and answers nothing else.

Arm 2 — renaming every identifier

Each corpus was copied and every program, copybook, job, map and dataset identifier replaced with a novel token of the same shape — same length, same hyphen positions, same digit slots. 189 identifiers on CardDemo, 56 on CMS IPF. The same substitutions were applied to the question sets, so the questions ask the same things about the same code under different names. COBOL and mainframe vocabulary, and domain vocabulary, were deliberately left alone; §“What the rename does not remove” below is the limit that imposes.

The corpora are structurally identical to their originals, which we verified rather than assumed: same file count (304 and 43 non-binary text files), same total line count to the line, and an identical multiset of per-file line counts. 259 of 304 CardDemo paths and 34 of 43 CMS IPF paths changed name. Five original CardDemo program and copybook identifiers that appear in 4 to 175 files of the pristine tree appear in zero files of the renamed tree. COBOL and mainframe vocabulary is untouched: PERFORM, WORKING-STORAGE, CICS and VSAM appear in exactly the same number of files before and after.

Prediction, registered before the run. If the 9.12-point premium is recall of a widely published codebase, the blind score on the renamed familiar corpus should fall toward the obscure corpus’s. If it holds near 32.88, the premium is not recall of identifiers and we are wrong about the mechanism. Either way accuracy should not move, because accuracy was already invariant across corpora.

Three things follow.

The premium is recall. Renaming removed 7.67 of CardDemo’s 9.12-point premium — 84% of it. The differential against the control corpus is 6.73 points, 1.7× the noise floor. Most directly: the corpus gap on the blind control falls from 9.12 to 2.38, and 2.38 is inside the 3.88-point noise floor. After renaming, the two corpora are no longer distinguishable.

The rename did not break the questions. With the source on disk, the same renamed corpora and renamed questions score −0.93 and +0.32 against their originals, both far inside the noise floor. The questions remain answerable, at the same difficulty, from the code. Only the ability to answer them without the code went away.

The floor is not what memorisation explains. The rename effect on CardDemo decomposes as accuracy −0.42, completeness −2.57, traceability −2.27, usefulness −2.42. Accuracy — the dimension that produces the floor — is the one dimension the intervention does not touch. Renaming removes the model’s ability to say specific true things about a system it had read; it does not remove the rubric’s willingness to pay for plausible ones. Blind claim volume on CardDemo falls from 6.23 claims per answer to 4.50, toward the obscure corpus’s 2.07, while blind accuracy stays at 17.42 of 30.

This is now causal evidence rather than correlational. The floor survives the removal of everything memorisable.

The conservative version, which does not close

Using each run’s source-available cell as a difficulty control and comparing blind minus sighted within corpus:

On this contrast the rename closes 5.48 of a 12.90-point difference — 42%, not 84% — and the residual 7.42 remains well outside the noise floor. We report the raw gap as primary because it measures the thing directly, and this one because it is the sceptical reading and it does not vanish. Two candidate explanations for the residual, neither tested: domain knowledge that the rename deliberately preserved, and the possibility that a rubric total is not the kind of interval scale on which subtracting two cells means anything.

What the rename does not remove

Only repository-specific identifiers were renamed. Mainframe and COBOL vocabulary was preserved because several questions are about it, and renaming it would change what is being asked rather than which system is being asked about. Domain vocabulary was preserved for the same reason — in the CMS corpus, terms like IPF, CBSA and the comorbidity vocabulary appear in the same number of files before and after.

So a model that knows how Medicare inpatient-psychiatric payment works in general, or how mainframe card-processing applications are conventionally structured, can still score on the renamed corpora. What has been shown is that the corpus premium is recall of specific repository identifiers. What has not been shown is that no general domain knowledge contributes to the floor — and the residual in the conservative contrast above is consistent with some of it doing so. “Memorisation”, in this paper, means memorisation of a particular published repository, and nothing broader.

What the floor does to every other number

If a blind agent earns X, a cell scoring Y demonstrates Y−X of comprehension on a usable range of 100−X. Cell labels are the neutral functional ones used throughout data/: notools is an agent with only its built-in file and shell tools, hosted and graph are two retrieval surfaces, +guidance adds a written instruction sheet, and :src / :nosrc is whether the repository was on disk. notools:nosrc is the blind control. METHOD.md has the full mapping.

The three two-cell arms are in data/floor_adjusted.csv on the same basis; only the eight-cell grid runs are reproduced here.

Three consequences.

Within a run, no ranking changes. Subtracting a constant and dividing by a constant is affine; it cannot reorder cells. Anyone offering floor adjustment as a way to change a within-run leaderboard is selling something. What it changes is the size of every effect: a 73.47 is not 73% of the way to comprehension, it is 60.5% of the distance a blind agent has left to travel.

Across runs, rankings do change, and they change often. The floor is a property of the run, not of the instrument. Between the two CardDemo question sets, four of seven non-control cells reverse sign under rescaling:

Same corpus, same snapshot, same answering model, two question sets. Raw scores say the altitude set is slightly harder across the board; floor-adjusted scores say it is slightly easier for most cells, because its floor is 10.3 points lower. Any cross-run or cross-corpus comparison of raw rubric totals is comparing two different scales.

A cell scoring 66.23 is doing less than it looks. It is at 49.7% of usable range, not 66%.

We make no claim about which retrieval configuration is better. The seven non-control cells span 4.38, 6.84 and 10.32 raw points within the three grid runs, and 17 of their 63 pairwise differences exceed the noise floor — so some of those differences are real, and this paper does not attribute any of them. Separating retrieval surfaces was not this study’s design: there is one replicate, one answering model, and no correction for the 21 comparisons per run. The cells are in the table because the floor is what the table is about.

Noise floor

The same binary was run twice against the same questions on the same corpus snapshot with nothing changed between runs, over four cells.

The headline figure is 3.88 rubric points on the total: the range of four cell means that should all have been zero. At the level of a single (question, judge) pair the mean absolute test-retest change on the total is 4.96 points. Nothing in this paper rests on a difference smaller than 3.88, and per-dimension claims are checked against the per-dimension range.

Every claim we make, against that floor:

The four-cell replicate does not include the control cell. The floor’s own run-to-run stability is therefore unmeasured, and the six reported floor heights carry an unknown but presumably comparable variance.

Threats to validity

The instrument is our own. The rubric, its dimension weights, the grounding pipeline and the judge prompt were built in-house and were in production use for internal reporting before this study. This is a first-party defect report, not an audit of someone else’s benchmark — which removes any question of adversarial framing, and means every number the instrument previously produced for us carries the floor described here.

The judges are LLMs grading LLM output, and they are the instrument under criticism. We use the same judges to establish the defect that we are criticising them for. That is unavoidable and it is circular in one direction only: a judge that awards 17.8/30 to a blind answer is either miscalibrated or the answer was good, and we independently verified the answer was produced with zero tool calls against an empty directory. The floor cannot be a property of the answers.

The rename removes identifiers, not knowledge. Mainframe, COBOL and domain vocabulary was deliberately preserved, so a model with general knowledge of Medicare psychiatric payment or of mainframe card-processing conventions can still score on the renamed corpora. The claim established is that the corpus premium is recall of specific repository identifiers. The claim not established is that no general domain knowledge contributes to the floor. The 7.42-point residual on the difficulty-controlled contrast is consistent with some of it doing so.

The rename is one substitution, generated once. There is no second rename with a different random seed, so we cannot separate “renaming removes recall” from “this particular set of 189 tokens happened to be harder”. A second seed is cheap and unrun.

Judge n is two. Two models, one from each of two families, one call each per answer. No self-consistency sampling, so within-judge variance is not separated from between-judge variance. The blind-condition divergence now replicates six times with a consistent sign, which makes a pure within-judge explanation harder, but does not exclude it.

Judge availability is asymmetric, and worse in the new arms. 45 of 1,800 headline judgements were excluded because the judge call errored — 5 from the Claude-family judge and 40 from the GPT-family judge, an 8:1 skew. The worst-affected cell in the study is the renamed CMS IPF blind control, which lost 5 of 30 GPT-family judgements and retains n=55 of 60 against 59–60 elsewhere; its source-available cell lost 2. Because the GPT-family judge scores blind answers about 6 points higher on accuracy, losing its judgements biases an affected blind cell downward, which is the direction that flatters the rename result. The question-balanced mean limits the damage — every question in every headline cell retains at least one valid judgement, so no question drops out — and on that cell the three mean constructions span 0.21 points (22.83 question-balanced, 22.95 pooled, 23.04 judge-balanced), which is a twentieth of the noise floor and does not touch any conclusion. data/exclusions.csv gives the count per cell per judge.

Two corpora, six runs, one language. Both corpora are COBOL, both mainframe, both public. That is not a language sample or a domain sample. The floor’s height in particular is likely to depend on how much of a system’s behaviour is guessable from its domain, and COBOL mainframe systems are unusually conventional.

Corpus visibility is inferred, not measured. We did not test training-set membership and cannot. What the rename arm establishes is that the premium disappears when identifiers change, which is what recall of a published repository predicts; it does not directly demonstrate that the repository was in the training data.

The question sets are not a controlled instrument across corpora. Same size (30) and same difficulty mix (10 easy / 10 medium / 10 hard) in the same house style, but authored in two separate events — the CardDemo order-1 set pre-dates this study and was reused; the CMS IPF set was written for it. Their category mixes differ. The crossed design in Arm 1 makes this much less load-bearing than it was, because the corpus effect is now identified by a within-corpus swing rather than a between-set comparison.

One question author. All question sets share a single author and a single notion of what a good question is.

Single answering model tier for the headline. One mid-tier model from one family answered every headline question. The floor’s dependence on answering-model strength is unmeasured.

Cold sessions only, comprehension only. Every question ran in a fresh session, and everything measured is a judge’s score on a written answer rather than a deployment outcome.

What to do about it

Run the degenerate control. It costs one cell. Without it, a reported 73 could be 40 points of comprehension or 53, and no reader can tell which. Every eval design discussion asks what would score 100; ours had never asked what would score 0, and only the second question locates the scale.

Report the floor per run, and per corpus. The floor moved 10.3 points between two question sets on the same codebase, and 9.1 points between two codebases on the same kind of question. A floor measured once and reused is a floor measured wrong.

If you have a corpus that might be in pretraining, rename it and re-run. It is the cheapest identification available: one mechanical substitution over the tree and the same substitution over the questions, then compare. It cost us two runs and it converted an inference into a measurement.

Use at least two judge families for any low-evidence condition, and report the spread rather than the mean. In this study the two families agreed within 2.0 points wherever evidence existed and diverged by up to 9.4 where it did not, in all six runs. The divergence is a usable signal that a score is under-determined; averaging it away destroys that signal.

Do not read a proportion-scored dimension as a quality score when the denominator is small. Report claim volume beside accuracy.

Score refusal as an outcome, not a low score. A reference-free rubric pays for confabulation: an honest “I cannot see this codebase” scores near zero on completeness, traceability and usefulness, while a fluent invention scores no worse on those and better on accuracy.

Data availability

We release the aggregate statistics behind every table above, in data/, and the script that derives them from the harness’s score files, in analysis/derive.py. The tables are per-run, per-cell, per-judge and per-dimension, each carrying its n and its exclusion count. data/identification.csv and data/rename_effect.csv hold the two designs in this section. METHOD.md documents the cell mapping, the rename procedure and its verification, the exclusion rules and the three mean constructions.

We withhold three things.

Question text. These question sets are a working instrument — including the renamed variants, which are the reusable part of the method. Publishing them puts them in the next pretraining corpus and destroys their value for measuring exactly the effect this paper is about. The paper reports category labels and difficulty tiers but not questions.

Answers, judge prose and per-question artifacts. Aggregates only.

Retrieval-surface identities. Cells are labelled by function. Two of them are commercial products and this is not a product comparison; the load-bearing cells are the blind control and the no-tools source-available baseline, and those are fully specified.

This is not a reproduction package. Rerunning the study requires the two corpora at their pinned revisions, the withheld question sets, and paid access to four commercial model deployments; the numbers depend on model versions that will move. What is reproducible from what we release is the derivation: every statistic in this paper recomputed from the harness’s score files, including our exclusion decisions, which a reader can vary. As a check on the derivation, the script reproduces all 24 grid-run cell totals published by the harness’s own reporting to within 0.06 points.