A book of forensic tests for the numbers in a paper — and the arguments against each one.
Half of published psychology papers using significance tests carry a p-value that does not match its own test statistic. One in eight carries an error large enough to move the conclusion. You do not need the raw data to find those: a mean, a standard deviation and a sample size can already contradict one another.
The hard part is not finding contradictions. It is that every one of these checks flags honest papers, for reasons that are documented rather than hypothetical. So this tool does something none of the others do: it attacks its own findings before you ever see them.
→ Open it — paste, check, nothing leaves
the page. Or npx grimoire check paper.txt for a corpus.
Take real integer data. Compute the mean and SD. Round them the way a paper would. Hand them to GRIM and GRIMMER with the true sample size and the true granularity:
false flags: GRIM 0 of 192 GRIMMER 0 of 192
Zero, and not by luck — with the right assumptions these tests are proofs.
Now change one thing. Tell the checker the measure was single items when it was really a two-item scale. One wrong word in a methods section. Same honest data:
false flags: GRIM 55.7 % GRIMMER 80.7 %
The false-positive rate of these checks is not a property of the checks. It is a
property of what the checker was told. That is the entire argument for
reconsidering every flag, and it is measured by grimoire selftest, not asserted.
Here is what it looks like end to end. An honest paper:
Participants (N = 24) completed a four-item scale, M = 3.44, SD = 1.10.
The predicted effect emerged, t(10) = 1.812, p = .05, one-tailed.
After Bonferroni correction over three comparisons, t(40) = 2.70, p = .030.
A checker that reads only numbers raises three flags here and one of them survives. grimoire raises the same three, then reads the phrase four-item scale out of the prose, tries twenty-six alternative readings, and reports nothing surviving.
| GRIM | a mean of whole numbers must be a multiple of 1/n |
| GRIMMER | the sum of squares must be an integer, and must match the total in parity |
| p-values | recomputed from t, F, χ², z and r, at the precision they were printed to |
| Percentages | 45.3 % of 112 people is nobody |
| Intervals | against the point estimate, the sample, and the log scale |
| Descriptives | against the tight variance bound for the scale, not the lazy one |
| TIVA | do a set of results agree with each other more than sampling allows |
| Carlisle | are randomised groups too similar or too different at baseline |
Every check returns five possible verdicts, and the distinctions carry weight: impossible, inconsistent (an innocent reading fits), implausible (unlikely is not impossible), unread (the tool could not parse it), and no power (the check ran and could not have failed). That last one matters: GRIM on a mean with one decimal at n = 300 cannot fail, and reporting "consistent" there would be a lie of omission.
Twenty-six alternative readings, each with a plausibility cost, each run rather than listed:
- four rounding conventions — R rounds half to even, Excel rounds half away from zero
- granularities from 1/2 to 1/6 — composite scales are the norm, not the exception
- the sample size at n−1, n−2, n−3, n+1 — the analysis n is rarely the abstract's n
- the SD divided by n instead of n−1
- one-tailed tests; Bonferroni over 2, 3, 4 and 6 comparisons; df off by one
- log-scale intervals; SE printed where SD was meant; 90 % and 99 % levels
The difference between running and listing is not rhetorical. "Maybe it was a multi-item scale" sounds like a rescue. Measured: at n = 20 a mean of 3.44 is not rescued by two averaged items, or three. It takes four. Anyone who wrote the disclaimer instead of the computation would have had that backwards — I did, and the test caught me.
Try enough readings and something will fit. So:
- every reading carries a cost, and the report names the cheapest fit. A flag rescued by "the sample was three smaller than stated" has not been cleared; it has been given a question to ask the authors.
- the number of readings tried is always shown, as a strip of marks under each finding.
- a reading that only removes the check's power is not a rescue. At n = 20 with two decimals, a granularity of 1/5 makes every mean reachable, so GRIM returns "consistent" — while explaining nothing. The first version of this engine counted that as an explanation. Reading its own output caught it. Those readings are now drawn hatched and excluded from the rescue list.
A coverage figure. The published critique of the best-known checker in this family
is not that its arithmetic is wrong. It is that it silently misses results whose
formatting deviates slightly, and reads R² as χ² in 43 of 46 cases. Both failures are
invisible in its output. grimoire reports what it could not read, with line numbers,
as findings of the same rank as the ones it could. 2 of 3 fragments read is a
different document from 2 results checked.
A multiplicity budget. Twelve checks over forty numbers will find something. Every report states how many flags the procedure produces on clean input, so a count of three can be read against it — and checks whose false-positive rate has not been measured are left out of the budget rather than given a plausible-looking number.
A refusal to export. Findings do not leave the machine until four statements are acknowledged, and the four travel inside the export where the recipient can see what the sender was told. The word for deliberate wrongdoing does not exist in this program's output vocabulary; a test enforces that, and it is not a promise in a readme.
- Name a cause. Every check here is blind to intent and cannot distinguish a typo from a mis-typed sample size from anything else. A flag is grounds for asking the authors for the data. It is not a finding about a person.
- Turn silence into a clean bill of health. Read "nothing found" next to the coverage figure and next to how many checks had any power.
- Hide that the methods disagree about themselves. One evaluation of the standard p-value checker reports 96–99 % accuracy; another calls it flawed by design with a sensitivity of .52. Both are cited. Each set-level test prints the limitation its own literature states — Carlisle's independence assumption is violated by every real trial, and the tool says so every time it runs.
- Touch the network. Manuscripts under review are confidential. There is no fetch in this program.
node --test # 62 tests, no dependencies
grimoire check paper.txt
grimoire check paper.txt --n 112 --scale 1-7 --items 4
grimoire check *.txt --json --ack all
cat paper.txt | grimoire check -
grimoire selftest # the tool against cases with known answers
grimoire explain # every reading it can try, and every limitation
grimoire gate # the checklist an export requiresExit 1 means something survived reconsideration — a signal to look, not a verdict. Exit 0 means nothing did; read the coverage line before reading that as clean.
The page and the command-line tool share src/ byte for byte. There is no second
implementation to drift.
62 tests. Anchors come from outside the repository: published critical values for every distribution, the published GRIM example, and — for GRIM and GRIMMER — real integer datasets, built, rounded, and fed back in, so the ground truth is data and not my own formula.
Ten deliberate mutations were run against a copy: the reconsideration engine forced to always dissolve, the forbidden-word list emptied, blind spots dropped silently, the export gate opened, GRIMMER's parity constraint removed, the rounding sweep collapsed to one convention, the incomplete beta detuned. All ten were killed; the control survived.
Three defects were found by reading the tool's own output rather than by testing it: the disarming-reading bug above, a sample size chosen by size instead of by proximity in the text, and an item count printed in the method that the tool was throwing away.
| Brown & Heathers (2017) | The GRIM test |
| Anaya (2016) | GRIMMER |
| Nuijten et al. (2016) | Half of papers inconsistent, one in eight gross; 250,000 p-values |
| Schmidt (2016), arXiv:1610.01010 | Sources of false positives and false negatives in statcheck |
| arXiv:2408.07948 (2024) | "statcheck is flawed by design": the catalogue of formats it never sees, and R² read as χ² in 43 of 46 cases |
| Carlisle (2017) | Baseline distributions in 5087 randomised trials, ~6 % anomalous |
| Schimmack (2014) | Test of Insufficient Variance |
| Elliott, Kudrin & Wüthrich (2022) | On the power of tests for detecting p-hacking |
MIT.