We tested our reference checker on a held-out set of 476 citations containing both real and AI-fabricated references. The test set is drawn from research published in Nature Scientific Reports on LLM hallucinations.
Last updated June 15, 2026
The benchmark includes both legitimate citations from published research and AI-fabricated references generated by GPT-3.5 and GPT-4.
Fabricated citations include fully invented papers, fake authors, and non-existent journals, typical LLM hallucination patterns.
Our reference checker combines multiple verification approaches:
A citation is marked as "not found" only after exhausting all available sources.
| Predicted Real | Predicted Fake | |
|---|---|---|
| Actually Real | 297True Positive | 2False Negative |
| Actually Fake | 5False Positive | 172True Negative |
Only 2 of 299 real citations were wrongly flagged
Of the 174 references R3 flagged as fabricated, 172 genuinely were. False alarms are rare enough that a flag is worth investigating rather than dismissing.
97.2% of fabricated citations caught
172 of 177 LLM-hallucinated references, including those with fake DOIs and plausible metadata, were correctly identified. The 5 that slipped through were classified as verified.
99.3% of real citations verified
297 of 299 legitimate references were successfully matched and linked to their original sources, even with formatting variations.
I. Wilmut, ..., K. H. Campbell. Viable offspring derived from fetal and adult mammalian cells. Nature, 385(6619), 810-813.
The famous Dolly the sheep cloning paper, correctly verified.
J. Smith. The impact of migration on the health of older adults. Journal of Gerontology: Social Sciences, 2015, 70(4), 497-505.
Generic author name and plausible-sounding title, but completely fabricated by GPT.
Y. Kang, J. Kim. A Comparison of the Environmental Impact of Molten Salt Reactors and Conventional Nuclear Fission Reactors. Journal of Cleaner Production, 2021, 288, 124959.
A plausible journal, volume, and article number, but no such paper exists — correctly rejected.
S. E. Carrell, J. E. West. Does professor quality matter? Evidence from random assignment of students to professors. Journal of Political Economy, 2010, 118(3), 409-432.
Influential education economics paper, verified through academic databases.
World Health Organization (WHO). Global oral health data bank. Geneva: World Health Organization, 2013.
Institutional reports often lack DOIs and standard metadata, making them harder to verify.
Y. J. Lee. Enhancing students' communication skills in the science classroom through socio-scientific issues-based instruction. International Journal of Science Education, 2017, 39(4), 414-434.
A frankencitation: a real title and journal paired with fabricated authors. The verifier's author check catches the mismatch and correctly rejects it.
A fabricated citation is a reference to a paper that does not exist. Language models produce them readily, because they often use next-token prediction to generate plausible-looking text rather than retrieving facts from a database.
Citations may be completely or partially fabricated, meaning that some or all metadata is invalid, including authors, title, journal, year, volume, pages, DOI, etc.
Fabricated citations may survive proofreading, co-author review, and often peer review, until a reader tries to follow the citation and cannot find a matching source.
A checker that flags everything catches every fabrication and is useless, because you now have to verify every reference by hand, which is the work you were trying to avoid. A checker that flags nothing is less irritating and quietly lets fabrications through. Both directions cost something, so precision and recall are reported separately here rather than collapsed into a single accuracy figure.
That separation is what makes the result readable at scale. Across a real submission workflow a single percentage point is thousands of references, and it decides how many fabrications get caught and how many are handed a permanent identifier.
Upload a paper or grant and get a full review in minutes.
Get a Review