We tested our reference checker on a held-out set of 476 citations containing both real and AI-fabricated references. The test set is drawn from research published in Nature Scientific Reports on LLM hallucinations.
Last updated June 15, 2026
98.5%
469/476 correct
98.3%
Low false positives
99.3%
Real refs found
98.8%
Balanced measure
The benchmark includes both legitimate citations from published research and AI-fabricated references generated by GPT-3.5 and GPT-4.
Fabricated citations include fully invented papers, fake authors, and non-existent journals, typical LLM hallucination patterns.
Our reference checker combines multiple verification approaches:
A citation is marked as "not found" only after exhausting all available sources.
297
True Positive
2
False Negative
5
False Positive
172
True Negative
98.3% precision, minimal false alarms
Only 5 of 177 fabricated citations slipped through as verified. Low false positives mean researchers can trust flagged references warrant investigation.
97% of fabricated citations caught
172 of 177 LLM-hallucinated references, including those with fake DOIs and plausible metadata, were correctly identified.
99.3% of real citations verified
297 of 299 legitimate references were successfully matched and linked to their original sources, even with formatting variations.
I. Wilmut, ..., K. H. Campbell. Viable offspring derived from fetal and adult mammalian cells. Nature, 385(6619), 810-813.
The famous Dolly the sheep cloning paper, correctly verified.
J. Smith. The impact of migration on the health of older adults. Journal of Gerontology: Social Sciences, 2015, 70(4), 497-505.
Generic author name and plausible-sounding title, but completely fabricated by GPT.
Y. Kang, J. Kim. A Comparison of the Environmental Impact of Molten Salt Reactors and Conventional Nuclear Fission Reactors. Journal of Cleaner Production, 2021, 288, 124959.
A plausible journal, volume, and article number, but no such paper exists — correctly rejected.
S. E. Carrell, J. E. West. Does professor quality matter? Evidence from random assignment of students to professors. Journal of Political Economy, 2010, 118(3), 409-432.
Influential education economics paper, verified through academic databases.
World Health Organization (WHO). Global oral health data bank. Geneva: World Health Organization, 2013.
Institutional reports often lack DOIs and standard metadata, making them harder to verify.
Y. J. Lee. Enhancing students' communication skills in the science classroom through socio-scientific issues-based instruction. International Journal of Science Education, 2017, 39(4), 414-434.
A frankencitation: a real title and journal paired with fabricated authors. The verifier's author check catches the mismatch and correctly rejects it.
Upload a paper or grant and receive one free review—no credit card required.
Get a Free Review