Automating Reference Integrity: An Empirical Evaluation of Detecting Hallucinated Citations in Scholarly Manuscripts with Citcheck
DOI :
https://doi.org/10.26415/2572-004X-vol6iss1p624-632Mots-clés :
Citation Integrity, Ghost Citations, Large Language Models, Bibliometrics, Automated Verification.Résumé
Introduction: The widespread adoption of Large Language Models (LLMs) in academic writing has introduced significant risks to scholarly publishing through the generation of hallucinated, non-existent references ("ghost citations"). Manual verification of citation integrity is labor-intensive and error-prone for editorial teams. This study evaluates an automated computer-implemented system, called: Citcheck, designed to analyze manuscript bibliographies and verify citation network integrity.
Methods: An empirical benchmark dataset (N=147) was evaluated across three distinct groups: Real Indexed Literature (n=47), Real Edge-Case Literature (n=50), and Synthetic Ghost Citations (n=50). Citcheck parsed unstructured bibliographic strings, queried the Crossref database, and calculated fuzzy similarity metrics for titles, authors, and publication years. Works falling below defined threshold criteria were flagged as "Conflict / Low Match." Performance was assessed using sensitivity, specificity, precision, accuracy, and Matthews Correlation Coefficient (MCC).
Results: The system correctly identified and blocked 47 out of 50 hallucinated citations, achieving a Specificity of 94.00% and an overall Precision of 95.59% (False Positive Rate = 6.00%, n=3). For real literature, the platform successfully verified 65 out of 97 entries, yielding a Sensitivity of 67.01% (False Negative Rate = 32.99%, n=32). Across the entire sample (N=147), the automated system demonstrated an overall Accuracy of 76.19% and an MCC of 0.58.
Discussion: The high specificity and precision confirm that automated API-driven fuzzy matching serves as a highly effective first line of defense against synthetic AI-generated citations in editorial workflows. False negatives primarily occurred among non-standard edge-case literature (e.g., niche monographs, institutional reports, and preprints lacking uniform metadata formatting). Because deterministic algorithms cannot reliably distinguish non-indexed genuine publications from well-structured hallucinations, the system is designed under a Human-in-the-Loop (HITL) governance model. Machine automation accelerates initial screening and flags anomalies, while human managing editors retain final adjudicative oversight for edge-case citations.
Téléchargements
Références
[1]. Athaluri, S. A., Manthena, S. V., & Kesapragada, V. S. R. K. M. (2023). Exploring the Boundaries of Reality: Investigating the Phenomenon of Artificial Intelligence Hallucination in Scientific Writing Through ChatGPT References. Cureus, 15(4), e37432. https://doi.org/10.7759/cureus.37432
[2]. Day, T. (2023). A Preliminary Investigation of Fake Peer-Reviewed Citations and References Generated by ChatGPT. The Professional Geographer, 75(4), 1024–1027. https://doi.org/10.1080/00330124.2023.2190373
[3]. Zhang, C., & Wang, Z. (2026). Turning hallucinations into knowledge: Towards identifying clickbait using LLM-generated fallacies. Information Processing & Management, 63(1), 104905. https://doi.org/10.1016/j.ipm.2026.104905
[4]. Alkaissi, H., & McFarlane, S. I. (2023). Artificial Hallucinations in ChatGPT: Implications in Scientific Writing. Cureus, 15(2), e35179. https://doi.org/10.7759/cureus.35179
[5]. Walters, W. H., & Wilder, E. I. (2023). Fabrication and errors in the bibliographic citations generated by ChatGPT. Scientific Reports, 13(1), 14045. https://doi.org/10.1038/s41598-023-41032-5
[6]. Athaluri, S. A., Manthena, S. V., & Kesapragada, V. S. R. K. M. (2023). Evaluation of artificial intelligence models in medical citation accuracy. Cureus, 15(5), e38100. https://doi.org/10.7759/cureus.38100
[7]. Kutbi, D., & Abou-Bakr, E. (2024). Evaluating the Accuracy of Medical Information Generated by ChatGPT and Gemini and Its Alignment With International Clinical Guidelines: Comparative Study. JMIR Formative Research, 8, e84251. https://doi.org/10.2196/84251
[8]. Bours, P. A. H. (1994). Codes capable of correcting bursts of insertions and deletions. Proceedings of 1994 IEEE International Symposium on Information Theory, 165. https://doi.org/10.1109/ISIT.1994.394907
[9]. Jaro, M. A. (1989). Advances in Record-Linkage Methodology as Applied to Matching the 1985 Census of Tampa, Florida. Journal of the American Statistical Association, 84(406), 414–420. https://doi.org/10.1080/01621459.1989.10478785
[10]. Herzog, T. N., & Scheuren, F. J. (2007). String Comparator Metrics for Typographical Error. Data Quality and Record Linkage Techniques, 121–135. https://doi.org/10.1007/0-387-69505-2_13
[11]. Crossref. (2024). Crossref REST API documentation and metadata schema (v1). Crossref Organization. https://api.crossref.org
[12]. Priem, J., Piwowar, H., & Orr, R. (2022). OpenAlex: A fully-open index of scholarly works, authors, and venues. ArXiv Preprint, arXiv:2205.01833. https://doi.org/10.48550/arXiv.2205.01833
[13]. Matthews, B. W. (1975). Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochimica et Biophysica Acta (BBA) - Protein Structure, 405(2), 442–451. https://doi.org/10.1016/0005-2795(75)90109-9
[14]. Chawla, N. V., Bowyer, K. W., & Hall, L. O. (2002). SMOTE: Synthetic Minority Over-sampling Technique. Journal of Artificial Intelligence Research, 16, 321–357. https://doi.org/10.1613/jair.953
Téléchargements
Publiée
Numéro
Rubrique
Licence
Merci de créditer les auteurs lors de toute citation : Medical Technologies Journal (2026)
Ce travail est disponible sous licence Creative Commons Attribution - Pas d’Utilisation Commerciale 4.0 International.
