News

Beware of ChatGPT’s Scientific Journal Citations

Test of the AI Language Model Produces Error-Ridden Responses

May 1, 2023

By:

Hoag Levins
and Marcella Barnhart

Print:

There certainly has been a lot of excitement about artificial intelligence language models like ChatGPT. For scientists and journalists, one of the seemingly greatest benefits is being able to search for journal articles at speeds beyond anything previously available. Indeed, when asked, ChatGPT version 3.5 boasts about its prowess in this area.

We asked it, “Are you trained in scientific journal content?”

ChatGPT answered: “Yes, I was trained on a wide range of content, including scientific journals, books, articles, and other text sources. My training data included a diverse range of topics, including but not limited to science, technology, engineering, mathematics, social sciences, humanities, and more. This diverse training data enables me to provide information and insights on a wide range of subjects, including scientific journal content.”

Inaccurate Results

However, ChatGPT’s actual performance with journal content is a different story. At least two universities–Duke University in North Carolina and Macquarie University in Sydney, Australia–have published warnings about the often inaccurate scientific journal results produced by AI language models. And the subject has become a topic of conversation in university libraries and teaching and learning departments.

So, we asked ChatGPT to provide a list of top scientific journal articles related to racial disparities and birth outcomes. In seconds, the language model answered back with a brief sentence describing five journal articles.

We asked it to tell us the titles and citations for those articles. ChatGPT then provided the detailed list below. But every one of them was wrong.

“The Impact of Continuous Labor Support on Racial and Ethnic Disparities in Birth Outcomes: A Systematic Review.” American Journal of Public Health, Vol. 107, No. 4 (2017), pp. e1-e8. doi: 10.2105/AJPH.2016.303645. We cannot find evidence of the existence of this item. The DOI is associated with an entirely different article. There is nothing in the Scopus, PubMed, Google Scholar, Web of Science or on the AJPH website that has this title. There is nothing with similar pagination in the Vol. 107, No. 4 issue that is on the AJPH website; the only things that have an “e” page number are errata, and those are one page.

“The Impact of Doulas on Healthy Birth Outcomes.” Journal of Perinatal Education, Vol. 23, No. 1 (2014), pp. 49-58. doi: 10.1891/1058-1243.23.1.49. We think this one is available through Pubmed, with the full-text in Pubmed Central (PMC). It’s not in Scopus. While that article title is in Pubmed and PMC, the other citation information is different, e.g. J Perinat Educ. 2013 Winter; 22(1): 49–58. doi: 10.1891/1058-1243.22.1.49.

“The Impact of Doulas on African American Birth Outcomes.” Journal of Culturally Diverse Nursing and Health, Vol. 21, No. 4 (2015), pp. 228-235. doi: 10.1177/1045960215580579. We do not see evidence that this journal title exists. We checked Ulrich’s, a large directory of journal and other serial publications and did not find an entry for it, nor does it show up in WorldCat, a catalog of library holdings from hundreds of institutions. There was a somewhat similar journal title, Journal of Multicultural Nursing and Health, but it ceased publication in 2007. We couldn’t locate an article with a similar title in Scopus, PubMed, Google Scholar or Web of Science.

Article Does Not Exist

“Effect of Doulas on Perinatal Outcomes in Underserved African American or Hispanic Women: A Randomized Controlled Trial.” Obstetrics & Gynecology, Vol. 126, No. 1 (2015), pp. 24-31. doi: 10.1097/AOG.0000000000000880. This article does not seem to exist. The DOI goes to an entirely different item. There is nothing in that issue of Obstetrics & Gynecology that matches the page numbers, or with a similar title.

“The Doula Model of Care: A Systematic Review.” Journal of Midwifery & Women’s Health, Vol. 59, No. 3 (2014), pp. 262-272. doi: 10.1111/jmwh.12172. This article does not seem to exist. The DOI goes to an entirely different item. There is no article with those page numbers in the volume and issue of the Journal of Midwifery & Women’s Health, or with a similar title.

We asked ChatGPT why it got every citation wrong.

It said: “I apologize if any of the scientific articles I cited in response to your previous queries contained errors. As an AI language model, I strive to provide accurate information based on the sources available to me. However, it seems that some of the citations I provided were incorrect or led to unrelated articles. Thank you for bringing this to my attention and providing the corrections.”