ChatGPT Hallucinates Non-existent Citations: Evidence from Economics
Location
Seminole B
Start Date
22-7-2024 2:45 PM
End Date
22-7-2024 3:15 PM
Description
In this study, we generate prompts derived from every topic within the Journal of Economic Literature to assess the abilities of both GPT-3.5 and GPT-4 versions of the ChatGPT large language model (LLM) to write about economic concepts. ChatGPT demonstrates considerable competency in offering general summaries but also cites nonexistent references. Additionally, our findings suggest that the reliability of the model decreases as the prompts become more specific. We provide quantitative evidence for errors in ChatGPT output to demonstrate the importance of LLM verification.
Recommended Citation
Shapoval, Olga, "ChatGPT Hallucinates Non-existent Citations: Evidence from Economics" (2024). Teaching and Learning with AI Conference Presentations. 26.
https://stars.library.ucf.edu/teachwithai/2024/monday/26
ChatGPT Hallucinates Non-existent Citations: Evidence from Economics
Seminole B
In this study, we generate prompts derived from every topic within the Journal of Economic Literature to assess the abilities of both GPT-3.5 and GPT-4 versions of the ChatGPT large language model (LLM) to write about economic concepts. ChatGPT demonstrates considerable competency in offering general summaries but also cites nonexistent references. Additionally, our findings suggest that the reliability of the model decreases as the prompts become more specific. We provide quantitative evidence for errors in ChatGPT output to demonstrate the importance of LLM verification.