When Chat GPT falters: Implications for reference services in academic libraries

If you’ve been following my blog lately, you may have noticed that I’ve been discussing Chat GPT in my recent posts. This artificial intelligence chatbot is capable of comprehending and responding to prompts from users, though it’s not without its flaws. I personally find it fascinating to experiment with and explore.

As a librarian, I’m always curious about the possibilities that the future may bring for information. I consider myself a creative problem-solver, and I believe that new technologies will require us to be increasingly innovative in our roles in higher education. I recently had an interesting experience when I interacted with a student who used Chat GPT to create citations for their upcoming assignment. This encounter was truly eye-opening, and it made me realize that this technology will significantly impact the way we approach reference interactions in the future.

Chat GPT generated citations

In many cases, the citations generated by Chat GPT are not genuine. Rather, they are fabricated based on the prompts given to the program. Despite this, students often mistake these citations for real ones, and it’s easy to understand why. The citations produced by Chat GPT are highly convincing and can appear to be authentic. Chat GPT provides convincingly confident answers.

I was working on a virtual chat service when the following question came in. I have rephrased the following transcript and removed any identifying information, but its core message remains the same.

Chat GPT will be challenging for academic librarians

The chat interaction

Student: I’m interested in finding some articles but they aren’t listed as available in the library.

Me: What kind of articles are you looking for? On what topic?

Student: They are on environmental injustice and activism.

Me: What search strategies have you used so far? Where have you looked?

Student: I tried looking for the articles directly but they don’t come up.

Me: Can you give me the names of the articles?

Student: “Is Personal Experience Persuasive? The Impact of Local TV News Stories on Public Attitudes Toward Environmental Risks” by Matthew C. Nisbet and Dietram A.

Me: Just a moment as I look into this.

[I started with the university federated search tool, then Google Scholar, then Google, without any results].

Me: Can you provide me with the full citation for this article? Including the journal title, volume, and issue?

Student: “Communication Research.” The article was published in 2009 in volume 36, issue 3 of the journal.

[While the student got this information for me, I also found Matthew Nisbet’s CV containing their publication history].

Me: Let me look into this, I do not see this article listed in Matthew Nisbet’s CV.

Student: I’m not sure why I can’t find it. I have the same issue with the other article citation I have. If I search it on Google, it doesn’t come up either. Does that mean it doesn’t exist?

Me: I tried going into the journal directly using the volume and issue numbers. I don’t see it in the journal either. Where did you get this citation out of curiosity?

[I asked this question because I had a very strong suspicion that the student had gotten the citation from Chat GPT, call it intuition].

Student: I also didn’t see it listed in the journal. I have been struggling to find relevant articles for my research so I asked Chat GPT and this is what it gave me.

The problem

We come across these kinds of searches quite frequently, and they highlight the importance of accurate and reliable citations. Good citation practices not only help avoid academic integrity violations but also ensure that the academic work is findable.

Previously, we would often attribute such errors to simple mistakes, such as incorrect journal names, author names, or publication dates. However, now we also have to consider the possibility that fake citations may be generated by certain tools.

Chat GPT can generate citations, but are they good?… or even real?

During the above interaction with a student, I suspected that they might have used Chat GPT to generate citations. When I asked the student where they found the citation, they admitted to using Chat GPT. I was then able to explain to the student how such tools may not always produce reliable citations.

But what if the student had told me that their professor had provided them with the citation? In that case, I would have spent an additional 10-15 minutes searching for an article that doesn’t exist. I may have then asked the student to double-check with their faculty member to verify the citation.

Wrap up

We have been spending the past few months discussing the impact that Chat GPT is going to have on student writing and assessment which certainly impacts us in libraries. However, we now have to compete with incorrectly generated citations that students are asking us to find. Although this does offer us a unique opportunity to further teach students about the need for information literacy skills, it also impacts our ability to help students find the information they are looking for. While we once compared navigating the web to the sea of information, we now have to compete with a seemingly infinite universe of information. One that generates its own information indefinitely.

Header Photo by Nick Smith on Unsplash

Leave a Reply