AI, Students, Studies, and Pizza
October 3, 2025
Google used to provide the best search results on the Web, because of accuracy and relevancy. Now Google search is chock full of ads, AI responses, and Web sites that manipulate the algorithm. Google searches, of course, don’t replace good, old-fashioned research. SSRN shares the paper: “Better than a Google Search? Effectiveness of Generative AI Chatbots as Information Seeking Tools in Law, Health Sciences, and Library and Information Sciences” by Erica Friesen & Angélique Roy.
The pair point out that students are using AI chatbots, claiming they help them do better research and improve their education. Sounds worse than the pathetic fallacy to me, right? Maybe if you’re only using the AI to help with writing or even a citation but Friesen and Roy decided to research if this conjecture was correct. Insert their abstract:
“is perceived trust in these tools speaks to the importance of the quality of the sources cited when they are used as an information retrieval system. This study investigates the source citation practices of five widely available chatbots-ChatGPT, Copilot, DeepSeek, Gemini, and Perplexity-across three academic disciplines-law, health sciences, and library and information sciences. Using 30 discipline-specific prompts grounded in the respective professional competency frameworks, the study evaluates source types, organizational affiliations, the accessibility of sources, and publication dates. Results reveal major differences between chatbots, which cite consistently different numbers of sources, with Perplexity and DeepSeek citing more and Copilot providing fewer, as well as between disciplines, where health sciences questions yield more scholarly source citations and law questions are more likely to yield blog and professional website citations. Paywalled sources and discipline-specific literature such as case law or systematic reviews are rarely retrieved. These findings highlight inconsistencies in chatbot citation practices and suggest discipline-specific limitations that challenge their reliability as academic search tools.”
I draw three conclusions from this:
- These AI chatbots are useful tools, but they need way more improvement, and shouldn’t be relied on 100%.
- Chatbooks are convenient. Students like convenience. Proof: How popular is carry-out pizza on a college campus.
- Paywalled data is valuable, but who is going to pay when the answers are free?
Will students use AI to complement old fashioned library research, writing, and memorizing? Sure they will. Do you want sausage or pepperoni on the pizza?
Whitney Grace, October 3, 2025
Comments
Got something to say?
-
- Subscribe to Beyond Search
Feature archive
News archive
-
