Chatbots suggest dementia
Print
A new study finds several AI chatbots show signs of cognitive decline.
A recent study published in The BMJ has found that almost all leading artificial intelligence (AI) chatbots exhibit signs of mild cognitive impairment, drawing parallels to early-stage dementia in humans.
The research assessed popular large language models (LLMs), including OpenAI’s ChatGPT versions 4 and 4o, Anthropic’s Claude 3.5 “Sonnet,” and Alphabet’s Gemini versions 1 and 1.5.
The study used the Montreal Cognitive Assessment (MoCA), a tool commonly employed to detect cognitive impairments in humans, along with additional diagnostic tests such as the Stroop test and various visuospatial tasks.
ChatGPT 4o achieved the highest MoCA score of 26 out of 30, meeting the threshold for normal cognitive function.
ChatGPT 4 and Claude followed closely with scores of 25, while Gemini 1.0 scored 16.
Despite strong performances in tasks related to language, attention, and memory, all models struggled with visuospatial and executive functions.
None successfully completed the clock-drawing or trail-making tasks - standard MoCA components used to assess spatial reasoning.
Notably, Gemini models failed at the delayed recall task, with Gemini 1.0 even acknowledging memory difficulties.
Older versions of AI models tended to perform worse, echoing cognitive decline patterns observed in human ageing.
“Older chatbots, like older patients, tend to perform worse on the MoCA test,” the study says.
The findings raise critical concerns about the reliability of AI in clinical settings.
While LLMs have previously outperformed human doctors in certain medical exams, their cognitive limitations - particularly in tasks requiring visual abstraction and empathy - may hinder their effectiveness in diagnostic roles.
“Not only are neurologists unlikely to be replaced by large language models any time soon, but our findings suggest that they may soon find themselves treating new, virtual patients - artificial intelligence models presenting with cognitive impairment,” the study concluded.
