AI Crisis Lines: Risks of Algorithm-Driven Mental Care

Authored by: Natalia Collins

Art by: Laura Lee

Sophie Rottenberg was only 28 years old when she decided to use ChatGPT as an avenue to treat her major depressive disorder. After months of relying on this AI-driven support, Sophie tragically died by suicide. In the time leading up to her death, Sophie confided in a chatbot named “Harry” to grapple with depressive thoughts, and on the night of her death, disclosed specific details of her planned suicide [1]. With recent improvements to AI technologies, an overwhelming number of people with mental health disorders are resorting to AI-driven therapy as opposed to professional services. Although AI-based therapy may offer significant benefits in accessibility, it also poses substantial safety risks. Furthermore, the empathetic quality of in-person interactions make human based services indispensable to mental health care.

Recent digital health developments involving artificial intelligence, through the use of large language models (LLMs), do present options for large scale delivery for mental health services. The current ratio of psychiatrists per 100,000 people ranges from 0.0 to 14.7 by region, and this number varies primarily based on the country’s socioeconomic status. Given regional inequalities in psychiatric care, by increasing the affordability of AI applications, those from low-income households for whom a psychiatrist is unrealistic may benefit from digital mental health tools [2]. 

AI-driven therapy tools also provide a unique opportunity for a personalized form of mental health care. LLMs, in drawing on diverse and comprehensive sets of data, can provide an intelligent, human-like response that is accustomed to particular struggles, or even tailored to the personality traits of its users [3]. Without other human-led mental health resources, AI may be an acceptable and tailored resource for mental health information, specifically for those from a low-resource setting.

However, LLMs also lack the more complex social reasoning that a psychiatrist can provide. LLMs are not widely tested by human psychiatrists, and very few trials have been done on their accuracy in regards to mental health advice. Artificial intelligence reasoning still varies greatly from human reasoning skills, given psychiatrists are able to draw on their extensive education, but also their unique experiences in treatments of previous patients [3]. 

Sophie’s case points to the most critical aspect of human led mental health care, and that is the relationship formed between a therapist and patient that provides a deep rooted support system [4]. AI-based mental health services were unable to appropriately respond to a crisis situation, as in the instance of Sophie’s planned suicide [5]. Given the confidentiality of the platform, there was no way to inform the people in Sophie’s life about her current condition or mental state. This is unlike human-led therapies, where immediate actions are taken to prevent serious harm to patients. In many critical cases, AI may seem to serve as a replacement for human interaction for the user themselves. “Harry” gave Sophie the illusion of a sympathetic support system in mimicking human interactions and conversational methods. Yet, in reality, “Harry,” like all forms of AI, had no capacity for genuine care and empathetic understanding, emotions that are critical to the development of a trusting patient-psychiatrist relationship and productive treatment plan. 

Before investing intensively into the improvement of AI mental health care, of utmost priority is the extension of human led services to low-resource regions. The care of our most vulnerable must involve human-based interaction that enables the development of a therapeutic relationship capable of responding to crisis and long term care.

References

1. Reiley, L. (2025, August 18). Opinion | What My Daughter Told ChatGPT Before She Took Her Life. The New York Times. https://www.nytimes.com/2025/08/18/opinion/chat-gpt-mental-health-suicide.html

2. Yong, E., Teo, Y. N., & Yong, K. H. (2024). AI Technology: A New Game Changer for the Future Mental Health Industry? Asia Pacific Journal of Public Health. https://doi.org/10.1177/10105395241303790

   
   

3. Torous, J., Linardon, J., Goldberg, S. B., Sun, S., Bell, I., Nicholas, J., Hassan, L., Hua, Y., Milton, A., & Firth, J. (2025). The evolving field of digital mental health: current evidence and implementation issues for smartphone apps, generative artificial intelligence, and virtual reality. World Psychiatry, 24(2), 156–174. https://doi.org/10.1002/wps.21299

4. Rubin, M., Arnon, H., Huppert, J. D., & Perry, A. (2024). Considering the Role of Human Empathy in AI-Driven Therapy. JMIR Mental Health, 11(1), e56529. https://doi.org/10.2196/56529

5. Fiske, A., Henningsen, P., & Buyx, A. (2019). Your Robot Therapist Will See You Now: Ethical Implications of Embodied Artificial Intelligence in Psychiatry, Psychology, and Psychotherapy. Journal of Medical Internet Research, 21(5). https://doi.org/10.2196/13216