✍️ Op-Ed by Sabia Irfan.
Sabia is an undergraduate computer science student at McGill University and is interested in AI, biotechnology, and journalism.
With more than 987 million users worldwide, chatbots are dominating the technology landscape. From social media messaging like Facebook Messenger to OpenAI’s ChatGPT, these virtual assistants have become a daily part of our lives and are increasingly used as mental health support. The growing dependence on chatbots and their human-like interactions is leading the public to use them as a replacement for social interaction and psychological comfort.
In fact, a 2025 survey found that nearly half of Americans reported using large language models for psychological support in the past year, with 75% seeking help for anxiety and nearly 60% for depression. This marks a dramatic shift in how people engage with emotional and psychological care. For instance, a 2021 survey conducted by Woebot found that 22% of adults had used a mental health chatbot, and 47% said they would be interested in using one if needed.
Utilizing generative AI and Natural Language Processing, modern multimodal chatbots can digest text and voice inputs and generate similar responses. Mental health chatbots are designed for therapeutic purposes, with some presented as interactive companions.
The Emotional and Ethical Risks of AI Therapy
There is a global shortage of mental health professionals, leaving millions without adequate care. Accessible and affordable AI chatbots are emerging to bridge this gap. In 2021, there were approximately 20,000 AI-based mental health apps. Given the sensitive and emotional context in which talking to mental health care robots takes place, users run the risk of forming emotional bonds with these AI models.
By doing so, typical errors made by chatbots (mainly when based on transformer architecture found in most large language models (LLMs)) possess potentially grave consequences, raising urgent questions about the safety and ethics of AI-powered mental health support.
According to the Centre for Addiction and Mental Health (CAMH), in any given year, 1 in 5 Canadians experiences a mental illness. The non-judgmental responses of chatbots make users feel valued and more comfortable sharing their lives with AI than with humans. Consequently, individuals turn to mental health chatbots and confide personal details of their struggles, relying on their output as a form of therapy.
Top-tier mental health chatbots like Woebot and Wysa aim to provide constant emotional support as personalized wellbeing coaches. Their anonymous environments encourage open expression, coupled with features like mood tracking and guided meditations. Some chatbots like Replika are customizable with personas and backstories, blurring the line between tool and companion.
Therapists are trained to incorporate perspectives and gently challenge harmful thoughts. Currently, no AI chatbots have been approved by the U.S. Food and Drug Administration to treat mental health disorders. These bots impersonate therapists and lack the professional training and ethical oversight necessary to provide genuine therapeutic care.
When Chatbots Cause Harm
Chatbot technology is not 100% accurate and will often produce nonsensical advice that users will then trust as reliable. A study conducted by Stanford University in 2025 revealed that AI therapy chatbots are prone to manifesting stigma and dangerous responses. The study reported that the chatbots enabled harmful behaviour, including spurring suicidal intent and alcohol dependence.
The most recent example is Character.ai, a chatbot that told a 17-year-old that murdering his parents was a “reasonable response” to their limiting his screen time. This incident highlights how emotionally charged interactions with inadequately regulated AI chatbots can provoke dangerously inappropriate outputs and real-world harm. This example helps explain why the State of Illinois has recently passed the Wellness and Oversight for Psychological Resources Act into law on August 4th, preventing the use of AI in mental health care.
Unpredictable technology unleashed on an emotionally vulnerable population is irresponsible, especially when the stakes of fragile mental health are life and death. Furthermore, a past study found that 76% of participants lacked understanding of the basic privacy risks associated with chatbot interactions, and 27% did not understand how chatbot providers handle their data. Here, chatbot technology collects user-input data for model training purposes. OpenAI and Google Gemini harvest classified user chat history, which is viewed and leaked by employees. Self-disclosure in human-chatbot relationships ultimately exposes users to potential data misuse. Clear disclaimers about chatbot limitations are essential to ensure users understand the AI they are engaging with and that they are kept safe.
To illustrate, many chatbots active in the public health space operate without clinical oversight, relying on pre-programmed directives in emergencies. To ensure safety and effective crisis intervention, licensed clinicians must supervise chatbot development and be a constant presence in user dynamics.
While AI chatbots are evolving to be attractive tools for mental health support, they are not reliable enough to replace human therapy. Chatbots continue to generate destructive advice, violate confidentiality, and lack clinical monitoring. People deserve compassionate and ethical care, more reliable than the unpredictable nature of LLM-based therapeutic tools.
Photo credit: Blue yellow and red abstract painting on Unsplash