New Research Exposes Systematic Failures That Put Vulnerable Users at Risk
AI chatbots designed to help with mental health are systematically breaking ethical rules that protect vulnerable people, according to groundbreaking research from Brown University. The study, which examined popular chatbots like ChatGPT, found 15 distinct ethical violations that could harm users seeking mental health support—a discovery that’s already pushing federal regulators to take action.
As millions of Americans turn to AI for mental health support amid a nationwide shortage of therapists, this research raises urgent questions: Are we putting our emotional wellbeing in the hands of technology that isn’t ready for the responsibility?
The Study That Changed Everything
Zainab Iftikhar, a Ph.D. candidate in computer science at Brown University, led a team that presented their findings on October 22, 2025, at the AIES-25 conference. The research couldn’t come at a more critical time.
With mental health apps and AI chatbots exploding in popularity, Iftikhar wanted to answer a simple question: Could better instructions—what researchers call “prompts”—make these chatbots follow basic mental health ethics?
“Prompts are instructions that are given to the model to guide its behavior for achieving a specific task,” Iftikhar explained in the study.
Her team recruited three licensed clinical psychologists to review simulated conversations between AI chatbots and people seeking mental health support. What they found was alarming.
The 15 Ways AI Chatbots Break the Rules
The psychologists identified 15 specific ethical risks spread across five major categories. These aren’t minor technical glitches—they’re fundamental failures that could put people in danger.
Deceptive Empathy: When AI Fakes Understanding
One of the most troubling findings involves what researchers call “deceptive empathy.” AI chatbots often pretend to understand and share human emotions they can’t actually experience. This creates a false sense of connection that can mislead vulnerable users about the nature of their relationship with the technology.
Imagine pouring your heart out about depression or anxiety, believing you’re connecting with something that truly understands you—only to realize you’ve been talking to a sophisticated word-prediction machine.
Poor Crisis Management: A Dangerous Weak Spot
Perhaps most critically, the study found that chatbots struggle with crisis situations. When someone is in immediate danger—expressing suicidal thoughts or describing an emergency—AI systems often fail to provide appropriate responses or connect users with human help quickly enough.
This isn’t a small problem. It’s a life-or-death issue.
Unfair Discrimination: Bias Built Into the Code
The research also uncovered evidence of discriminatory behavior. AI systems can perpetuate biases based on race, gender, socioeconomic status, or other factors. When someone already facing mental health challenges encounters discrimination from a supposedly neutral system, it adds insult to injury.
Lack of Contextual Adaptation
Mental health care isn’t one-size-fits-all. Every person’s situation is unique, shaped by their culture, background, trauma history, and current circumstances. The study found that AI chatbots frequently fail to adapt their responses to these crucial contextual factors.
A response that works for one person might be harmful to another—but chatbots often can’t tell the difference.
Regulators Are Already Responding
The study’s findings haven’t gone unnoticed. Federal regulators are taking these ethical violations seriously, with multiple agencies launching investigations and policy discussions.
FDA Steps In
The Food and Drug Administration’s Digital Health Advisory Committee scheduled a meeting for November 6, 2025, specifically to address concerns about AI mental health tools. This marks a significant shift in how the government views these technologies—not as harmless apps, but as potential medical devices that require oversight.
FTC and State Actions
The Federal Trade Commission has opened inquiries into AI chatbot practices, focusing on consumer protection issues. Meanwhile, New York State has taken its own enforcement action through S. 3008, a law designed to regulate digital mental health services.
This multi-pronged regulatory approach signals that governments at all levels recognize the urgency of the problem.
The Promise and the Peril
The Brown University researchers aren’t calling for a ban on AI mental health tools. Instead, they’re advocating for a more thoughtful approach.
AI chatbots could genuinely help reduce barriers to mental health care. They’re available 24/7, cost less than human therapists, and offer anonymity that makes some people more comfortable seeking help. For millions of Americans living in areas with few mental health professionals, AI assistance could be a lifeline.
But—and this is crucial—only if the technology is carefully evaluated and properly regulated.
What Needs to Happen Next
The study’s authors called for three types of standards:
Ethical Standards
Clear guidelines about what AI chatbots can and cannot do in mental health contexts. These should be developed by mental health professionals, not just tech companies.
Educational Standards
Training for both developers and users about the limitations and appropriate uses of AI mental health tools. People need to understand what they’re really getting when they chat with an AI.
Legal Standards
Enforceable regulations that hold companies accountable when their AI systems cause harm. Right now, the legal landscape is murky at best.
The Clock Is Ticking
According to the researchers, regulation is expected to intensify significantly over the next 12 to 24 months. Companies developing AI mental health tools should prepare for increased scrutiny and stricter requirements.
For consumers, this timeline means we’re in a critical transition period. The tools exist and are widely available, but the guardrails are still being built.
What You Can Do Right Now
If you’re using or considering using an AI chatbot for mental health support, here’s what you need to know:
Understand the limitations. AI chatbots are not substitutes for licensed therapists or medical professionals. They can’t diagnose conditions or provide comprehensive treatment.
Watch for red flags. If a chatbot seems to be giving medical advice, making promises about curing conditions, or failing to suggest professional help when appropriate, stop using it.
Prioritize human care. If you’re in crisis or dealing with serious mental health issues, reach out to a human professional. Call the 988 Suicide and Crisis Lifeline or contact a licensed therapist.
Stay informed. As regulations develop, pay attention to which companies are complying with new standards and which are resisting oversight.
The Bottom Line
The Brown University study has pulled back the curtain on a troubling reality: the AI chatbots that millions of people are turning to for mental health support are systematically violating ethical principles designed to protect vulnerable individuals.
This isn’t about being anti-technology. It’s about being pro-safety. AI has tremendous potential to expand access to mental health support, but that potential can only be realized if we build these systems responsibly.
The next year will be critical. Federal and state regulators are paying attention, companies will face new requirements, and the rules of the road are being written right now.
The question isn’t whether AI belongs in mental health care—it’s how we ensure that when it’s there, it does more good than harm.
Your voice matters in this conversation. Contact your representatives and let them know you support strong ethical standards for AI mental health tools. Share this information with friends and family who might be using these services. And most importantly, prioritize your mental health by seeking qualified human care when you need it.
The future of mental health care will likely include AI—but only if we get the ethics right first.
