Riding the subway in New York, I’ve watched AI advertisements proliferate across the cars. Companies like Friend promise companionship. Others like Dawn promise a listener and 24/7 mental health services at the touch of a button. For reasons of cost, comfort, or convenience, many people will now turn to AI for mental health support before they reach out to another human being.
One analysis in Harvard Business Review ranked companionship and therapy as the current leading use of generative AI, and the fact that the two categories blur together is itself revealing. Critics have raised serious concerns about AI therapy: chatbots may miss signs of psychosis, mania, or suicidality, and a Stanford study found that they sometimes hold a stigma toward conditions like alcohol dependence and schizophrenia. These concerns are genuine. But they are ultimately questions of competence, and competence can indeed improve.
In the midst of this discourse around clinical competency, it seems to me that we have ignored a more fundamental question. Even if AI therapists become, on some level, good enough, can we trust them with our interior lives? I argue that even a perfect AI therapist carries a built-in conflict of interest.
The “frame” of an LLM chatbot — soft background with a single textbox — can make it feel like one is speaking into a digital confessional; that intimacy is part of the appeal. Talking to a chatbot feels personal and private. As AI systems grow more sophisticated and far-reaching, however, we need to ask ourselves how these services will handle our private material.
Psychotherapy, online or in person, produces incredibly personal information. Without prompting, a therapy bot could learn about your sexual history, your financial stressors, your childhood trauma, and, crucially, what makes you feel most ashamed and what makes you feel reassured. In psychotherapy, this information becomes part of an ongoing, contained therapeutic relationship. Within an AI system, it also becomes something very different: consumer data.
As a psychiatrist, I am bound by professional duties of confidentiality and, in my clinical practice, by laws such as HIPAA, which fosters patient trust. The obligations governing a consumer AI system are very different: they depend on company privacy policies, terms of service, business arrangements, and the complex technological infrastructure that remains largely invisible to the person using it. Instead of an ethical relationship, it becomes a contractual one focused on corporate engagement rather than patient care. Most of us don’t read the fine print each time we log onto a chatbot. When you speak with AI, do you trust the system you are conversing with to protect your deepest secrets?
Dedicated AI therapy services may offer restrictions on sharing your data, but in practice, many people are turning to general-purpose LLMs for therapy, which often use our data for training purposes. In the current heterogeneous landscape of AI tools, it can be difficult to know where it is safe, if at all, to have these conversations.
Consider not only who holds and owns our data, but what these AI models could do with it. What if a system could use what it knows about our psychological vulnerabilities (and about human behavior in general) to influence us? A system that knows what makes us anxious, lonely, or hopeful possesses information that could be used to shape what we buy, what we read, what we believe, and how long we remain engaged. It could sell us more services when we are vulnerable, optimize its responses to keep us talking, or prioritize engagement over directing us toward care. The same process already occurs on social media through targeted advertising, and with AI platforms drawing out our private thoughts, targeting could unfold in much more intimate, and possibly pernicious, ways.
Assuming AI companies can address the broader challenge of AI alignment and global risk, the immediate threat in therapy isn't a rogue AI acting against its user's wishes. It is the ordinary logic of the marketplace. We do not need a malicious machine to create a conflict of interest. We need only a commercial system that knows us extraordinarily well. There are many possible pitfalls: What if investigators use our therapy data to judge our likelihood of committing a crime? What if AI companies mine our therapy data to determine our suitability for a job or our mental fitness for a life insurance policy?
There are, of course, counterarguments. Given our current shortage of therapists, chatbots can immediately provide greater access to many people seeking help. Chatbots are also available around the clock, whereas human therapists are often confined by certain hours. And more AI experts are now arguing for stricter data privacy terms to limit how much AI chatbots can collect on users. But for an AI therapist to become more personalized and an effective listener, it needs a chronological memory of who we are and what we have shared with it. This creates a catch-22 for regulators: the more an AI therapist is designed to be effective and to “listen,” the more it knows about us and the more vulnerable we become.
Additionally, one could argue that given the sheer volume of information, it would be unlikely that LLMs or their engineers would be that interested in therapy data on an individual level; thus we are protected within the herd. However, the opposite also holds true — with so much data available to these companies at scale, it is more likely that they could use it to influence users on a systems level, which is in many ways more concerning.
As clinicians, we need to educate our patients about the pitfalls of speaking to AI therapists without seeming overly defensive. Most likely, patients will continue to use AI to explore their mental health, and rather than dismissing these technologies outright or focusing too narrowly on how they may take our job, we should counsel our patients on safe use. We need to advocate for more studies on how companies use psychological data. At intake, we need to ask our patients how they are using AI: to track symptoms? As a confidant? To journal? We need to develop norms and regulation around psychological data specifically, and we should provide psychoeducation to patients to distinguish between an AI designed as a clinical service with HIPAA protection and a general-purpose LLM. And with AI filling a clear demand among the public, we need to continue expanding access to human therapy for those who need it.
But perhaps we also need to reframe the question. We should stop arguing only over whether an AI system can replace the work of a therapist and instead discuss whether we want to live in a world in which our most intimate psychological lives are mediated by corporations. Even clinician-mediated platforms like BetterHelp create complicated conflicts of interest over our privacy, and if we plan to use online platforms as providers, whether to scribe or to call patients, we need to demand clearer norms about what parts of our patient conversations are feeding the machine.
When I sit with a therapist, I am talking to a person bound by ethical and legal duties. When I talk to an AI, it may feel like someone is listening, but I am speaking to a commercial product, and by extension, to the engineers, executives, and board members who govern it: people whose obligations to me are not clearly defined. The chat window may feel like a confessional, but it is not one. Before we hand over a map of our psyches, we should ask ourselves a simple question: would we say the same things if those engineers and board members were in the room?
