Controversy erupts over a non-consensual AI experiment on mental health

On Friday Koko co-founder Rob Morris announced on Twitter that his company conducted an experiment to provide mental health services to 4,000 people in writing without informing them first, The Verge reports. have critics called the experiment was highly unethical because Koko was not getting informed consent from people seeking help.

Koko is a nonprofit mental health platform that connects teens and adults in need of mental health help with volunteers through messaging apps like Telegram and Discord.

On Discord, users log into the Koko Cares server and send direct messages to a Koko bot that asks several multiple-choice questions (for example, “What’s the darkest thought you have about this?”). It then shares one person’s concerns – written as a few sentences of text – anonymously with someone else on the server who can reply anonymously with their own short message.

During the AI ​​experiment, which applied to about 30,000 messages, according to to Morris: Volunteers helping others had the option to use an answer automatically generated by OpenAI’s GPT-3 large language model instead of writing one themselves (GPT-3 is the technology behind the recently popular ChatGPT- chatbot).

In his tweet thread, Morris say that people rated AI-crafted responses highly until they discovered they were AI-written, suggesting a significant lack of informed consent during at least one stage of the experiment:

Posts composed by AI (and checked by humans) were rated significantly higher than posts written by humans (p < .001). Response times dropped by 50% to well under a minute. And yet… we took this off our platform pretty quickly. Why? Once people learned that the messages were co-created by a machine, it didn't work. Simulated empathy feels weird, empty.

In the server’s introduction, the maintainers write, “Koko connects you with real people who really understand you. No therapists, no counselors, just people like you.”

Shortly after posting the Twitter thread, Morris received many responses criticizing the experiment as unethical, citing concerns about the lack of informed consent and ask if a Institutional Review Board (IRB) approved the experiment. In the United States, it is illegal to conduct research on human subjects without legally valid informed consent, unless an IRB rules that consent can be waived.

In a tweeted response, Morris said that the experiment would be “exempt” from informed consent requirements because he had no intention of publishing the results, leading to a parade of shocked replies.

The idea of ​​using AI as a therapist is far from new, but the difference between Koko’s experiment and typical AI therapy approaches is that patients typically know they are not talking to a real human being. (Interestingly, one of the first chatbots, ELIZA, simulated a psychotherapy session.)

In Koko’s case, the platform offered a hybrid approach where a human intermediary could preview the message before sending it, rather than a direct chat format. But without informed consent, critics claim Koko violated ethical rules designed to protect vulnerable people from harmful or abusive investigative practices.

On Monday, Morris shared a post responding to the controversy explaining Koko’s path forward with GPT-3 and AI in general, writing, “I receive criticism, concerns and questions about this work with empathy and openness. We share an interest in to ensure that any use of AI is handled with care, with great concern for privacy, transparency and risk mitigation. Our Clinical Advisory Board meets to discuss guidance for future work, particularly with regard to IRB approval.”