This week, Sam Altman, CEO of OpenAI, and Arianna Huffington, founder and CEO of healthcare company Thrive Global, published an article in Time touting Thrive AI, a startup backed by Thrive and OpenAI’s Startup Fund. The piece suggests that AI could have a huge positive impact on public health by nudging people toward healthier behaviors.
Altman and Huffington write that Thrive AI is working on “a fully integrated personal AI coach that provides real-time nudges and recommendations unique to you, so you can take action on your daily behaviors to improve your health.”
Their vision puts a positive spin on what could prove to be one of the sharpest AI double-edged swords. AI models are already adept at persuading people, and we don’t know how much more powerful they could become as they advance and gain access to more personal data.
Aleksander Madry, a professor on sabbatical from the Massachusetts Institute of Technology, leads a team at OpenAI called Preparedness that works on this topic.
“One of the work streams in Preparedness is persuasion,” Madry told WIRED in an interview in May. “Essentially, you’re thinking about how far you can use these models to persuade people.”
Madry says he was drawn to OpenAI because of the remarkable potential of language models, and because the risks they pose have been largely unexplored. “There’s literally almost no science,” he says. “That was the impetus for the Preparedness effort.”
Persuasiveness is a key element in programs like ChatGPT and one of the ingredients that makes such chatbots so appealing. Language models are trained on human writing and dialogues that contain numerous rhetorical and persuasive tricks and techniques. The models are also typically refined to lean toward utterances that users find more appealing.
Research published in April by Anthropic, a competitor founded by OpenAI exiles, suggests that language models have gotten better at persuading people as they’ve gotten bigger and more sophisticated. The research involved giving volunteers a statement and then watching how an AI-generated argument changes their opinion of it.
OpenAI’s work extends to analyzing AI in conversation with users, something that could yield more persuasiveness. Madry says the work is being conducted on consenting volunteers, and declines to reveal the findings yet. But he says the persuasiveness of language models runs deep. “As humans, we have this ‘weakness’ that if something communicates with us in natural language [we think of it as if] “It's a human,” he says, referring to an anthropomorphism that can make chatbots seem more realistic and convincing.
The Time article argues that the potential health benefits of persuasive AI require strong legal safeguards, since the models could have access to so much personal information. “Policymakers should create a regulatory environment that fosters AI innovation while protecting privacy,” Altman and Huffington write.
This is not the only thing policymakers need to consider. It may also be crucial to consider how increasingly persuasive algorithms can be abused. AI algorithms can amplify the resonance of misinformation or generate particularly convincing phishing scams. They can also be used to advertise products.
Madry says a key question, yet to be explored by OpenAI or others, is how much more compelling or coercive AI programs that interact with users over long periods of time might prove to be. A number of companies already offer chatbots that role-play as romantic partners and other characters. AI girlfriends are growing in popularity—some are even designed to yell at you—but how addictive and convincing these bots are is largely unknown.
The excitement and hype that ChatGPT created after its launch in November 2022 led OpenAI, outside researchers, and many policymakers to focus on the more hypothetical question of whether AI could one day turn against its creators.
Madry says this risks ignoring the more subtle dangers of slippery algorithms. “I worry that they’re going to focus on the wrong questions,” Madry says of the work of policymakers so far. “That in a sense, everyone says, ‘Oh yeah, we’re addressing it because we’re talking about it,’ when in fact we’re not talking about the right thing.”
