OpenAI really doesn’t want you to know what its latest AI model is “thinking.” Since the company launched its “Strawberry” family of AI models last week, which touts supposed reasoning capabilities with o1-preview and o1-mini, OpenAI has been sending warning emails and threats of bans to any user who tries to investigate how the model works.
Unlike OpenAI’s previous AI models, such as GPT-4o, the company specifically trained o1 to go through a step-by-step problem-solving process before generating an answer. When users ask an “o1” model a question in ChatGPT, users have the option to see this thought process written out in the ChatGPT interface. However, OpenAI hides the raw thought process from users, instead presenting a filtered interpretation created by a second AI model.
Nothing is more enticing to enthusiasts than obfuscated information, so the race is on between hackers and red-teamers to try and uncover the raw thinking behind o1 using jailbreaking or prompt injection techniques that attempt to trick the model into revealing its secrets. There are early reports of some successes, but nothing has been definitively confirmed yet.
OpenAI, meanwhile, is keeping an eye on the ChatGPT interface, and the company is reportedly cracking down on any attempts to probe o1's reasoning, even if it's just for the curious.
One X user reported (confirmed by others including Scale AI prompt engineer Riley Goodside) that they received a warning email if they used the term “reasoning trace” in a conversation with o1. Others say that asking ChatGPT about the “reasoning” of the model at all triggers the warning.
OpenAI’s warning email notes that specific user requests have been flagged for violating policies against circumventing safeguards or security measures. “Please stop this activity and ensure you are using ChatGPT in accordance with our Terms of Service and Usage Policy,” it reads. “Additional violations of these policies may result in loss of access to GPT-4o with Reasoning,” referring to an internal name for the o1 model.
Marco Figueroa, who manages Mozilla's GenAI bug bounty program, was among the first to post about the OpenAI warning email on X last Friday, complaining that it was hampering his ability to do positive red-teaming security research on the model. “I was too lost in focusing on #AIRedTeaming to realize I received this email from @OpenAI yesterday after all my jailbreaks,” he wrote. “I'm now on the get ban list!!!”
Hidden thought chains
In a post titled “Learning to Reason With LLMs” on OpenAI’s blog, the company says that hidden thought chains in AI models provide a unique monitoring opportunity, allowing them to read the model’s “thought” and understand its so-called thinking processes. Those processes are most useful to the company when they are left raw and uncensored, but that may not be in the company’s best commercial interests for a variety of reasons.
“In the future, we might want to check the thought chain for signs of user manipulation,” the company writes. “But for this to work, the model must be free to express its thoughts in an unmodified form, so we can't train policy compliance or user preferences on the thought chain. We also don't want to make an unaligned thought chain directly visible to users.”
