Benj Edwards / OpenAI
Few companies have inadvertently contributed more to the rise of AI-generated noise online than OpenAI. Despite best intentions (and in violation of its terms of service), its AI language models are often used to write spam, and its groundbreaking research has inspired others to build AI models that can potentially do the same. This influx of AI-generated content has further diminished the effectiveness of SEO-driven search engines like Google. Indeed, in 2024, web search is in a sorry state.
It’s interesting, then, that OpenAI is now offering a potential solution to that problem. On Thursday, OpenAI unveiled a prototype of an AI-powered search engine called SearchGPT that aims to give users fast, accurate answers pulled from the web. It’s also a direct challenge to Google, which has also tried to apply generative AI to web search (with limited success).
The company says it plans to integrate the most useful aspects of the temporary prototype into ChatGPT in the future. ChatGPT (and Microsoft Copilot) can already perform web searches with Bing, but SearchGPT appears to be OpenAI's purpose-built interface for AI-assisted web searching.
SearchGPT aims to streamline the process of finding information online by combining OpenAI’s AI models (such as GPT-4o) with real-time web data. Like ChatGPT, users will reportedly be able to ask SearchGPT follow-up questions, with the AI model retaining context throughout the conversation.
Perhaps most importantly from an accuracy standpoint, the SearchGPT prototype (which we did not test ourselves) reportedly includes features that prominently cite web-based sources. Comments include in-line citations and links, while a sidebar displays additional source links.
OpenAI has not yet disclosed how it obtains its real-time web data, and whether it will partner with an existing search engine provider (as it currently does with Bing for ChatGPT) or build its own web crawling and indexing system.
A way to get around publishers blocking OpenAI
ChatGPT can already perform web searches with Bing, but since OpenAI unveiled a way to block its web crawler last August, that feature hasn’t been nearly as useful as it could have been. Many sites, like Ars Technica (which blocks the OpenAI crawler as part of our parent company’s policy), aren’t showing up as results in ChatGPT because of this.
SearchGPT appears to untangle the association between OpenAI's web crawler for scraping training data and OpenAI chatbot users' desire to search the web. Interestingly, OpenAI says in its new announcement of SearchGPT, “Sites can appear in search results even if they choose not to undergo generative AI training.”
Still, OpenAI says it is working on a way for publishers to manage how they appear in SearchGPT results, so that “publishers have more choices.” And the company says that SearchGPT’s ability to browse the web is separate from the training of OpenAI’s AI models.
An uncertain future for AI-driven search
OpenAI claims that SearchGPT will make web searches faster and easier. However, the effectiveness of AI-powered searches compared to traditional methods is unknown, as the technology is still in its early stages. But let’s face it: the most prominent web search engine right now is pretty terrible.
Over the past year, Perplexity.ai has gained traction as a potential replacement for Google’s AI-powered search. However, the service has been plagued by issues with fabrications and accusations of plagiarism from publishers, including Condé Nast, the parent company of Ars Technica.
Unlike Perplexity, OpenAI has many content deals with publishers and emphasizes that it wants to work with content creators in particular. “We are committed to a thriving ecosystem of publishers and creators,” OpenAI says in its press release. “We hope to help users discover publisher sites and experiences while providing more choice in search.”
In a statement for the OpenAI press release, Nicholas Thompson, CEO of The Atlantic (which has a content deal with OpenAI), expressed optimism about the potential of AI search: “AI search will become one of the most important ways people navigate the web, and it’s critical that the technology is built in these early days in a way that values, respects, and protects journalism and publishers,” he said. “We look forward to working with OpenAI in this process and creating a new way for readers to discover The Atlantic.”
OpenAI has experimented with other forks of its AI language model technology that haven’t been blockbusters (GPTs in particular come to mind), so time will tell whether the techniques behind SearchGPT have staying power—and whether it can deliver accurate results without the hallucinations. But the current state of web search beckons for new experiments to separate the signal from the noise, and OpenAI appears to be throwing its hat into the ring.
OpenAI is currently rolling out SearchGPT to a small group of users and publishers for testing and feedback. Those interested in trying out the prototype can sign up for a waitlist on the company’s website.
