AI search engine PerTlexity uses stealth bots and other tactics to avoid the no-crawl guidelines from websites, a statement that violates as true internet standards that have been present for more than three decades, said Network Security and Optimization Service Cloudflare Monday.
In a blog post, CloudFlare researchers said the company received complaints from customers who had rejected astonishing bots by implementing institutions in the robots of their sites.txt files and via web application Firewalls that blocked the indicated perplexity cawlers. Despite those steps, Cloudflare said, Perplexity continued to gain access to the content of the sites.
The researchers said they then wanted to test it themselves and discovered that well -known perplexity crawlers came to block blocks of robots. TXT files or firewall rules, perplexity then searched the sites using a stealth -bot that followed a series of tactics to mask the activity.
> 10,000 domains and millions of requests
“This non -specified crawler used several IPs that were not mentioned in the official IP range of Perflexity and would rotate through these IPs in response to the limiting robots. TXT policy and blocking Cloudflare,” the researchers wrote. “In addition to the rotating IPs, we have observed requests that came from various ASNs in attempts to avoid website blocks further. This activity was observed over tens of thousands of domains and millions of requests a day.”
The researchers gave the following diagram to illustrate the power of the technology they claim that perplexity is used.
If it is true, the evasion claps on internet standards for more than three decades. In 1994, engineer Martijn Koster proposed the robot exclusion protocol that a machine-readable size offered for informing crawlers that they were not allowed on a certain site. Sites that indexed their content the simple robots.txt file at the top of their homepage. De Standaard, which has since been observed and approved on a large scale, became formally a standard under the Internet Engineering Task Force in 2022.
