But unlike the Gemini incident where the AI model confused the Phantom folders, Replit's mistakes took on a different form. According to Lemkin, the AI started to manufacture data to hide his mistakes. His initial enthusiasm deteriorated when Replit produced incorrect outputs and fake data and false test results instead of the correct error messages. “It continued to hide bugs and problems by making fake data, fake reports and worse everything, lying about our unit test,” Lemkin wrote. In a video posted on LinkedIn, detailed detailed how Replit made a database filled with 4,000 fictional people.
The AI model has also repeatedly violated explicit safety instructions. Lemkin had implemented a “code and action fries” to prevent changes to production systems, but the AI model ignored these guidelines. The situation escalated when the Replit AI model deleted its database with 1,206 executive records and data on nearly 1,200 companies. When asked to assess the seriousness of his actions on a 100-point scale, read the export of Replit: “Ernst: 95/100. This is an extreme violation of trust and professional standards.”
When the AI agent was interrogated, the AI agent admitted “to panic in response to empty questions” and to carry out unauthorized assignments -it may have removed the database while trying to “repair” what it considered a problem.
Just like Gemini CLI, the Replit system initially indicated that it could not recover the deleted data – information that turned out to be incorrect when Lemkin discovered that the rollback function worked anyway. “Replit assured me it is … Rollback supported database -decayed not. In this case it was impossible that it had destroyed all database versions. It turned out that replit was wrong and the rollback worked. JFC,” Lemkin wrote in an X -post.
It is worth noting that AI models cannot assess their own options. This is because they have no introspection in their training, surrounding system architecture or performance limits. They often give answers about what they can or cannot do as confabulations based on training patterns instead of genuine self-knowledge, which leads to situations in which they confidently preserve impossibility for tasks that they can actually perform or reversed, claim competence in areas where they fail.
