A third AI-related proof-of-concept attack that attracted attention used a quick injection to cause GitLab's Duo chatbot to add malicious lines to an otherwise legitimate code package. A variant of the attack successfully exfiltrated sensitive user data.
Yet another notable attack targeted the Gemini CLI encryption tool. This allowed attackers to execute malicious commands (such as wiping a hard drive) on the computers of developers using the AI tool.
Using AI as bait and hacking assistants
Other hacks involved in LLM used chatbots to make attacks more effective or stealthy. Earlier this month, two men were charged with stealing and deleting sensitive government data. One of the men, prosecutors said, tried to cover his tracks by asking an AI tool “how do I clear system logs from SQL servers after deleting databases.” Shortly afterwards, he allegedly asked the tool: “how do you clear all event and application logs from Microsoft Windows Server 2012.” Regardless, investigators were able to track the suspects' actions.
In May, a man pleaded guilty to hacking an employee of The Walt Disney Company by tricking the person into using a malicious version of a widely used open source AI image generation tool.
And in August, Google researchers warned users of the Salesloft Drift AI chat agent to consider all security tokens associated with the platform as compromised after discovering that unknown attackers used some of the credentials to access email from Google Workspace accounts. The attackers used the tokens to gain access to individual Salesforce accounts and from there steal data, including credentials that could be used in other breaches.
There were also multiple cases of LLM vulnerabilities coming back and biting the people who used them. In one case, CoPilot was caught exposing the contents of more than 20,000 private GitHub repositories from companies like Google, Intel, Huawei, PayPal, IBM, Tencent, and, ironically, Microsoft. The repositories were also originally available via Bing. Microsoft eventually removed the repositories from searches, but CoPilot continued to make them public anyway.
