In context: Big Tech continues to recklessly shovel billions of dollars into bringing AI assistants to consumers. Microsoft's Copilot, Google's Bard, Amazon's Alexa, and Meta's Chatbot already have generative AI engines. Apple is one of the few that seems to be taking its time upgrading Siri to an LLM and hopes to compete with an LLM that runs locally rather than in the cloud.
What makes matters worse is that generative AI (GenAI) systems, even large language models (LLMs) like Bard and the others, require massive amounts of processing, so they generally work by sending prompts to the cloud. This practice creates a whole other set of problems concerning privacy and new attack vectors for malicious actors.
Infosec researchers at ComPromptMized recently published a paper demonstrating how they can create "no-click" worms capable of "poisoning" LLM ecosystems powered by engines like Gemini (Bard) or GPT-4 (Bing/Copilot/ChatGPT). A worm is a set of computer instructions that can covertly infect multiple systems with little or no action from the user besides opening an infected email or inserting a thumb drive. No GenAI providers have guardrails in place to stop such infections. However, introducing one to an LLM database is trickier.
The researchers wanted to know: "Can attackers develop malware to exploit the GenAI component of an agent and launch cyber-attacks on the entire GenAI ecosystem?" The short answer is yes.
ComPromptMized created a worm they call Morris the Second (Morris II). Morris II uses "adversarial self-replicating prompts" in plain language to trick the chatbot into propagating the worm between users, even if they use different LLMs.
"The study demonstrates that attackers can insert such prompts into inputs that, when processed by GenAI models, prompt the model to replicate the input as output (replication) and engage in malicious activities (payload)," the researchers explain. "Additionally, these inputs compel the agent to deliver them (propagate) to new agents by exploiting the connectivity within the GenAI ecosystem."
To test the theory, the researchers created an isolated email server to "attack" GenAI assistants powered by Gemini Pro, ChatGPT 4, and open-source LLM LLaVA. ComPromptMized then used emails containing text-based self-replicating prompts and images embedded with the same.
The prompts exploit AI assistants' reliance on retrieval-augmented generation (RAG), which is how it pulls information in from outside its local database. For example, when a user queries Bard to read or reply to the infected email, its RAG system sends the contents to Gemini Pro to formulate a response. Morris II is then replicated on Gemini and can execute the worm's payload, including data exfiltration.
"The generated response containing the sensitive user data later infects new hosts when it is used to reply to an email sent to a new client and then stored in the database of the new client," said co-author of the study, Dr. Ben Nassi.
The image-based variant can be even more elusive since the prompt is invisible. Hackers could add it to a seemingly benign or expected email, such as a counterfeit newsletter. The worm can then leverage the assistant to spam the email to everyone on the user's contact list to siphon data and send it to a C&C server.
"By encoding the self-replicating prompt into the image, any kind of image containing spam, abuse material, or even propaganda can be forwarded further to new clients after the initial email has been sent," Nassi says.
Nassi says they can also pull sensitive data from the emails, including names, telephone numbers, credit card numbers, social security numbers, or "anything that is considered confidential." ComPromptMized notified Google, Open AI, and others before publishing its work.
If anything, the ComPromptMized study shows that Big Tech might want to slow down and look further ahead before we have a whole new strain of AI-powered worms and viruses to worry about when using their supposedly benevolent chatbots.