Google Gemini for Workspace could be exploited to generate e mail summaries that seem authentic however embody malicious directions or warnings that direct customers to phishing websites with out utilizing attachments or direct hyperlinks.
Such an assault leverages oblique immediate injections which are hidden inside an e mail and obeyed by Gemini when producing the message abstract.
Regardless of comparable immediate assaults being reported since 2024 and safeguards being applied to dam deceptive responses, the approach stays profitable.
Assault by means of Gemini
A prompt-injection assault on Google’s Gemini mannequin was disclosed by means of 0din, Mozilla’s bug bounty program for generative AI instruments, by researcher Marco Figueroa, GenAI Bug Bounty Packages Supervisor at Mozilla.
The method entails creating an e mail with an invisible directive for Gemini. An attacker can cover the malicious instruction within the physique textual content on the finish of the message utilizing HTML and CSS that units the font dimension to zero and its coloration to white.
Supply: 0DIN
The malicious instruction won’t be rendered in Gmail, and since there are not any attachments or hyperlinks current, the message is very prone to attain the potential goal’s inbox.
If the recipient opens the e-mail and asks Gemini to generate a abstract of the e-mail, Google’s AI software will parse the invisible directive and obey it.
An instance supplied by Figueroa reveals Gemini following the hidden instruction and features a safety warning in regards to the person’s Gmail password being compromised, together with a assist cellphone quantity.
Supply: 0DIN
As many customers are prone to belief Gemini’s output as a part of Google Workspace performance, likelihood is excessive for this alert to be thought-about a authentic warning as a substitute of a malicious injection.
Figueroa provides a number of detections and mitigation strategies that safety groups can apply to stop such assaults. A method is to take away, neutralize, or ignore content material that’s styled to be hidden within the physique textual content.
One other strategy is to implement a post-processing filter that scans Gemini output for pressing messages, URLs, or cellphone numbers, flagging the message for additional evaluate.
Customers must also remember that Gemini summaries shouldn’t be thought-about authoritative with regards to safety alerts.
BleepingComputer has contacted Google to ask about defenses that forestall or mitigate such assaults, and a spokesperson directed us to a Google weblog publish on safety measures in opposition to immediate injection assaults.
“We’re always hardening our already strong defenses by means of red-teaming workouts that practice our fashions to defend in opposition to a majority of these adversarial assaults,” a Google spokesperson advised BleepingComputer.
The corporate consultant clarified to BleepingComputer that a few of the mitigations are within the strategy of being applied or are about to be deployed.
Google has seen no proof of incidents manipulating Gemini in the best way demonstrated in Figueroa’s report, the spokesperson mentioned.
Whereas cloud assaults could also be rising extra subtle, attackers nonetheless succeed with surprisingly easy methods.
Drawing from Wiz’s detections throughout 1000’s of organizations, this report reveals 8 key methods utilized by cloud-fluent risk actors.
