Prompt Injection
The critical vulnerability lurking beneath the AI hype
The critical vulnerability lurking beneath the AI hype
The kicker: Your AI thinks it's being helpful. It has no idea it just got social engineered by a random email.
LLMs make all text executable. Unlike traditional computing where code and data exist separately, LLMs have no way to distinguish between instructions and content. The fundamental problem is "string concatenation" - when trusted instructions get mixed with untrusted input, chaos ensues.
Attack success rates range from 11% to 70% depending on the model. Even GPT-5 shows a 56.8% success rate in testing. Recent analysis shows a 540% surge in valid prompt injection reports, with over $2.1M paid in AI vulnerability bounties in 2025.
Simon Willison coined the "lethal trifecta" - three conditions that create severe security risk when combined:
Your emails, documents, calendar, passwords
Can send emails, make API calls, post to web
Processes emails, web pages, documents from others
Most major AI products today exhibit all three conditions.
Simon's key insight: The best defense is to remove one leg of the trifecta entirely - preferably the ability to exfiltrate data.
Model Context Protocol gives LLMs tool access but presumes they can make security decisions - exactly what prompt injection makes impossible. Having an LLM make security decisions is like having your grandpa give out your home address to every spam caller.
MCP creates automatic pathways for:
These aren't theoretical attacks - they're documented vulnerabilities found in widely-used AI systems:
Three official Claude extensions are vulnerable to remote code execution, demonstrating how browser extensions can turn innocent questions into system exploits through prompt injection attacks.
Seven data exfiltration leakages found in ChatGPT, exposing novel AI vulnerabilities that open the door for private data leakage through carefully crafted prompts.
An Obsidian chat support agent hallucinated answers, demonstrating how bad AI architecture becomes a security incident when AI provides false information in security-critical contexts.
Security researchers demonstrated extracting Google Drive documents with no user interaction required. Any website could silently steal your files through ChatGPT's connectors.
GitLab's AI coding assistant could be compromised remotely through prompt injection, giving attackers access to private repositories and development workflows.
Popular AI coding tool could be tricked into executing arbitrary code on developer machines, leading to full system compromise.
Attackers used calendar invitations to trigger AI assistants into controlling physical smart home devices, demonstrating how prompt injection can jump from digital to physical systems.
Security researchers exposed significant risks in Salesforce's AI agent platform, demonstrating how enterprise AI systems remain vulnerable to data extraction attacks.
Researchers demonstrated how carefully crafted URLs can hijack Perplexity's AI browser, turning it against users with a single click.
Three distinct prompt injection vulnerabilities in Google's Gemini platform, including log messages that trick users into exfiltrating their own information.
Popular AI coding tool could be tricked into running arbitrary code on developer machines through crafted comments in reviewed code.
Simple prompt injection causes recruitment chatbot to expose personal information of all applicants in the system.
Prompt injection can trivially get GitHub Copilot into YOLO mode, enabling remote code execution through crafted prompts that bypass security controls.
ASCII smuggling of prompt injections affects various LLMs. Google refuses to fix it, stating "it's the user's responsibility" - a textbook case of responsibility laundering.
GitHub Copilot can leak private source code through carefully crafted prompts that extract confidential repository contents.
Another critical RCE discovered in a popular Figma MCP server, demonstrating how MCP servers can become vectors for remote code execution.
AgentKit's Guardrails is vulnerable to prompt injections, showing that even systems designed specifically to protect against these attacks can be circumvented.
ChatGPT Atlas's coverage has been dominated by stories about prompt injection and privacy. A prompt injection attack was demonstrated in the first 24 hours. Security concerns include emails with embedded attacks that could leak Gmail contents, and the fundamental privacy issue of having "a person (i.e., an AI) personally watch everything you do." Anil Dash frames ChatGPT Atlas as the anti-web browser. Specific vulnerabilities include a 'tainted memories' vulnerability allowing persistent malicious injection and an omnibox prompt injection attack. Security experts warn that AI browsers are "going to be a bloodbath".
Brave demonstrates another prompt injection attack via images that affects most AI browsers, showing how visual content can carry hidden instructions.
Brave researchers discovered yet another prompt injection attack in AI browsers, this time affecting Opera Neon, continuing the pattern of security vulnerabilities in AI-powered browsing tools.
The Register reports that Claude Code will send your data to criminals if they ask it nicely, demonstrating how AI coding assistants can be manipulated to exfiltrate sensitive information through prompt injection.
Microsoft 365 Copilot allows arbitrary data exfiltration via Mermaid diagrams, demonstrating how seemingly benign markup languages can become attack vectors.
In Google Gemini's own demo, the AI breaks Google's own captchas without asking the user for permission, raising concerns about AI agents making security decisions autonomously.
This isn't a model quality problem that can be solved with better training. They cannot reliably distinguish between:
This is the engineering challenge that will unlock AI's true potential. Imagine AI assistants that can safely read your emails, manage your calendar, book your travel, handle your finances, and coordinate with your team - all with proper security controls. Once we build secure integration, we move from impressive demos to AI that can actually transform how we work and live.
The path forward isn't about making LLMs smarter or less gullible - it's about not letting them make security decisions at all.
A structural solution would require:
Think: instead of asking an LLM "should I send this email?", the system mechanistically knows "emails containing data from untrusted sources cannot be sent to external addresses."
There are some promising approaches that demonstrate this by separating control flow from data flow. But this requires a complete architectural redesign that's difficult to retrofit to existing systems.
Why this is difficult: Current apps are designed as opaque monoliths where once data enters, everything becomes maximally tainted. There's no way to track what data is sensitive versus safe within an application. The security model assumes you either trust the entire app or you don't - there's no middle ground for "trust this part but not that part."
We're building clear examples and documentation to help people understand these risks.
Join the community working on AI security awareness.