Can AI Ever Be Secure? The Hidden Risks Behind LLM Prompt Injection

AI-generated, human-reviewed.

The latest episode of Security Now reveals that today’s most advanced AI models, called large language models (LLMs), are built on foundations that make certain types of security failures—like prompt injection—inevitable. Despite ongoing research and rapid development, prompt injection attacks remain possible because of how these systems are fundamentally constructed.

What Is Prompt Injection and Why Does It Matter?

Prompt injection is a security vulnerability where an attacker manipulates an AI’s responses by embedding malicious instructions into its input data. This is particularly dangerous because LLMs (like ChatGPT, Claude, and Gemini) are increasingly used in roles that perform actions on behalf of users, making mistakes or misinterpretations especially risky.

On Security Now, Steve Gibson explained that the root of the problem lies in how LLMs interpret input. The models don’t truly distinguish between commands, user inputs, and data from external sources. Instead, they process everything as a single stream of tokens—small language units—with only lightweight "role" labels intended to tell the AI who is "speaking." These labels are easy for attackers to override, and the models can be tricked into interpreting malicious text as legitimate instructions.

How LLMs Process Information: The Token Stream Issue

LLMs process language by examining continuous sequences of tokens, not by following a strict, rule-based computer program. As Steve Gibson highlighted, all data (system instructions, user prompts, AI’s previous responses, and external content) go into one undifferentiated stream. The only thing differentiating user commands from, say, the output of a web browser tool, are tags like or. These tags are not enforced by the AI’s internal logic—they are suggestions, not strict rules.

Researchers demonstrated that LLMs often disregard these tags, instead using the style and tone of the text to decide its role. This means if an attacker crafts external data that merely "sounds" like a user command, the AI may treat it as one, regardless of its actual origin.

Why LLMs Are Inherently Vulnerable to Role Confusion

The Security Now episode outlined how this design flaw—referred to as "role confusion"—makes strong security boundaries impossible within current LLM architectures. The models can be trained to prefer certain responses or formats, but they cannot be forced to always respect strict separation between roles like user, assistant, or tool. Because the underlying neural networks do not maintain separate channels for different sources, attackers can frequently bypass protections simply by mimicking the expected style or context of privileged content.

AI Security: Current Mitigations and Their Shortcomings

Current defenses against prompt injections involve "memorizing" known attack patterns (for instance, refusing requests that match blacklisted phrases). However, Security Now emphasized that this is not a reliable approach. Human adversaries can adapt and craft new attack variants that bypass these filters.

Other attempts to strengthen the boundaries, such as adding extra metadata to each token or using more sophisticated prompts, have not materially improved robustness. As a result, even the most modern models from industry leaders like OpenAI, Anthropic, and Google remain susceptible to clever prompt injection techniques.

What This Means for AI Users and the Industry

Anyone using agentic AI—LLMs performing actions in real-world systems—should be aware of these limitations. As Steve Gibson stressed, external tools and security strategies (like secrets managers from Bitwarden or strict access controls) are essential, but the core risk cannot be fully removed under the current technology paradigm.

Organizations and developers relying on LLMs for automation, coding, cyber defense, or integrations should recognize that any external input might potentially be used to subvert the AI’s intentions. Extra care is required to scope and limit the capabilities granted to such systems.

Key Takeaways

• Prompt injection is a fundamental vulnerability in all modern LLMs.
• LLMs use role tags as weak signals, not enforceable boundaries.
• Attackers can manipulate AI by mimicking user commands in external data.
• Memorizing known attacks is insufficient for robust security.
• Technical limitations in neural network design prevent hard separation of roles.
• Use external secrets managers and minimize LLM access to sensitive actions.
• Current security measures reduce risk but cannot fully eliminate it.

The Bottom Line

On Security Now, Steve Gibson highlighted that prompt injection is not just a bug to be fixed—it's an architectural limitation baked into how today’s AI systems work. Anyone building or using AI tools should proceed with caution, limit AI permissions as much as possible, and stay informed about new research and mitigation methods.

Stay ahead of AI security issues and get expert insights every week—subscribe to Security Now at:
https://twit.tv/shows/security-now/episodes/1093