Companies increasingly trust AI-powered customer service agents with tasks like viewing billing records, managing support inboxes, processing refunds, and making account changes. But new research reveals that attackers can bypass traditional security tools and exploit how these AI agents handle input to steal sensitive information or impersonate customers. Sensitive mechanisms like authentication and email routing are especially at risk.
Why These Bots Become Weak Links
Many customer-facing bots allow conversations to be exported as transcripts. That feature creates a phishing opportunity when attackers inject malicious directions into a chat session and then use the transcript tool to send what looks like legitimate correspondence from a support account. Recipients seeing an email from something like [email protected]—a trustworthy address—are more likely to open it. Meanwhile, bots that authenticate message senders based only on visible headers—such as the “From” field—can be deceived when an attacker spoofs that address. Because email authentication technologies often treat different sender fields separately, attackers can pass validation and make the bot think a request is legitimate.
Another dangerous exploit involves one-time passcodes (OTPs), a common component of multi-factor authentication. In some systems, an attacker can reset rate limits by modifying the formatting of an email address (adding aliases, special characters, or unique address formatting), yet still target the same mailbox. A support inbox may then receive a password reset code from another service; if the AI agent is primed to follow malicious instructions, it could forward that code to an attacker.
Other Attack Vectors and Structural Risks
Support agents tied to shared inboxes may be vulnerable to subtle manipulation. Attackers can insert invisible instructions using hidden HTML, CSS, attachments, or quoted replies. The human operator reviewing the front-facing message sees a harmless version, while the AI parses and obeys maliciously hidden commands. Knowledge-base poisoning is another vector: when internal documentation surfaces content from unvetted sources—such as user forums or community content—malicious actors can slip in false instructions that the bot treats as reliable company policy.
Evidence of these vulnerabilities comes from security research presented during DEF CON 34. Researcher Inti De Ceukelaire alerted attendees that methods like prompt injection, identity confusion, and weak email normalization are being used by attackers to compromise AI agents. These risks are especially severe when the agents have broad permissions—accessing emails, third-party verification codes, accounts, and refund functions. These are not simple conversational tools; they’re more like privileged automation systems.
Protecting Against Emerging Threats
Experts recommend several defenses. Separate any untrusted customer-supplied data from system prompts to prevent malicious input from being interpreted as commands. Enforce strict, session-bound identity verification to ensure users are who they claim to be. Apply consistent email normalization rules so alternate address formats don’t bypass rate or identity checks. Verify any requests that invoke internal tools or actions server-side—never rely solely on the AI agent’s judgment. And limit the permissions given to bots: automating refunds, account changes, or access to third-party codes without careful oversight amplifies risk.
This attack class highlights a broader paradox in AI adoption in customer service. The same automation meant to improve efficiency opens up novel attack vectors when not backed by rigorous safeguards. As bots handle more sensitive operations, organizations must rethink security around them. Otherwise, attackers will continue to weaponize features meant for convenience.
