Revolutionizing the Web: Adapting Internet Infrastructure for AI Agents
The internet’s architecture, traditionally tailored for human users engaging in predictable activities like searching, clicking, and streaming, is undergoing a significant transformation to accommodate the burgeoning presence of artificial intelligence (AI) agents. These agents operate differently, initiating rapid bursts of activity by deploying multiple sub-agents that can query vast databases, search extensive documents, and call numerous APIs within seconds, only to vanish just as swiftly.
Recognizing this shift, Amazon Web Services (AWS) has introduced an enhanced version of OpenSearch Serverless, a fully managed search and vector database system designed specifically for agentic workloads. This new system can instantly scale up in response to tasks initiated by AI agents and scale down to zero when idle, ensuring efficient resource utilization.
This development underscores a broader industry realization: the existing internet infrastructure, built for human-centric interactions, is less effective in a landscape increasingly dominated by machine-generated activities.
Currently, AI agents constitute a growing segment of internet traffic. Cloudflare reports that bots accounted for 31% of overall HTTP traffic over the past six months, with AI crawlers, search engines, and assistants comprising approximately a quarter of all bot requests during that period. Lai Yi Ohlsen, senior product manager at Cloudflare, predicts that non-human traffic will exceed human traffic sometime in the first half of 2027.
The expansion of AI agents is not limited to consumer applications. Enterprises are increasingly deploying these agents internally and for customer interactions, generating new forms of machine-to-machine traffic. This trend compels cloud providers and infrastructure companies to rethink and adapt systems originally designed for human users to accommodate the autonomous and continuous operations of AI agents.
AWS’s updated OpenSearch Serverless addresses this need by decoupling compute from storage, allowing for rapid scaling in response to agent-induced traffic spikes and scaling down to zero during idle periods. Tia White, general manager for Amazon OpenSearch Service, explains, Agents are moving from experimentation into production, and they create traffic patterns that previous infrastructure simply wasn’t designed for. They spike without warning, they go idle without notice, and enterprises need search that keeps up without paying for empty or idle compute.
This evolution is evident across the cloud industry. Companies like Databricks and Snowflake are repositioning themselves as AI memory and retrieval systems for enterprise data. Microsoft has updated Azure to handle AI agent bursts and facilitate memory sharing between agents. Similarly, Cloudflare has introduced infrastructure aimed at providing agents with persistent environments and instant scalability.
As AI agents become more prevalent, the pressure to redesign internet infrastructure to support machine-generated workloads intensifies. This shift is poised to make AI agents more cost-effective and easier to deploy at larger scales, marking a significant milestone in the evolution of the internet.
