The Rise of agent.txt and ai.txt — New Web Rules for AI Agents

If 'AI agents' become more common than humans surfing the internet, how should websites change? Until now, a single robots.txt file to block search bots was enough. However, with the emergence of intelligent agents that go beyond simply scraping data to making payments and calling APIs, the situation has completely changed. Now, new web standards are emerging to safely guide these new artificial intelligence visitors.

Why is the existing robots.txt insufficient?

Past search engine crawlers were passive entities that simply copied information. The robots.txt file that controlled them was nothing more than a simple roadblock sign saying, "Do not enter this page."

But today's AI agents are completely different. They are active participants that go beyond reading information on a website; they can sign up for memberships, call APIs, and make payments. They aren't just 'collecting' data—they are 'taking action.'

Therefore, we now need a completely new set of agreements that go beyond simple roadblock signs, letting agents know what they can do on our site and how they should be authenticated. For example, leaving an agent-specific guide at the website root directory.

{
  "capabilities": ["search", "checkout"],
  "mcp_server": "https://api.example.com/mcp",
  "auth": "OAuth2"
}

This is exactly why the existing robots.txt cannot handle the age of agents. Beyond just coordinating how information is displayed, we now need standards to safely control the functions and permissions that machines execute directly.

The 4-file stack for a machine-readable web

With the IETF and open-source communities leading the way, a 'machine-readable web stack' for agents is quickly taking shape. These 4 key file standards help AI agents understand how a website works at a glance, just as humans use their eyes to navigate a site.

First, llms.txt is a friendly 'Markdown summary' for AI. It strips away complex HTML tags and concisely provides only the essential information and sitemap that an agent can immediately understand.

In contrast, agents.txt limits the scope of 'actual actions' an agent can perform beyond data collection. It clearly defines the authorization scope for actions involving real transactions like payments or bookings, and it also lists the Model Context Protocol (MCP) server address for AI tool integration.

For example, you can place a simple machine-readable configuration in the .well-known/agents.txt path of your website, as shown below.

{
  "capabilities": ["search", "checkout"],
  "mcpServers": {
    "booking-helper": {
      "url": "https://api.example.com/mcp"
    }
  }
}

By declaring this, agents can utilize site features safely and intelligently.

Meanwhile, ai.txt is the gateway for declaring whether your precious content can be used for AI training. It specifically satisfies requirements like 'text and data mining opt-out' from the EU AI Act and European copyright directives in a machine-readable format. Instead of long, wordy terms of service, it allows machines to read and decide immediately, providing much stronger protection for your legal rights.

Finally, ai-safety.txt provides safety guidelines for the website. It informs agents how well the site defends against malicious prompt injection attacks, or whether it is safe for autonomous loops without human intervention, preventing unexpected AI malfunctions.

How agents discover and communicate with web services on their own

Beyond just uploading files with information, automated rules are emerging that allow agents to identify each other and coordinate transactions directly. For instance, the agents.txt file stored at a predefined path on a website clearly defines what transaction permissions and tools that site allows for agents.

{
  "capabilities": ["search", "checkout"],
  "mcp": {
    "url": "https://api.example.com/mcp"
  }
}

To this, the AID and AIDIP standards proposed by organizations including the IETF are added. While the names may sound complex, the principle is simple. AID is a technology that automatically verifies an opposing agent's identity certificate and public key through the DNS system, which serves as the backbone of internet addressing. It's like mechanically verifying the digital business card of the other party.

Once identities are verified, agents begin official communication based on the AIDIP specification. Instead of humans clicking through complex signup or payment screens, these standards unify the process, allowing machines to read each other's functional specifications to safely book services or call payment APIs. A new web structure is being created where machines can cooperate safely and intelligently without human intervention.

The next step for the web: humans and machines together

If the existing web was a visual space tailored for human eyes, the future web will evolve into a space that AI agents can also read and operate in. Files like llms.txt or agents.txt serve as signposts, helping agents understand how a website works without misunderstandings and ensuring they follow the agreed-upon rules.

Now, builders who create and manage websites must move beyond simple screen design and begin preparing to politely welcome a new guest: artificial intelligence. Why not take the first steps toward the next stage of the web, where machines and humans coexist in harmony?