Blog

Google Just Told Websites to Let Its AI Agent In: Is Your Firewall Blocking Customers?

For years, the advice about bots was simple: keep the bad ones out and let Googlebot in. This week Google made that advice a little more complicated.

On November 17, Google published a new guide called “Allow Google-Agent traffic”. It is aimed at website owners and the security services that sit in front of their sites, and its message is clear: if you want Google’s AI agent to be able to use your website on behalf of a customer, your firewall may need to be told to let it through.

Here is what Google-Agent is, how it differs from Googlebot, and what I would check on your website this week. For most sites, this is a 15-minute review, not a rebuild.

What Is Google-Agent?

Abstract illustration of an AI agent icon moving through a website journey: a search box, a detail page, then a booking calendar and shopping cart with a green checkmark

In Google’s own words, “Google-Agent is the name used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request.” In plain English: a person asks a Google AI tool to do something, and the agent visits websites to get it done. Google added the name to its documentation back in March, so as Search Engine Roundtable noted, the agent is not new. The new part is a guide on making sure it can actually get in.

Here is the part that caught my attention. Google’s troubleshooting list says your allow rules should cover more than the homepage, including “on-site search, filtering, clicking into detail pages, add-to-cart, booking,” plus the behind-the-scenes requests that make those features work. That is a bot trying to do something on your site, the way a customer would.

Google’s new guide does not say which of its products send Google-Agent traffic. According to PPC Land, when the agent first appeared in March, Google pointed to Project Mariner, its experimental agent for completing tasks in a web browser, as the example. Earlier this week I wrote about Google’s Ask Maps calling businesses to book appointments. The direction is obvious: AI tools acting for customers are becoming a real way people reach businesses.

Why This Is Different From Googlebot

Googlebot crawls your site so your pages can appear in search results, and it follows the rules in your robots.txt file. Google-Agent is in a different group. Google lists it among its user-triggered fetchers, and that page says: “Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules.”

So robots.txt is not the switch here. Access is decided by whatever stands between the internet and your website: your CDN, firewall, security plugin or bot-protection settings.

A few weeks ago I wrote about Meta’s AI crawlers overloading small business websites, and blocking some of that traffic was the right call. But an agent trying to book an appointment for a real customer is a different kind of visitor. Make that decision on purpose, not by accident.

Where Google Says Its Agent Gets Stuck

Abstract illustration of an AI agent icon stopped at a brick firewall, next to a puzzle-style challenge grid with a warning sign and a speed gauge pushed into the red

Google’s guide includes a list of six reasons Google-Agent can still be blocked even after a site has tried to allow it. Here they are, translated for a business owner:

  • Rule order. Your allow rule has to run before broader rules, like data center blocks or a one-click “block AI agents” setting.
  • Challenges. Verified agent visits should skip CAPTCHAs and JavaScript challenges, including CAPTCHA widgets on search boxes and forms.
  • The whole journey. Search, filters, detail pages, add-to-cart, booking and the background requests behind them all need to work, not just the homepage.
  • Shared addresses. Many users share the same pool of Google addresses, so a strict per-IP limit can throttle all of them at once.
  • Outdated IP lists. Rules based on Google’s published address file, user-triggered-agents.json, have to stay in sync with it.
  • Proxies. Proxies must pass along the real IP address and identifying headers, or your firewall cannot verify anything.

This is not a theoretical problem. PPC Land cites an audit by Blend of 200 UK and US retailers that found bot protection blocked its AI shopping agent at 29 of them. And as Cloudflare put it in a blog post earlier this month: “Blocking agents altogether can feel like the safest choice, but it can mean losing customers.”

How to Tell the Real Google-Agent From a Fake

One warning: Google’s fetcher page cautions that “the user agent string can be spoofed.” Anyone can claim to be Google-Agent. That is why Google’s guide says to check both the user agent and that the IP address is on Google’s published list. Google is also testing signed requests (Web Bot Auth), but PPC Land reports only some requests are signed.

The takeaway: never trust the name alone. Use your security provider’s verified-bot features, which do this checking for you.

What I Would Check on Your Website This Week

Abstract illustration of a checklist with green checkmarks beside a settings panel of on and off switches, with a shield and an AI agent icon below
  1. Find out what is in front of your site. Cloudflare, a security plugin like Wordfence, your host’s firewall, or nothing? You need to know where the decision is made.
  2. If you use Cloudflare, look at one screen. Go to Security, then Settings, filter by bot traffic, and open AI Crawler and Agent access policies. Cloudflare separates Search, Agent and Training traffic, with Block, Block on pages with ads, or Allow for each. For a service business or store, I would generally keep Agents on Allow and save blocking for Training crawlers. Cloudflare itself recommends Allow for personal agents.
  3. Test your money pages. Booking, contact, quote and checkout pages are what an agent needs. If they sit behind an aggressive challenge, real customers feel that friction too.
  4. Check your rate limits. If your host or plugin throttles by IP, ask whether verified Google ranges are exempt.
  5. Look for Google-Agent in your logs. Your access logs or security dashboard can show whether it visited and got blocked. Cloudflare now shows a Personal Assistants category in its security analytics.
  6. Keep your basics strong. Allowing a verified agent is not lowering your guard. Keep updates current, two-factor on admin accounts and spam protection on forms.

Should Every Business Let It In?

Not necessarily. Sites that earn money from ads may have good reasons to limit AI tools. But if your website exists to get calls, bookings, quotes or sales, an agent acting for a real customer is closer to a lead than a threat. Blocking it by accident is like locking the front door during business hours.

There is still a lot we do not know. Google has not said how much traffic Google-Agent sends or which products use it. And PPC Land reports OpenAI is asking merchants for similar firewall changes for ChatGPT Wallet. Expect more of these requests. A clear policy now saves you guessing later.

Want Help Checking Your Site?

If you are not sure what is filtering traffic on your website, I am happy to take a look. I will check your firewall and bot settings, test your key forms, and make sure you block what you should without turning away customers. Contact Final Web Design and let’s make sure your website is open for business, for people and for the tools they use.

Sources: Google for Developers, “Allow Google-Agent traffic” (updated Nov 17, 2026); Google for Developers, “Google user-triggered fetchers”; Google crawling documentation changelog; Search Engine Roundtable, “New Google Search Help Document On How To Allow Google-Agent Traffic” (Nov 17, 2026); PPC Land, “Google lists six ways sites still block its AI agent after allowing it” (Nov 17, 2026); Cloudflare Docs, “AI Crawler and Agent access policies”; Cloudflare Blog, “See the agents, set the terms” (Nov 4, 2026).

Related Articles