Most small business owners want more visitors to their website. This week I want to talk about the opposite problem: visitors that aren’t people, never buy anything, and can use up your hosting plan until your site goes down.
On November 2, Fast Company reported on small business owners whose websites were overwhelmed by Meta’s web crawlers, the automated bots the company behind Facebook and Instagram uses to read websites for its AI products. Business Insider followed up on November 3 with more detail on one of those cases. If your site is on a shared hosting plan, like most small business sites, this is worth ten minutes.
What Happened to These Small Business Sites
Chelsea Paulus is a children’s author and illustrator in California who sells her work online. According to Fast Company, a Meta bot called Meta-ExternalAgent got stuck in a loop on her site’s “remove from cart” function. It hit more than a hundred times per minute, adding up to about 550,000 hits over a few days.
Business Insider reports that on October 24, her host, SiteGround, emailed her that she had used 80% of her allocated CPU resources. Her four websites share a single hosting plan. Her Wordfence dashboard showed requests arriving several times a second from an IP address at a Meta location in Dallas. She blocked that bot. A couple of days later a second Meta bot, Meta-WebIndexer, showed up and pushed her resource usage to 100%, which crashed her site.
She wasn’t alone. Fast Company also spoke with Robyn Gough, who runs a small digital design business in Australia. Between September 30 and October 28, her servers delivered 2.45 TB of data, and 97.6% of it went to Meta’s crawlers. That included 94,154 automatic downloads of her graphics files. Her host told her the crawler traffic may have contributed to a resource spike that took her sites down for several hours, and customers contacted her with database errors when they tried to download purchases.
Gough summed up the problem better than I could: “Micro businesses like mine have no developer and no security team, and we only learn we need to block a crawler once the bill or the outage arrives.”
Meta’s response to both outlets was the same. A spokesperson said website managers can decide how its crawlers interact with their sites through tools like robots.txt, and that Meta respects the preferences site owners set. Paulus doesn’t believe it was intentional, but either way, the bill and the downtime land on the business owner.
Know Which Meta Bots Are Visiting Your Site
Meta doesn’t have just one crawler, which tripped up both business owners. Its Meta Web Crawlers documentation lists several, each with a different job:
- FacebookExternalHit fetches a page when someone shares your link on Facebook, Instagram or Messenger, so it can show the title, description and thumbnail image. Block this one and your link previews break.
- Meta-ExternalAgent crawls the web “for use cases such as training foundation AI models or improving products by indexing content directly.” This is the bot that hit Paulus first.
- Meta-WebIndexer crawls to improve Meta AI search results. Meta says allowing it helps it cite and link to your content in Meta AI’s answers.
- Meta-ExternalAds crawls for advertising and other business products.
- Meta-ExternalFetcher fetches individual links at a user’s request, including for AI agents completing tasks, and Meta says it may bypass robots.txt.
Why Robots.txt Alone Didn’t Stop Them
Robots.txt is a small text file at the root of your site that tells crawlers what they may and may not read. It’s the first thing Meta points to, and you should have one. But both stories show its limits:
- It isn’t instant. Meta’s own documentation says to allow up to 24 hours for changes to take effect, because its crawlers may cache your robots.txt that long. When your site is already at 80% CPU, a day is a long time.
- Each bot needs its own rule. Blocking Meta-ExternalAgent did nothing to stop Meta-WebIndexer. Fast Company reports that the day after Gough blocked the first bot, requests from the second jumped from under 100 to about 6,700.
- Some bots can ignore it. Meta says Meta-ExternalFetcher, and FacebookExternalHit during security checks, may bypass robots.txt.
What actually stopped the bots for Paulus was a rule her host gave her for the site’s .htaccess file, a server configuration file that blocks the requests before WordPress ever loads. That’s the difference between asking a bot to leave and locking the door.
How I’d Protect Your Site This Week
Here’s what I recommend for any small business site, and especially online stores, where cart and checkout links give a crawler endless URLs to loop through:
- Check your hosting dashboard. Look at CPU, bandwidth and visit numbers for the last month. A sudden spike with no matching rise in sales or inquiries is a red flag.
- Look at who’s visiting. If you use Wordfence, its live traffic view shows the user agent and IP of each request, which is how Paulus spotted the problem. Your host’s access logs show the same thing.
- Decide which AI bots you want. If you’d rather not have your content used to train AI models, disallow Meta-ExternalAgent in robots.txt. If you’d like a shot at being cited in Meta AI answers, you might keep Meta-WebIndexer allowed. Either way, leave FacebookExternalHit alone so your social posts keep their previews.
- Keep bots out of carts and checkout. There’s no reason for any crawler to be adding and removing items from your cart. Disallow those paths and block them at the server or firewall level.
- Add a firewall in front of your site. A server rule or a service like Cloudflare can block or slow bots right away, instead of waiting a day for robots.txt to be re-read. Paulus told Business Insider she’s exploring Cloudflare’s human verification tools for exactly this reason.
- Ask your host what’s included. Many hosts have server-side bot protection. Find out before you get the 80% warning email.
For reference, a basic robots.txt rule that tells Meta’s AI training crawler to stay off your site and keeps all crawlers out of a WooCommerce cart looks like this:
User-agent: meta-externalagent
Disallow: /
User-agent: *
Disallow: /cart/
Disallow: /checkout/Treat that as a starting point. The server-level block does the heavy lifting when a bot is already hammering your site.
Should You Just Block Meta Entirely?
I wouldn’t, and neither would the business owners in these stories. Both told Fast Company they rely on Facebook and Instagram to reach customers. Blocking everything Meta sends can break your link previews and possibly interfere with ad features. The smarter move is to be specific: allow the bots that help your business, block or slow the ones that don’t, and make sure no bot can take down your site.
This also isn’t only about Meta. Fast Company cites Cloudflare data showing bots made up 55.1% of U.S. web traffic over the last four weeks. Every one of those bots uses the same hosting resources as your paying customers.
Want Us to Check Your Site’s Bot Traffic?
This is the kind of thing we handle for our clients so they never have to find out the hard way. Contact Final Web Design and we’ll review your hosting usage and traffic logs, set up sensible robots.txt and firewall rules for AI crawlers, protect your cart and checkout pages and give you a clear, up-front plan to keep your site fast for real customers.
Sources: Fast Company, “Meta web crawlers crash small business websites as AI bots surge” (Nov 2, 2026); Business Insider, “An artist says Meta’s bots overwhelmed her website and left her scrambling to protect her business” (Nov 3, 2026); Meta for Developers, “Meta Web Crawlers”.
