Meta Can Crawl Your Business Site From October 30: The Clause and the robots.txt Lines

Meta changed its commercial terms so that, from October 30, 2026, any business with a Meta account authorizes Meta to crawl the websites tied to that account. After criticism reported on October 16 and 17, Meta removed the word "train" from the clause and narrowed the permitted uses. The crawling permission itself stayed, and Nieman Lab reports business accounts cannot decline the update. If you run ads, a Facebook Page or an Instagram profile for a business, below are the clause, the crawlers it may involve, and the robots.txt lines I would add before the 30th.

How the Meta commercial terms clause reads now

The story broke when Nieman Lab flagged new language in the draft terms. The first version said Meta could use scrapers and other automated tools to "access, index, cache, analyze, collect, retrieve, or crawl" the domains and pages associated with your products or services, and that Meta "may use any content" from them to "provide, develop, improve, research, train, and maintain" its products. Nieman Lab noted that virtually every news organization has at least one Meta business account, so the clause reached most publishers.

On Friday, October 16, Meta published a revision. I read the live preview page on Meta's own site, and the clause now reads that Meta may pull updates from those domains "in order to provide, develop, improve, research and maintain advertising and business products." The comparison:

Meta's spokesperson told Nieman Lab the revision was made because "it wasn't clear in the initial consolidation of terms that robots.txt preferences are respected," and that the terms give no new rights that supersede them. Meta also said the language only applies to its ads and business products, and that it came from folding its Ad Creative Generative AI Terms into the general commercial terms. TechCrunch and Engadget carried the same statement.

Why lawyers are still uneasy

Three legal experts told Nieman Lab the edit does not fix the problem. James Grimmelmann of Cornell said "develop...advertising and business products" is "broad enough to drive a truck through" and that nothing restricts the "associated with your products or services" scope. Matt Topic, an attorney at Loevy & Loevy, read "develop" as broad enough to include "train", because "business product" is not a defined term.

The robots.txt gap is the practical problem. Benjamin Sobel of the University of Wisconsin likened robots.txt to a sign that says "Don't look at my house": it states a preference, and unless a law makes it enforceable, it can be ignored. Meta's promise to respect it lives in a spokesperson's statement. The contract text says nothing about it. Meanwhile the European Publishers Council said it would petition EU regulators, and the News/Media Alliance called the terms "outrageous and probably illegal."

The exit is also narrow. The terms say that if you do not agree to an update, you "may stop all access or use of our Products for business or commercial purposes." I would not call that a real choice for a company that sells through Instagram ads.

Which Meta crawler this actually touches

The terms do not name a crawler. Meta's web crawlers page, last updated May 21, 2026, lists five, and the descriptions are the only mapping available:

• meta-externalagent crawls "for use cases such as training foundation AI models or improving products by indexing content directly."
• meta-externalads crawls the web to improve advertising and other business products and services. This one matches the new clause best, though Meta has not said so.
• meta-webindexer improves Meta AI search results. Meta says allowing it "helps us cite and link to your content in Meta AI's responses."
• meta-externalfetcher fetches links at a user's request, including to help AI navigate websites for users, and Meta says it "may bypass robots.txt rules."
• facebookexternalhit builds link previews when someone shares your URL, and "might bypass robots.txt" for security or integrity checks.

My read: the clause most likely rides on meta-externalads, and the old "train" wording would have pointed at meta-externalagent. That is inference from the descriptions, not something Meta confirmed. It is also why I would block both.

Robots.txt lines to add before October 30

Meta's page shows the pattern for blocking: add a disallow for the crawler you want to stop. Here is the version I would use for a business site that wants no AI training and no ad crawling, but still wants link previews and Meta AI citations:

User-agent: meta-externalagent
Disallow: /

User-agent: meta-externalads
Disallow: /

User-agent: meta-webindexer
Allow: /

The last rule is optional, since crawling is allowed by default. I include it so the intent is written down.

Robots.txt is a plain text file at the root of your site (yourdomain.com/robots.txt) that tells crawlers, the bots that fetch pages automatically, what they may fetch. The name after User-agent is how each bot identifies itself. WordPress SEO plugins and Shopify's robots.txt template both let you add rules, and a developer or your host can do it in minutes. If you do not use Meta products for business, the new terms do not apply to you.

Leave facebookexternalhit alone: it builds the title and thumbnail when someone shares your link. Meta says crawlers may cache your robots.txt for up to 24 hours, so deploy this by October 29. Then check your server logs for the user agents. If they keep showing up after a day, robots.txt is not holding. If your host only gives you a log viewer, search it for the same names or ask your host to check:

grep -iE "meta-external|meta-webindexer|facebookexternalhit" access.log | awk '{print $1, $NF}' | sort | uniq -c | sort -rn | head

What I would do before October 30

1. Read the preview page once. It is the version that takes effect on the 30th.
2. Add the robots.txt lines above, or the equivalent for your CMS, and confirm they load at /robots.txt.
3. List what is on the domains tied to your Meta accounts. The clause covers pages "associated with your products or services," so my reading is that the website you link from your Page or ad account is in scope, along with a product catalog or a staging site on that domain. Meta does not define the phrase. Gated content should sit behind real authentication, since the clause also lets Meta "create test credentials."
4. Tell your agency. The terms say you will ensure that any third party acting for you abides by them, so whoever runs your ads has accepted the same clause on your behalf.
5. Keep a dated copy of your robots.txt. If Meta's behavior ever disagrees with its statement, a record helps.

If you want an outside look at what your site exposes to AI systems, that is part of my AI readiness assessment. For more on Meta's AI work, I covered Muse Spark Open earlier this month.

Frequently asked questions

When do the new Meta Commercial Terms take effect?

October 30, 2026. Meta published the updated terms on a preview page, and Nieman Lab reports business accounts cannot decline the update before then, and the terms say continuing to use Meta's products for business after notice of an update counts as agreement.

Did Meta remove AI training from its commercial terms?

It removed the word "train" and limited the permitted uses to advertising and business products. Legal experts quoted by Nieman Lab say "develop" and "business products" are broad enough to cover AI training. Meta says its crawler use has not changed, according to Nieman Lab.

How do I block Meta's AI crawler?

Add a Disallow rule under User-agent: meta-externalagent in your robots.txt, and do the same for meta-externalads. Meta says changes can take up to 24 hours to apply. Meta-ExternalFetcher, which acts on a user's request, may bypass robots.txt.

Will blocking Meta's crawlers break my Facebook link previews?

Not if you leave facebookexternalhit alone. That crawler fetches the title, description and thumbnail when your link is shared. The robots.txt lines above only target meta-externalagent and meta-externalads, and they allow meta-webindexer, which feeds Meta AI answers.