Is your website blocking AI search? LadyBugz checked five sites to find out

Roodepoort, South Africa. LadyBugz, a B2B (business-to-business) marketing agency, has checked whether five websites it manages turn away the bots that AI tools send out to read web pages. It found no sign of AI search bots being shut out on any of them. The check followed a reminder from Google that a robots.txt file controls less than many owners assume.

Three weeks of visitor records

LadyBugz read three weeks of server logs, the running record of every visit, for four of the five sites and confirmed which visitors really were the bots they claimed to be, using the address lists the bot owners publish.

On ladybugz.co.za, Google's crawler made more than 1,600 confirmed visits and was refused none of them. OpenAI's search crawler came by more than 1,000 times.

Real bots met very little resistance. Some AI training bots were slowed by a speed limit, mostly on one site, and a handful hit one-off refusals. They still got through. Every refused visit under Google's name came from an impostor borrowing it. No changes were needed.

What Google said

Robots.txt is a plain file at the root of a site that crawlers (automated programs that fetch web pages) read before going further. Search Engine Journal reported this month that Gary Illyes, from Google's Search team, called it "an access preference protocol" on Google's Search Central video series. He added that it "doesn't dictate what a system does with the data after it has already crawled it."

So blocking a bot today does nothing about pages it has already copied.

Which bots do what

OpenAI, Anthropic and Perplexity each run separate crawlers for separate jobs, according to their own documentation.

One group collects pages that may be used to train AI models, such as OpenAI's GPTBot and Anthropic's ClaudeBot. Google handles training through a setting called Google-Extended, which covers use in its Gemini AI models, and Google says it has no effect on Google Search.

The other group fetches pages so an AI tool can answer a question and cite where the answer came from. OpenAI states that sites which opt out of OAI-SearchBot "will not be shown in ChatGPT search answers." PerplexityBot plays that role for Perplexity. Google's guidance says a page must be in its ordinary search results to appear in AI Overviews, the AI summaries above many Google results, so blocking Googlebot loses you both.

Blocking a training bot is your call. Block a search bot and AI tools have no way to point a buyer to you.

"When we pulled the logs I half expected to find something to fix. We didn't. The blocks I worry about are the ones added in a hurry by someone who never checked which bots they catch. Before you ask your web person to block the AI, find out which bots read you and which ones recommend you," says Noleen Thompson, founder of LadyBugz.

How to check your own website

Type your web address followed by /robots.txt into a browser, for example yourcompany.co.za/robots.txt. A line starting "User-agent:" names a bot. A line under it reading "Disallow: /" tells that bot to stay off the whole site. If OAI-SearchBot, PerplexityBot or Googlebot is named with "Disallow: /", that bot is shut out. "User-agent: *" with "Disallow: /" shuts out every bot, Google included.

A clean file is only part of the check. Ask your web host or developer whether a firewall or security add-on, software that screens visitors before they reach the site, turns away bots by name.

Getting found and quoted by AI tools is called answer engine optimisation (AEO), and for most B2B firms it starts with letting the right bots in. LadyBugz has seen it work: since late September at least four people have contacted the agency after finding it through AI tools, one of them after asking an AI tool who to hire.