Google’s dominance in search has quietly become the foundation of its power in generative AI, because the same infrastructure that indexes the web for discovery is increasingly used to harvest content for machine learning. That dual use gives Google a structural edge over rivals and leaves publishers with a false choice between visibility and exploitation. If the Internet is going to stay open and competitive, I believe regulators and platforms will have to force a clean separation between search crawlers and AI crawlers.
The stakes are not abstract. Publishers are already reporting that Google’s AI products can reproduce their work while siphoning away traffic, and antitrust authorities are probing whether this behavior entrenches a single hyperscaler’s dominance. Crawler separation is emerging as the simplest, most technically realistic way to restore balance.
How Google’s search power became an AI weapon
Google’s search engine is still the front door of the web for billions of people, and that reach is what makes its AI strategy so potent. The company’s core index, built by Google over decades, is a case study in scale advantage and network effects, because Google and Because Google Cloud handle far more search queries and data than any rival. That scale does not just improve search quality, it also supplies a vast corpus of text, images, and interactions that can be repurposed to train and ground generative models.
Publishers understand that they cannot realistically walk away from this traffic. Reporting on Google’s market position notes that Google search dominance leaves sites with little choice about whether their pages are crawled, because losing placement in results can devastate advertising and subscription revenue. That dependency is exactly what turns a neutral indexing tool into leverage for AI: if blocking the crawler means disappearing from the modern Internet, then any “consent” to AI use is compromised from the start.
The blurred line between discovery and extraction
At the technical level, the problem is deceptively simple. The same content that Google scrapes for search indexing is also used for inference and grounding in products like AI Overviews, which means a single visit by Google can power both traditional blue links and AI answers that compete directly with the underlying sites. When a user types a question and gets a synthesized response that quotes or paraphrases publisher content, the line between helping people discover information and extracting value from that information has effectively vanished.
Industry voices like Jan have warned that this dual use distorts competition by letting Google’s generative AI applications enter into direct competition with publishers while still relying on their work as raw material. Analysis of the UK policy debate notes that mandatory crawler separation is not just a technical tweak, it is a way to prevent the Internet from becoming an extraction layer for a single hyperscaler’s dominance. When the same crawl powers both search and AI, any attempt to opt out of one becomes an opt out of both, which is why the current setup is so skewed.
Why publishers cannot meaningfully opt out today
On paper, Google offers controls that let sites block AI training while remaining in search, and Jan and other advocates acknowledge that these tools exist. In practice, publishers say the settings are confusing, incomplete, and difficult to monitor, which is why Jan argues that Publishers’ reluctance to block Google because of its dominance in search gives Google an unfair competitive advantage in the market that its competitors cannot match. When the same user agent handles both indexing and AI, any misconfiguration or ambiguity tends to default in favor of more data for Google, not more control for the site owner.
Reporting on the company’s recent policy changes notes that, technically, Technically, Google’s existing tools let publishers block AI training, but in practice those controls do not fully shield traffic or content from being used in generative features. Separate analysis of AI scraping explains that traditional bots like Search and Googleb were designed for ranking pages, while newer AI crawlers are optimized for large scale data collection, which often ignores the economic impact on the sites being scraped. Without clear user agent separation, publishers cannot reliably tell which visits are for search and which are for AI, so they cannot make informed choices.
Regulators are starting to connect AI scraping and antitrust
Regulators in Europe have begun to treat AI data collection as a competition issue, not just a privacy or copyright question. Authorities in the European Union have opened a fresh antitrust probe into Google over the content it uses for AI, with By KELVIN CHAN, Business Writer in LONDON reporting that investigators are examining whether the company’s practices shut out rival AI model developers. A related inquiry in the EU is focused on Google’s use of the Googlebot crawler to collect data for AI models, with one summary noting that According to reports, the investigation centers on whether this gives Google’s AI models an advantage that competitors cannot replicate.
These probes are not happening in a vacuum. Cloudflare’s own traffic analysis found that Googlebot Top AI when it comes to successful requests for HTML content from leading AI crawlers, which suggests that Google’s search infrastructure is already the primary pipeline for AI data collection. When a single company’s crawler dominates both search indexing and AI scraping, it becomes difficult for regulators to separate normal competitive behavior from conduct that entrenches monopoly power.
Why mandatory crawler separation is the cleanest fix
Against this backdrop, Jan and other advocates have converged on a straightforward remedy: force a hard split between search crawlers and AI crawlers, with distinct user agents, policies, and enforcement. In a widely shared analysis, Jan argues that Jan sees crawler separation as the only realistic way to give publishers a genuine choice about AI without sacrificing search visibility. A related commentary stresses that mandatory separation is not about punishing one company, it is about preventing the Internet from becoming an extraction layer for the benefit of a single hyperscaler, and that Separate the crawlers, and the Internet gets a fairer playing field.
Cloudflare has already moved in this direction by giving customers tools to block AI bots by default, framing its update as a shift From Default Blocking to Opportunity Cloudflare, where Their customers can choose to share content or demand payment. Cloudflare has also introduced an AI Audit feature that lets site owners inspect logs and see exactly when AI bots hit their pages, with one practitioner describing how they used Cloudflare Audit tools to track AI bot activity through server logs. These experiments show that once crawlers are labeled and separated, it becomes far easier to enforce policy, negotiate licenses, or even build new business models around controlled data access.
