Publishers reassess their block-all-AI-bots stance

Some publishers are reassessing their block-all-AI-bots stance.

The debate over whether to block AI bots has been simmering with publishers for more than a year, as AI crawlers proliferated and publishers weighed traffic, licensing leverage, and legal risk. There is still no clear consensus, but four publishing execs whose companies have previously taken a hard line on blocking bots told Digiday that shifts in how AI engines surface and route their content are forcing them to question whether blocking is the right call. They’re now questioning whether they’re sacrificing the ability to monetize traffic that could be driven to them via AI engines. 

Execs told Digiday that their strategies need to be more nuanced compared to a year or so ago, when AI bot traffic was on the rise. Especially after (at least one exec said) they’re seeing more traffic from AI platforms like Perplexity, Anthropic and ChatGPT, though they did not provide specific figures.

Will Allen, vp of product at Cloudflare, said its data shows a continuous increase in AI bot traffic to websites.

“We’re seeing broadly — and also what we’re hearing anecdotally — there’s growth [in AI bot traffic], and it comes from desire in content for training as well as content for inference and search,” Allen said. 

Anecdotally, Allen said he still sees publishers take an aggressive approach to blocking AI bots. Publishers need to understand which bots are accessing their sites, how frequently, and what the value exchange is (such as referral traffic or conversions), Allen said. Then publishers need to figure out whether they are getting value from these visits, and determine which particular crawlers they want to give access to their content.

This year, new tools from companies like Cloudflare, Tollbit, Fastly and DataDome mean publishers have a better sense of which AI bots are coming to their sites, and how often.

In a DataDome report out last month, LLM crawler traffic quadrupled across the cybersecurity company’s customer base in 2025, rising from 2.6% of verified bot traffic in January to over 10.1% by August. DataDome detected nearly 1.7 billion requests from OpenAI crawlers in a single month.

Some publisher SEO teams have pushed back internally on bot blocking, a topic that surfaced at Digiday’s Publishing Summit town hall in Miami, Fla. in September.

“We started to see all this increased referral traffic coming from LLMs, and then our SEO team said, ‘Please don’t block these guys,’” said one publishing exec at the DPS town hall. “We block bot traffic and now we’re concerned that that’s hurting us… Are we losing out on traffic we could be getting in that time when we’re losing traffic? Maybe we need to turn this off… We can assume most traffic [to our sites] is going to be coming from LLMs in the next few years, maybe sooner.”

That exec said their company’s AI committee analyze its bot traffic quarterly to see if it’s necessary to reevaluate its strategy. Currently, their company redirects AI training bots through Tollbit, but lets AI search bots through. AI training bots crawl the web for data to feed into LLMs, while AI search bots (also called RAG AI bots, or agents), retrieve information in real time in response to user prompts in AI products like ChatGPT by searching the web — and often include links or citations to the original sources, such as publishers’ sites. AI companies have to self-identify these bots for publishers to know what they are (for example, OpenAI’s RAG AI bot is called “ChatGPT-User” while “GPTBot” is its training data bot).

The publisher’s site receives weekly AI bot traffic “in the millions,” the exec said. Blocking that traffic may mean not having their content surfaced in AI products like ChatGPT, potentially missing the opportunity to get in front of the AI chatbot’s 800 million weekly users, they said.

“No decisions have been made yet [on whether to soften its former bot-blocking stances],” said another publishing exec who spoke to Digiday under the condition of anonymity. “[We] leaned toward blocking AI bots because the value exchange wasn’t clear [and] content was being ingested without compensation or attribution. Today, it’s become a more nuanced discussion. We’re weighing the risks of free use against potential opportunities around visibility, attribution and licensing… is it time to open that up to not forsake potential traffic when SEO traffic isn’t improving?”

A third anonymous publishing exec said their company is still trying to determine the broader implications, and how it would “actually work” to allow some AI bots in and not others. Their company has blocked all AI bots for the past two years.

“The conversation is on the edge now. What is it that we want the LLMs to know about? Seems self-defeating to block 100% of everything when there are things we want the world to [see what we publish],” they said. “Much of our thinking’s evolution has had to do with realizing that bucketing all content as ‘content writ large’ is a bit simplistic, maybe knee-jerk. Not all of a publisher’s content should be valued the same.”

It’s also just part of the natural evolution of getting more comfortable with AI, the third publishing exec said. As their company uses the technology more internally, other execs at the company are gaining a better understanding of the value exchange. “Discussions of ‘tools in exchange for content’ are no longer non-starters,” they said.

At the Prebid Summit on Tuesday, Paul Bannister, chief strategy officer at Raptive, spoke about this dilemma during an onstage session: “Most of the LLMs have multiple bots, and you have to understand which ones you want to block in which scenarios. And it’s not as straightforward [as] ‘always block.’ It’s figuring out the nuance of, what do you need to do in what situation, and what things might you be able to make money from in the future, and what things are unlikely to ever… get paid for? And figure out that gap.”

What we’ve heard

“I would define us as a free rider. Like, I’m not the guy that’s going to go off and sue OpenAI. We’re too small, and it’s too time consuming… You want to be aligned with all those guys [like Cloudflare and Tollbit], so you’re getting close to it.”

—An anonymous CEO at a media company on their AI bot blocking strategy.

Regulator CMA sharpens focus on AI Overviews and publisher traffic

U.K. regulator the Competition and Markets Authority (CMA) has officially put Google’s AI search features that affect publisher traffic — AI Overviews, AI Mode, and Discover — under the microscope. 

The CMA formally designated Google as having Strategic Market Status on Oct. 10. In a nutshell, it’s a procedural win with practical upside. For publishers, it opens the door for the CMA being able to push for minimum attribution standards inside AI answers, link prominence rules, and perhaps even testing obligations before widescale UI changes. Bottom line: it would mean regulatory recognition that AI summaries siphon clicks. 

With any regulatory changes, there are some caveats, one being how long it takes to get any remedies into play, but it’s a vital step in the right direction, according to the Professional Publishers’ Association (PPA). Also, Gemini wasn’t included within the scope, meaning the CMA will need to guard against scope-shift workarounds. 

The PPA, which counts Hearst UK, Condé Nast and Immediate Media among its members, has submitted in-depth evidence to the CMA that documents a direct correlation between publisher citations in Google AI Overviews increasing, and click-throughs dropping — what the PPA referred to as “the crocodile phenomenon” in its evidence to the CMA.

“The big problem we wanted to resolve is that, as things currently stand, you’re seeing claims from Google saying that the AI products are actually good for publishers, and increase their traffic, but not publishing any of the data that could prove that,” said Eilidh Wilson, head of policy and public affairs at the PPA. 

This crocodile trend that the PPA has identified in its evidence shows search case examples within its membership, where publishers are seeing an increase in appearances in search and a decrease in traffic, she stressed. “So the impressions and rankings go up, demonstrating that their content is more desirable to the [AI] product, but the traffic is going down, so the actual reward for the publisher is diminishing and in some cases non-existent,” she told Digiday.

Here is what the PPA is lobbying the CMA hard for about Google’s level of transparency: 

• Publish separate crawl reports that clearly distinguish indexing crawls from crawls used to generate AI Overviews, AI Mode, or any future AI summary/aggregation features.
• Add Search Console visibility showing when and where a publisher’s content is surfaced inside AI-powered SERP elements (AI Overviews, AI Mode) and AI Overviews within Discover.
• Provide clear acquisition source labeling in Google Analytics so AI-driven referrals are identifiable.
• Mandate clear attribution and clickable source links whenever publisher content powers AI Overviews, AI Mode, or Discover.

In November, the CMA will publish a draft set of rules for how Google should behave and then ask publishers and Google for feedback. Publishers aren’t asking to kill AI snippets, noted Wilson, rather that they want the AI features to work for them too, with clear credit and links that actually send readers to their sites. — Jessica Davies

Numbers to know

$500 million: The amount The Los Angeles Times wants to raise ahead of its plans to go public by 2027.

$130 million: The amount Bustle Digital Group is expecting to earn in revenue this year.

226: The number of people laid off at People Inc. (formerly Dotdash Meredith), representing 6% of its workforce.

150: The number of people expected to be let go at NBC News, which has about 2,000 employees.

What we’ve covered

Overheard at the 2025 Prebid Summit

• This year’s edition of the annual, publisher-focused ad tech conference had a particular sense of urgency given Google’s AI Overviews looming over the media business.
• Some of the industry’s OGs were on hand to clarify controversial moves, and to tease soon-to-be revealed solutions.

Get the key takeaways from the event here.

Why 1440 is evolving from a newsletter company to a destination of explainers

• Daily email newsletter 1440 is evolving into what it’s calling a “knowledge collective,” creating a library of explainers on its own website on topics that its readers want to learn more about, as the way readers find and consume information is changing.
• There are now over 200 Topics pages across categories like business & finance, science & technology, health & medicine, world history and society & culture. The goal is to get to 10,000 topics.

Read more about 1440’s repositioning here.

Publisher alliance Ozone makes a larger play for U.S. advertisers

• The publisher alliance Ozone is on a growth tear in the U.S., bringing on companies like Wall Street Journal, New York Post, the BBC (US) and CNN.
• What began as a pure inventory play in 2018 has since evolved into a broader audience-connection platform that can map audience content consumption patterns across the publishers to provide a stitched-together view of readers’ behavioral patterns. To date, the model has driven more than £400 million ($532 million) in incremental spend back into the U.K. publishing sector.

Read more about Ozone’s U.S. push here.

Podcast industry juggles using AI tools with maintaining medium’s intimacy

• While the podcast medium prides itself on its intimate relationship with its listeners and the personal connection between host and audience, AI tools are increasingly used to translate and edit podcast content.
• Podcasters are grappling with striking a balance between protecting the nature of podcasts and evolving the space for the AI era.

Learn more about how podcasters are weighing the use of AI technology here.

The Independent maps four-pillar growth plan for the AI era

• The Independent is leaning on four growth pillars — its talent and IP-led verticals, U.S. expansion, e-commerce and internal AI products — to withstand the zero-click era and the knock-on effect that may have on display ad revenues.
• Each pillar plays a different role, from driving new revenue streams to testing how AI can be turned from a threat to an opportunity. 

Read the conversation with The Independent’s CEO Christian Broughton here.

What we’re reading

Plans for NBC News revealed

Cesar Conde, chairman of the NBCUniversal News Group, shared in an internal memo what the future of NBC News will look like, including the launch of a new subscription product later this year and expanding its sports hub, Adweek reported.

New York Times beefs up AI initiatives team

The New York Times named Mukul Devichand editor of the AI Initiatives team to lead two new user-facing projects focused on audio and search, according to MediaPost. Dylan Freedman was promoted to AI projects editor to help reporters with research and investigations.

CNN tests new creator unit

CNN’s international division is creating a new unit called CNN Creators with a weekly show and multiplatform content on AI, tech, art, culture, sport and social trends, according to The Hollywood Reporter.

Perplexity no longer taking new advertising deals

Perplexity is not taking on new advertisers (publisher participants in its revenue share model get a cut of ad revenue), and ads will not be part of the monetization strategy for its AI browser Comet, Adweek reported.

Gannett Media president on Perplexity partnership

In an op-ed, Kristin Roberts, president of Gannett Media, explained that the local media network partnered with Perplexity to surface its local news coverage in the AI platform and to generate new revenue.