This Media Briefing covers the latest in media trends for Digiday+ members and is distributed over email every Thursday at 10 a.m. ET. More from the series →

This week’s Media Briefing looks at the catch-22 publishers face when deciding whether or not to block AI crawlers, especially as they seek more leverage in negotiations with AI marketplaces and platforms as more and more players move into this space.

• Publishers are still far from holding the upper hand when it comes to controlling and monetizing AI crawlers, but the outlook no longer looks quite so bleak.
• Perplexity announces new publisher participants in AI subscription revenue share program, The Root returns to Black ownership, and more.

Publishers may stand to gain from a blur of AI standards, protocols and marketplaces

It’s become publishers’ eternal dilemma in the AI era: to block or not to block — weighing the need to protect content against the lure of potential licensing and visibility deals. 

This debate still dominates discussions at many of the working groups established between publishers and LLMs, like the IAB Tech Lab’s last week, among others. There is still no clear one-size-fits-all solution. For publishers, the decision to block or not block AI crawlers isn’t just about principle — it’s about leverage. 

Cut off access, and they gain bargaining power, but risk invisibility. Leave the door open, and they stay discoverable, but give away value for free. Your classic catch-22. 

But there are new levers for publishers to test in the AI era. Moves by CDN vendors like Cloudflare to crack down more aggressively on bots, combined with the emergence of AI marketplaces like Microsoft’s, the push toward pay-per-usage, and intensifying competition among LLMs, are slowly shifting the conversation toward standards and licensing. Publishers are still far from holding the upper hand, but compared to a year ago, the outlook no longer looks quite so bleak.

Tech infrastructure partners providing the necessary friction

Until recently, publishers were virtually defenseless against unconsented AI scraping — with robots.txt serving as little more than a polite request most crawlers ignored. And of course, there is the trickiness of Google’s AI and search crawler overlapping, making it hard for publishers to opt out of appearing in AI Overviews without risking their search rankings. 

Now, thanks to CDNs like Cloudflare stepping in to create real friction, publishers are at least able to contemplate how much access to allow. Call it smart marketing on Cloudflare’s part, but their buy-in has shifted the outlook for publishers. When Cloudflare made blocking AI crawlers by default a possibility for publishers in July, it caused a lot of internal chatter within the major LLMs, according to a publishing exec who spoke on background due to the sensitivity of the topic. It got their attention. 

Trusted Media Brands uses Tollbit and Cloudflare’s AI bot-blocking mechanisms to put up some barriers to LLMs scraping content without payment. Though some AI bots still bypass these bot-blocking efforts, these “gates” help give TMB a bit more leverage when negotiating AI content licensing deals, according to Jacob Salamon, vp of business development at TMB.

“It asserts us legally… If there is a major settlement or a major [legal] decision, we can say, ‘Look, we’ve established those rates. We were bypassed. And now there’s much more ground for a claim,” he said.

It’s more common now for publishers to have not one, but four or five licensing LLM partners. Three publishers Digiday spoke to said they had this number, though they didn’t want to be named due to the sensitivity of the agreements. That’s not due to Cloudflare’s moves, but there has been a more open attitude towards speaking with publishers about compensation for usage of their content for RAG purposes since the default block move in July, they say. 

“There’s a collective deference to Cloudflare here, amongst publishers, for picking up the gauntlet,” said Justin Wohl, vp of strategy at Aditude and former CRO for Salon. “Their scale and positioning as a network positions them to better speak to these needs of the technology, more so than just from the business of an individual publisher,” he said. 

Robots.txt has had a much-needed upgrade in the last month. Cloudflare introduced “content policy signals” to let publishers specify whether material could be used for search, training or AI inputs, while the newly released Really Simple Licensing protocol embeds machine-readable licensing terms into robots.txt so publishers can spell out not just access rules, but how their content can be used once it’s scraped. Of course, the common flaw is that these can still be flouted by crawlers. But some believe it does at least move a step towards setting more legal parameters. 

“I think what Cloudflare is trying to do – organize blocking at the CDN level and creating friction, and hopefully creating enough friction and enough blocking – that’s the intention of trying to generate some aspect of leverage,” said a publishing exec at a media group, who requested anonymity in exchange for candor. “Right now, it’s been proven that the technological firepower of the LLMs to get around any of the mechanisms that we’re currently using is that they can do it,” they said. 

LLMs are hitting a new competitive streak 

Meanwhile, LLMs continue to rack up deals. Amazon now has an extensive roster of AI licensing partners for Alexa+ and, more recently, its shopping assistant Rufus. Google has reportedly been speaking more with publishers, as has Meta. This is partly because the AI community itself is hitting limits, stressed Brooke Hartley Moy, CEO of Infactory, a company that helps publishers monetize their content archives through AI-ready APIs.

After an early period of exponential gains, model performance has plateaued, she noted. LLMs are still groundbreaking, but ultimately constrained by the quality of the human-made content they’re trained on. Once you’ve scraped the open web, synthetic content isn’t enough to keep driving big leaps forward. That’s forcing AI builders to recognize they need publishers as partners, not just free suppliers. “There’s a feeling of, oh, actually we can’t just take and raid what we need. We need to work hand in hand with content creators and publishers, because if not, we’re going to be faced with an increasingly slow amount of pace and growth on the large language model side,” Hartley Moy said. 

Meanwhile, Microsoft has now made its intentions clear when it comes to how it wants to work with publishers. Its AI content marketplace is in its very early stages of working with a small ring of publisher partners, but its instant ability to plug into the demand side has caught publishers’ attention. 

Two publishers Digiday has spoken to in the last few weeks have said that they believe Microsoft’s message to be clear: we believe you should be compensated for the quality of your IP. 

Naturally, it’s rare for a big tech company’s motives to be altruistic. It’s not some burst of ethical good intentions towards publishers. It’s because they believe this is a massive competitive advantage if they’re running the entire global information marketplace to “grind AI,” as one publisher exec put it. They already lost the search race to Google. They don’t want to lose the AI one. 

“I expect that Microsoft’s potential struggles to be taken seriously here parallel their last few years of having Bing Chat and Co-Pilot in Edge not really take much market share versus ChatGPT,” said Wohl.  

In the next six months, we’ll likely see a blur of more standards, protocols, infrastructure changes, and a flurry of new content AI marketplaces as companies jostle to establish the future economic model for the open web. As for publishers, they know they need all the help they can get. 

“We’re outgunned from a technological standpoint, from a resources standpoint, from an engineering standpoint – and right now, even collectively, we’re outgunned,” said a senior executive at a publisher. After all, the big tech companies are chasing stakes worth trillions in market value — a scale the media industry simply can’t match. 

 “You’ve got to believe they are all building products that are going to need our content for credibility and consumer values, and if your [publication] doesn’t provide any value to them, then you have bigger questions about your viability as a business,” they added. 

What we’ve heard

“It’s in the millions… 10-30% are coming from bots.”

– A head of business development at a publisher discussing weekly traffic coming from AI platforms and bots.

Numbers to know

26.7%: The share in The Economist going up for sale by British philanthropist and investor Lynn Forester de Rothschild.

$150 million: The price Paramount paid to buy Bari Weiss’s site Free Press.

500: The number of Guardian staff that attended meetings to figure out how to combat a “transformative project” to combat the challenges of AI and referral traffic – more than half of its total editorial staff.

“Tens of millions”: The number of queries The Washington Post’s chatbot has received.

What we’ve covered

Publishers and tech giants push weekly talks on AI content use

• Amazon, Meta, Microsoft and Google all had seats at the table alongside 35 publishers at the IAB Tech Lab’s LLM working group in NYC last Thursday. 
• The clearest action point: the initiative has shifted to weekly meetings as it races to find standards on how AI uses and pays for content. 

Read more about the efforts here.

AWNY 2025: Creators emerge as the industry’s new power brokers

• At this year’s Advertising Week, the ascendance of content creators within both the ad industry and wider culture is on full display.
• This year, Advertising Week has four content tracks dedicated to creators — its highest number of creator tracks ever.

Read more about the rising presence of creators at AWNY 2025 here.

In the AI dealmaking rush, Trusted Media Brands is at the table but holding back

• Trusted Media Brands is in talks with big tech over AI licensing deals, but is holding off on signing any contract for now. 
• The sticking point is scope: tech companies are still pushing for broad access to content.

Read more about TMB’s AI dealmaking strategy here.

Inside The Economist’s plan to grow revenues in a post-search, AI-driven future

• The Economist is sketching out its roadmap for a post-search world by investing in formats that are more difficult for machines to mimic, like video and audio.
• The company is also holding a hard line against licensing deals with AI firms it views as competitors.

Read more about The Economist’s plan here.

What we’re reading

AI platform Perplexity announces publisher participants in subscription revenue share program

CNN, Conde Nast, Fortune, The Los Angeles Times and The Washington Post have all signed up to receive a share of revenue from Perplexity’s subscription model tied to its AI-powered browser Comet, Press Gazette reported.

Washington Post exec editor on the challenges of the newsroom overhaul

Washington Post executive editor Matt Murray talked about the challenges of overseeing an overhaul of the newsroom, the staff exodus, and owner Jeff Bezos’ role (or lack thereof) in these efforts, in an interview with Status.

The Root returns to Black ownership

Ashley Allison, a Democratic political strategist and CNN commentator, is acquiring The Root from G/O Media for an undisclosed sum, CNN reported.

Daily Mail launches two social publishers led by creator talent

Daily Mail has created two new arms (Newmedia and Creator Media) to become the home of social news, entertainment and lifestyle brands, according to The Media Leader.

Hearst Networks creates new business unit for digital brands

Hearst Networks has launched a business unit called Hearst Canvas (based out of its London offices) to develop content brands across digital, audio and emerging platforms, with a focus on YouTube, World Screen reported.