TollBit, which helps publishers charge AI bots for content access, already mentioned in its latest Q2 report (published just before Atlas hit the market) that the next wave of AI visitors increasingly look like humans on sites. And the latest AI browsers are indistinguishable from humans in site logs. It’s also been referred to as headless browsing.

So why are they a new headache?

They pose a sticky problem for publishers and the third-party vendors reporting them. They’re not bots, but they’re not people either. They’re the crawlers that hit a publisher’s site for information to answer a question or prompt typed into an AI engine by individuals who are increasingly using them as their de facto search engines. So technically, they’re a bot seeking information that a human has requested. 

Figuring out how to track and value them is one of the challenges publishers face in the AI era. 

“Agents and bots mimicking humans is an anti-pattern that has dangerous implications, such as eroding advertiser trust,” said Olivia Joslin, co-founder and COO of TollBit. 

Case in point: Just over two weeks ago, digital publisher Salon had a “mid-sized” buyer shut off its entire advertising spend as a direct result of a spike in agentic visitors on the site, according to Justin Wohl, vp of strategy for Adtitude who still works for Salon — where he was previously chief revenue officer — on a contract basis. 

The buyer had received a report from Integral Ad Science, which had raised an invalid traffic (IVT) alarm on the publisher’s site. IVT is a core metric in digital advertising used to flag impressions, clicks or other activity that doesn’t come from a genuine audience. Third-party companies like IAS and DoubleVerify typically detect these on publisher sites so advertisers can be informed whether they’re spending against human eyeballs.

But the rise of agentic visitors, courtesy of the AI engines, which are now becoming the search interface for people requesting information on the internet, has meant that some publisher sites are getting swamped by crawlers trying to fulfill these RAG queries. To the tech vendors, they look like fraudulent traffic, so they flag them as such.

For now, DoubleVerify and IAS don’t split out agentic visitors; they count them within the general invalid traffic parameters. Data from DoubleVerify, published in January, showed that general invalid traffic — basically non-human visits like bots, scrapers and automated crawlers — exploded in late 2024. It nearly doubled year over year, jumping 86% in the second half of the year. For the first time ever, monthly volumes passed 2 billion ad requests from this kind of traffic. At the time, DV said the main driver was the rapid growth of AI crawlers and scrapers sweeping the internet to gather content.

“IAS has a proven track record in detecting and blocking AI bots – including ‘declared’ AI agents like those used for RAG purposes, chatbots, and sophisticated scrapers. We report these as General Invalid Traffic (GIVT), not fraud (or SIVT), so advertisers do not pay for non-human traffic media,” said an IAS spokesperson in an email statement.

It’s worth stressing that this isn’t something all publishers are seeing, but most are on high alert. “I flag it as ‘something we all need to investigate,’” said Paul Bannister, chief revenue officer of Raptive. “We’ve done a lot of digging and haven’t seen that there is enough traffic like this that it’s worth worrying about, but that may be different for different verticals. I think it’s more fear right now than reality, but worth watching.”

For publishers like Salon, however, it’s a big problem. “It’s super reactive by the buy side to say we’re not going to spend anymore, because not all of my views are agentic visitors — there’s plenty of humans in there,” said Wohl. “But rather than trying to figure out how to set the right flags and not show ads or not pay for those ads, they just take this decisive action of shutting it off completely.”

The most immediate threat to publishers isn’t just the loss of traffic — it’s the potential loss of ad revenue, stressed Wohl. Even while AI crawlers were scraping content, publishers still earned money from ads. But if buyers start saying they can’t trust the traffic because too many agent-like or bot-like visits are mixed in, they may pause spend entirely. That’s the real danger: advertisers pulling back because they no longer believe the impressions are human. Once that happens, “the publisher is totally dead in the water,” added Wohl. 

WordPress VIP, the content management system for publishers that also owns analytics platform Parse.ly, is able to track the level of AI scraping across its client base. “We see at least 90% of what’s coming from AI scrapers, and it looks like DDoS (distributed denial-of-service) attacks,” said Brian Alvey, CTO of WordPress VIP. “It looks like just these massive waves — like invasions when they come — it’s a crazy event, and you [the publisher] feel it,” said Alvery. “The ones where you won’t see it are the sneaky things that some of these deceptive bots are doing, like going to a Google or Bing search cache to take the content from there [to evade detection],” he added. (WordPress VIP is a Digiday vendor.)

Doesn’t bot blocking help? 

Well. There are publishers (People Inc. for example) that do block most bots except the ones they have deals with, like OpenAI. That strategy can work for the biggest publishers, but not all. TollBit CEO and co-founder Toshit Panigrahi says there are some negative ripple effects to blocking bots — but in isolation, that’s not enough. “Blocking alone only incentivizes bots to evade detection, resulting in an unsustainable cat-and-mouse game where site owners increasingly must attempt to spend on cybersecurity as bots get more creative in evading it,” he said.

Even Google finds it challenging to block the scrapers, he stressed, as evident in the recent Reddit lawsuit. The proliferation of AI-powered browsers (which often utilize their human users’ IP addresses) is more evidence that this game is a losing battle as consumer adoption grows, he noted. “We see the only real solution as a future where non-human site traffic is required to self-identify — something which may require regulation as well. Otherwise, the cost of determining ‘bot or not’ unfairly falls on the website owner,” said Panigrahi.

AI browser misdirection 

Alvey doesn’t believe AI browsers like Atlas and Comet yet have the scale to generate enough traffic to reliably access or replicate paywalled content at scale. But they do give AI companies convenient liability cover.

When researchers or Reddit users run tests using traps like “robots_llm.txt” files that instruct bots not to crawl, and then catch an AI company crawling anyway, that company can claim the access came from an AI browser user, not its own systems, he noted. 

Alvey said publishers and testers are increasingly setting up these honeypot-style experiments to see whether AI crawlers respect rules or ignore them — essentially checking whether these bots behave ethically or blow through the stop signs.

“It gives them [AI browsers] this interesting little bit of misdirection, or like a cloud around how did they really get that content.