Big publishers like The Guardian, The New York Times, and the Financial Times, as well as many Gannett papers, are either blocking or limiting Internet Archive’s access to their content, Andrew and Hanaa’ reported this week, out of concerns that AI companies will then scrape that content from Internet Archive.

This is happening even as publishers respect the nonprofit’s mission to preserve the web: “The Internet Archive tends to be good citizens,” Robert Hahn, Guardian head of business affairs and licensing, said. “It’s the law of unintended consequences: You do something for really good purposes, and it gets abused.”

• The Guardian “has taken steps to exclude itself from the Internet Archive’s APIs and filter out its article pages from the Wayback Machine’s URLs interface. The Guardian’s regional homepages, topic pages, and other landing pages will continue to appear in the Wayback Machine.”
• The New York Times is blocking the Internet Archive’s crawlers and recently “added one of those crawlers — archive.org_bot — to its robots.txt file, disallowing access to its content.” “We are blocking the Internet Archive’s bot from accessing the Times because the Wayback Machine provides unfettered access to Times content — including by AI companies — without authorization,” a Times spokesperson told Hanaa’ and Andrew.
• The Financial Times “blocks any bot that tries to scrape its paywalled content, including bots from OpenAI, Anthropic, Perplexity, and the Internet Archive. The majority of FT stories are paywalled, according to director of global public policy and platform strategy Matt Rogerson. As a result, usually only unpaywalled FT stories appear in the Wayback Machine because those are meant to be available to the wider public anyway.”
• USA Today Co. (formerly known as Gannett) told us it “introduced new protocols to deter unauthorized data collection and scraping, redirecting such activity to a designated page outlining our licensing requirements.”

People who read our story both understand why the publishers are doing this, and think it’s a bummer. “I feared this likely outcome,” artist, designer, and developer Nick Heer wrote. “Publishers’ understandable desire to control the use of their work is going to make the Internet Archive less useful because neither AI scrapers nor the Internet Archive matches the robots.txt rules at the original domain with their policies on archival websites.”

“This absolutely sucks. It’s also rational,” J. Emory Parker, a data editor at Stat, wrote on Bluesky. “And I think you’ll see more of it because AI scraping breaks the loose social contract of the internet that used to facilitate things like the Internet Archive.”

***

We’re aiming to cover events in Minneapolis through a Nieman Lab lens. This week, I spoke with Bellingcat’s lead editor, Eoghan Macguire, about the forensic visual investigations the nonprofit is doing in the city — and how it wants to partner with local news organizations to get its work to more people. And Neel talked to Crisis Ready Media’s Bryan Woolston about the new chemical irritants and “less than lethal” rounds federal agents are using at protests — and what journalists should know about them.

Beyond Nieman Lab’s walls, I also really liked this Storybench interview with The Wall Street Journal’s Brenna T. Smith about the paper’s investigation of Alex Pretti’s killing using bystander videos.

“In an incident like this, it actually is not that hard to find the footage — the footage hits you in the face,” Smith told Dan Zedek. “What is harder is trying to figure out when that footage was first posted and trying to find who originally posted it. Because the gold standard, especially when you’re trying to figure out timing for these things, is to be able to get in contact with the original poster, to be able to verify further things from there. In this case, that’s hard to do when everybody’s looking at the same story and everybody’s trying to find the same person.”