MY KOLKATA EDUGRAPH
ADVERTISEMENT
regular-article-logo Saturday, 28 September 2024

Reddit wants Microsoft to pay to scrape data

Last month, Reddit updated its robots.txt file to block web crawlers it doesn’t have agreements with

Mathures Paul Published 02.08.24, 08:03 AM
Reddit CEO Steve Huffman wants to block companies from scraping the site without an AI agreement

Reddit CEO Steve Huffman wants to block companies from scraping the site without an AI agreement Illustration: The Telegraph

Reddit’s chief executive officer Steve Huffman wants Microsoft and other AI search engines like Anthropic and Perplexity to pay to search the site. It comes close on the heels of striking deals with Google and OpenAI.

“Without these agreements, we don’t have any say or knowledge of how our data is displayed and what it’s used for, which has put us in a position now of blocking folks who haven’t been willing to come to terms with how we’d like our data to be used or not used,” Huffman has told The Verge. He mentioned Microsoft, Anthropic and Perplexity in particular for refusing to negotiate and said it has been “a real pain in the a** to block these companies”.

ADVERTISEMENT

Last month, Reddit updated its robots.txt file to block web crawlers it doesn’t have agreements with. Huffman said that Microsoft has been using Reddit’s data to train its AI and summarising its content in Bing results “without telling us”.

Reddit hasn’t disclosed the amount needed for deals enabling scraping from Microsoft, Perplexity and Anthropic. Reddit spokesperson Tim Rathschmidt told Ars Technica last week that Reddit has been speaking “with multiple search engines” and that Reddit’s “open to working with partners big and small”.

Taking the licensing deals route makes Reddit join the ranks of traditional media publishers that are seeking payment for letting their content feed generative AI. “I think the traditional value exchange from search engines has changed,” said Huffman. “Search and summarisation and training are merging, and the value exchange of crawling in exchange for traffic back is becoming muddied.”

Follow us on:
ADVERTISEMENT
ADVERTISEMENT