Reported about 1 year ago
Reddit announced its plan to update its web standard in order to block automated data scraping on its platform. This decision comes after reports revealed that AI startups were skirting the rules to extract content for their systems. The company will be enhancing the Robots Exclusion Protocol, or 'robots.txt,' to limit crawling access, implement rate-limiting, and block unknown bots and crawlers. The move aims to prevent tech firms from using content without permission to train AI algorithms and create summaries. Reddit assured that researchers and organizations like the Internet Archive can still access its content for non-commercial purposes.
Source: YAHOO