Google wants to turn the decades-old Robots Exclusion Protocol (REP) into an official internet standard — and it’s making its own robots.txt parser open source as part of the push.
While the REP is often referred to as a “standard,” it has never in fact become a true internet standard, as defined by the Internet Engineering Task Force (IETF) — the internet’s not-for-profit open standard’s organization. And that is what Google is now pushing to change.