Robots.txt is a file that is is accessible as part of an http request. It’s a backend configuration file that sets rules for what automatically running web crawlers are allowed. It can set both who is and who isn’t allowed. Google is usually the most widely allowed domain for bots just because their crawler is how they find websites for search results. But it’s basically the honor system. You could write a scraper today that goes to websites that it is being told it doesn’t have permission to view this page, ignore it, and still get the information
Comment on AI companies are violating a basic social contract of the web and and ignoring robots.txt
i_have_no_enemies@lemmy.world 7 months agowhat is it?
mrnarwall@lemmy.world 7 months ago
echodot@feddit.uk 7 months ago
I do not think it is even part of the HTTP protocol I think it’s just a pseudo add-on. It’s barely even a protocol it’s basically just a page that callers can look at with no really pre-agreed syntax.
wise_pancake@lemmy.ca 7 months ago
robots.txt is a file available in a standard location on web servers (website.com/robots.txt) which set guidelines for how scrapers should behave.
That can range from saying “don’t bother indexing the login page” to “Googlebot go away”.
IT’s also in the first paragraph of the article.