Comment

Comment on FediDB has stoped crawling until they get robots.txt support

It's been a consensus for decades

Let's see about that.

Wikipedia lists http://www.robotstxt.org as the official homepage of robots.txt and the "Robots Exclusion Protocol". In the FAQ at http://www.robotstxt.org/faq.html the first entry is "What is a WWW robot?" http://www.robotstxt.org/faq/what.html. It says:

A robot is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced.

That's not FediDB. That's not even nodeinfo.

source

Sort:hotnew top

WhoLooksHere@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
From your own wiki link

robots.txt is the filename used for implementing the Robots Exclusion Protocol, a standard used by websites to indicate to visiting web crawlers and other web robots which portions of the website they are allowed to visit.

How is f3didn not an “other web robot”?

source
- rimu@piefed.social ⁨2⁩ ⁨weeks⁩ ago
  Ok if you want to focus on that single phrase and ignore the whole rest of the page which documents decades of stuff to do with search engines and not a single mention of api endpoints, that's fine. You can have the win on this, here's a gold star.
  
  source
  - WhoLooksHere@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
    Okay,
    
    So why should reinevent a standard when one that serves functionally the same purpose with one of implied consent?
    
    source