Comment on FediDB has stoped crawling until they get robots.txt support
rimu@piefed.social 3 weeks agoOk if you want to focus on that single phrase and ignore the whole rest of the page which documents decades of stuff to do with search engines and not a single mention of api endpoints, that's fine. You can have the win on this, here's a gold star.
WhoLooksHere@lemmy.world 3 weeks ago
Okay,
So why should reinevent a standard when one that serves functionally the same purpose with one of implied consent?