WhoLooksHere
@WhoLooksHere@lemmy.world
- Comment on FediDB has stoped crawling until they get robots.txt support 1 month ago:
Okay,
So why should reinevent a standard when one that serves functionally the same purpose with one of implied consent?
- Comment on FediDB has stoped crawling until they get robots.txt support 1 month ago:
From your own wiki link
robots.txt is the filename used for implementing the Robots Exclusion Protocol, a standard used by websites to indicate to visiting web crawlers and other web robots which portions of the website they are allowed to visit.
How is f3didn not an “other web robot”?
- Comment on FediDB has stoped crawling until they get robots.txt support 1 month ago:
Robots.txt started I’m 1994.
It’s been a consensus for decades.
Why throw it out and replace it with imied consent to scrape?
That’s why I said legally there’s nothing they can do. If people want to scrape it they can and will.
This is strictly about consent. Just because you can doesn’t mean you should yes?
- Comment on FediDB has stoped crawling until they get robots.txt support 1 month ago:
Why invent implied consent when complicit consent has been the standard in robots.txt for ages now?
- Comment on US adds web and gaming giant Tencent to list of Chinese military companies 2 months ago:
Tik Tok is the only tencent company I know if that’s been labeled a national security risk.
- Comment on US adds web and gaming giant Tencent to list of Chinese military companies 2 months ago:
Tik Tok is what they’re referring to
- Comment on Almost the entire US South is now being blocked by Pornhub 2 months ago:
You are using a device capable of finding the information you so desire. Instead you decided to contribute you whataboutism.