Comment

Comment on I'm Starting A Search Engine For The Fediverse

TimLovesTech@badatbeing.social ⁨9⁩ ⁨months⁩ ago

Reading a post and having a bot thrashing a server indexing everything are 2 different things. If a user used the site like that they would be throttled and if repeated afterwards, banned. It is also one thing to read/interact with a site as that adds value to the site as a whole. A bot that just mass hits links cataloging everything is just a strain on the server an Admin needs to support, with no upside for the instance, as it’s a bot ingesting and no real interaction actually took place.

source

Sort:hotnew top

0x1C3B00DA@kbin.social ⁨9⁩ ⁨months⁩ ago

and having a bot thrashing a server indexing everything

This is a completely separate argument and one that we already have mechanisms for. Servers can use status codes and headers to warn about rate limits and block offenders.

It is also one thing to read/interact with a site as that adds value to the site as a whole

A search index adds value as well; that's why this keeps coming up. And, again, there are existing mechanisms to handle this. A robots.txt file can indicate you don't want to be crawled and offenders can be IP blocked

source
- Rednax@lemmy.world ⁨9⁩ ⁨months⁩ ago
  Should a dedicated search not use/index ActivityPub instead of the html interface?
  
  If so, instances can simply defederate from search engine instances. So the point you are trying to make still holds.
  
  source