Comment on I'm Starting A Search Engine For The Fediverse
loobkoob@kbin.social 1 year agoIdeally, OP's crawlers will just come from their own instance that other instance owners can defederate from if they want to opt out.
Comment on I'm Starting A Search Engine For The Fediverse
loobkoob@kbin.social 1 year agoIdeally, OP's crawlers will just come from their own instance that other instance owners can defederate from if they want to opt out.
lautan@lemmy.ca 1 year ago
Yeah that would be the case.
scrubbles@poptalk.scrubbles.tech 1 year ago
That’s a good idea. Listen to public data being broadcasted out, then you aren’t worrying people with scraping or anything. It would only be from go live onward, but you would just be listening to the protocol.
TimLovesTech@badatbeing.social 1 year ago
For that to happen on an instance organically users would need to visit all these instances/communities. To speed that up you would need a bot to do all.that “seeding” for you. That brings you full circle to the server resources on bigger instances.
This seems like an opt-in, not an opt-out activity.