Comment on I'm Starting A Search Engine For The Fediverse
loobkoob@kbin.social 10 months agoIdeally, OP's crawlers will just come from their own instance that other instance owners can defederate from if they want to opt out.
Comment on I'm Starting A Search Engine For The Fediverse
loobkoob@kbin.social 10 months agoIdeally, OP's crawlers will just come from their own instance that other instance owners can defederate from if they want to opt out.
lautan@lemmy.ca 10 months ago
Yeah that would be the case.
scrubbles@poptalk.scrubbles.tech 10 months ago
That’s a good idea. Listen to public data being broadcasted out, then you aren’t worrying people with scraping or anything. It would only be from go live onward, but you would just be listening to the protocol.
TimLovesTech@badatbeing.social 10 months ago
For that to happen on an instance organically users would need to visit all these instances/communities. To speed that up you would need a bot to do all.that “seeding” for you. That brings you full circle to the server resources on bigger instances.
This seems like an opt-in, not an opt-out activity.