How much bandwidth do you suppose a crawler would use? I’d guess very little
Comment on I'm Starting A Search Engine For The Fediverse
TimLovesTech@badatbeing.social 11 months agoAs the fediverse is almost exclusively run by volunteers that are paying server bills and being admins, I could see some larger instances not taking kindly to this, especially depending on how much stress it would be putting on some already at capacity servers.
TrickDacy@lemmy.world 11 months ago
TimLovesTech@badatbeing.social 11 months ago
I was thinking more in terms of resources (number of spider threads X posts/communities/users being indexed) that would be now dedicated to a bot, not so much network traffic that is probably tiny if not downloading images.
TrickDacy@lemmy.world 11 months ago
Right, it would be an initial hit but if the bot was properly built it wouldn’t need to do full reindexing very often. I’m no expert but I think it could be done in a way that there is no noticeable spike in traffic or anything
TimLovesTech@badatbeing.social 10 months ago
That’s the thing, it would need to be done in chunks and have its revisits scheduled if you want to do a complete indexing of an instance. And for a large instance that’s a lot of DB thrashing if you aren’t spacing that out, or just sampling like “top 10 posts” or something, but that kind of data is going to make a useless search engine depending on the goal of the search engine. If you wanted to just catalog the daily top posts of the fediverse that might work, but if you want to catalog everything it’s going to take a lot of resources and a long time to make sure you’re not hammering people’s servers.
lautan@lemmy.ca 11 months ago
It will be very little if not downloading full html pages.
loobkoob@kbin.social 11 months ago
Ideally, OP's crawlers will just come from their own instance that other instance owners can defederate from if they want to opt out.
lautan@lemmy.ca 11 months ago
Yeah that would be the case.
scrubbles@poptalk.scrubbles.tech 11 months ago
That’s a good idea. Listen to public data being broadcasted out, then you aren’t worrying people with scraping or anything. It would only be from go live onward, but you would just be listening to the protocol.
TimLovesTech@badatbeing.social 11 months ago
For that to happen on an instance organically users would need to visit all these instances/communities. To speed that up you would need a bot to do all.that “seeding” for you. That brings you full circle to the server resources on bigger instances.
This seems like an opt-in, not an opt-out activity.