It will be very little if not downloading full html pages.
Comment on I'm Starting A Search Engine For The Fediverse
TrickDacy@lemmy.world 10 months agoHow much bandwidth do you suppose a crawler would use? I’d guess very little
lautan@lemmy.ca 10 months ago
TimLovesTech@badatbeing.social 10 months ago
I was thinking more in terms of resources (number of spider threads X posts/communities/users being indexed) that would be now dedicated to a bot, not so much network traffic that is probably tiny if not downloading images.
TrickDacy@lemmy.world 10 months ago
Right, it would be an initial hit but if the bot was properly built it wouldn’t need to do full reindexing very often. I’m no expert but I think it could be done in a way that there is no noticeable spike in traffic or anything
TimLovesTech@badatbeing.social 10 months ago
That’s the thing, it would need to be done in chunks and have its revisits scheduled if you want to do a complete indexing of an instance. And for a large instance that’s a lot of DB thrashing if you aren’t spacing that out, or just sampling like “top 10 posts” or something, but that kind of data is going to make a useless search engine depending on the goal of the search engine. If you wanted to just catalog the daily top posts of the fediverse that might work, but if you want to catalog everything it’s going to take a lot of resources and a long time to make sure you’re not hammering people’s servers.