Comment on Internet Archive breached again through stolen access tokens
towerful@programming.dev 4 weeks ago4 years ago (best number I can find, considering IAs blog pages are down) IA used about 50 petabytes on servers that have 250 terabytes of storage and 2gbps network.
From this, we can conclude that 1 TB of storage requires 8mbps of network speed.
Let’s just say that average/all residential broadband has spare bandwidth for 8mbps symmetrical.
We would need 50,000 volunteers to cover the absolute minimum.
Probably 100k to 200k to have any sort of reliability, considering it’s all residential networking and commodity hardware.
In the last 4 years, I imagine IA has increased their storage requirements significantly.
And all of that would need to be coordinated, so some shards don’t get over-replicated
rottingleaf@lemmy.world 4 weeks ago
This seems to confirm my critique of “manual” solutions with torrents and such offered in other comments, resulting in the idea shortly described in the comment you were answering.
Yes, this would require a lot of people, but some would contribute more and some less, just like with other public P2P solutions.
From my POV the biggest problem is synchronizing indexes (similar to superblock maybe) of such a storage, and balancing replication based on them, in a decentralized way. Because it would seem that those indexes by themselves would be not small.
There should also be all the usual stuff with controlling data integrity.
I think it’s realistic to attract many volunteers, if the thing in question will also be the user client, similar to Freenet and torrents socially, and bigger storage will allow them to faster get things they access more often, as a cache. But then balancing between that and storing necessary, but unpopular parts of the space, is a question.
I think I need to read up.
locuester@lemmy.zip 4 weeks ago
There are really good, incentivized versions of decentralized storage networks. Unfortunately discussions about them are stigmatizes under the “crypto” umbrella so the mere mention typically gets you buried.
If you have an open mind, check them out!