Comment on Cloudflare blames massive internet outage on 'latent bug'
FauxLiving@lemmy.world 23 hours ago
If you want a technical breakdown that isn’t “lol AI bad”:
blog.cloudflare.com/18-november-2025-outage/
Basically, a permission change cause an automated query to return more data than was planned for. The query resulted in a configuration file with a large amount of duplicate entries which was pushed to production. The size of the file went over the prealloctaed memory limit for a downstream system which died due to an unhandled error state resulting from the large configuration file. This caused a thread panic leading to the 5xx errors.
It seems that Crowdstrike isn’t alone this year in the ‘A bad config file nearly kills the Internet’ club.
AldinTheMage@ttrpg.network 17 hours ago
So the actual outage comes down to pre-allocating memory, but not actually having error handling to gracefully fail if that limit is or will be exceeded… Bad day for whoever shows up on the git blame for that function
witx@lemmy.sdf.org 17 hours ago
This is the wrong take. Git blame only show who wrote the line. What about the people who reviewed the code?
sugar_in_your_tea@sh.itjust.works 6 hours ago
If you have reasonable practices, git blame will show you the original ticket, a link to the code review, and relevant information about the change.
floquant@lemmy.dbzer0.com 17 hours ago
Plus the guys who are hired that systems don’t fail even under inexperienced or malicious employees, management who designs and enforces the whole system, etc… “one guy fucked up and needs to be fired” is just a toxic mentality that doesn’t actually address the chain of conditions that led to the situation
AldinTheMage@ttrpg.network 14 hours ago
That should also come up in a reviews also. Not trying to imply one guy should get fired as a scapegoat, just talking from experience how much it sucks to know your code caused major issues.