Cloudflare blames massive internet outage on 'latent bug'

Submitted ⁨⁨5⁩ ⁨weeks⁩ ago⁩ by ⁨tonytins@pawb.social⁩ to ⁨technology@lemmy.world⁩

https://techcrunch.com/2025/11/18/cloudflare-blames-massive-internet-outage-on-latent-bug/

Around the same time, Cloudflare’s chief technology officer Dane Knecht explained that a latent bug was responsible in an apologetic X post.

“In short, a latent bug in a service underpinning our bot mitigation capability started to crash after a routine configuration change we made. That cascaded into a broad degradation to our network and other services. This was not an attack,” Knecht wrote, referring to a bug that went undetected in testing and has not caused a failure.

source

Comments

Sort:hotnew top

Kolanaki@pawb.social ⁨5⁩ ⁨weeks⁩ ago
lImage

source
- eager_eagle@lemmy.world ⁨5⁩ ⁨weeks⁩ ago
  thanks for illustrating the corpo speak
  
  source
  - moseschrute@lemmy.world ⁨5⁩ ⁨weeks⁩ ago
    Nobody ever asks how the bug is doing
    
    source
    -> View More Comments
PiraHxCx@lemmy.ml ⁨5⁩ ⁨weeks⁩ ago
I wonder if all recent outages aren’t just crappy AI coding

source
- MagicShel@lemmy.zip ⁨5⁩ ⁨weeks⁩ ago
  Shitty code has been around far longer than AI. I should know, I wrote plenty of it.
  
  source
  - floofloof@lemmy.ca ⁨5⁩ ⁨weeks⁩ ago
    They trained it on the work of people like you.
    
    source
    -> View More Comments
  - wreckedcarzz@lemmy.world ⁨5⁩ ⁨weeks⁩ ago
    The AI was the shitty code we wrote along the way
    
    source
  - mycodesucks@lemmy.world ⁨5⁩ ⁨weeks⁩ ago
    Now… I don’t like to brag…
    
    source
  - tdawg@lemmy.world ⁨5⁩ ⁨weeks⁩ ago
    I too have looked at my earliest repos in dispair
    
    source
    -> View More Comments
  - foo@feddit.uk ⁨5⁩ ⁨weeks⁩ ago
    But, AI can do the work of 10 of you humans, so it can write 10 times the bugs and deploy them to production 10 times faster. Especially if pesky testers stay out the way instead of finding some of the bugs.
    
    source
- AbidanYre@lemmy.world ⁨5⁩ ⁨weeks⁩ ago
  Humans are plenty capable of writing crappy code without needing to blame AI.
  
  source
  - KazuyaDarklight@lemmy.world ⁨5⁩ ⁨weeks⁩ ago
    Absolutely, but it does feel like things have spiked a bit recently.
    
    source
  - tonytins@pawb.social ⁨5⁩ ⁨weeks⁩ ago
    Train on shitty code, get shitty code. Garbage in. Garbage out.
    
    source
- ThePantser@sh.itjust.works ⁨5⁩ ⁨weeks⁩ ago
  AI coding, AI compiling, AI bug testing, AI users, etc.
  
  source
- renegadespork@lemmy.jelliefrontier.net ⁨5⁩ ⁨weeks⁩ ago
  Indirectly, this was. He said this was a bug in their recent tool that allows sites to block AI crawlers that caused the outages. It’s a relatively new tool released in the last few months, so it makes sense it might be buggy as the rush to stop the AI DoS attacks has been pertinent.
  
  source
FauxLiving@lemmy.world ⁨5⁩ ⁨weeks⁩ ago
If you want a technical breakdown that isn’t “lol AI bad”:

blog.cloudflare.com/18-november-2025-outage/

Basically, a permission change cause an automated query to return more data than was planned for. The query resulted in a configuration file with a large amount of duplicate entries which was pushed to production. The size of the file went over the prealloctaed memory limit for a downstream system which died due to an unhandled error state resulting from the large configuration file. This caused a thread panic leading to the 5xx errors.

It seems that Crowdstrike isn’t alone this year in the ‘A bad config file nearly kills the Internet’ club.

source
- AldinTheMage@ttrpg.network ⁨5⁩ ⁨weeks⁩ ago
  So the actual outage comes down to pre-allocating memory, but not actually having error handling to gracefully fail if that limit is or will be exceeded… Bad day for whoever shows up on the git blame for that function
  
  source
  - witx@lemmy.sdf.org ⁨5⁩ ⁨weeks⁩ ago
    This is the wrong take. Git blame only show who wrote the line. What about the people who reviewed the code?
    
    source
    -> View More Comments
- phutatorius@lemmy.zip ⁨5⁩ ⁨weeks⁩ ago
  
  ‘A bad config file nearly kills the Internet’ club
  
  There’s no such thing as bad data, only shitty code to create it or ingest it, and bad testing that failed to detect the shitty code. The overflow of the magic config-file size threw an exception, and there was no handler for that? Jeez Louise.
  
  And as for unhandled exceptions, you’d think static analysis would have detected that.
  
  source
  - FauxLiving@lemmy.world ⁨5⁩ ⁨weeks⁩ ago
    Someone should make a programming language like Rust, but that doesn’t crash.
    
    /s
    
    source
PattyMcB@lemmy.world ⁨5⁩ ⁨weeks⁩ ago
Blame it on the massive tech sector layoffs

source
- sunbeam60@feddit.uk ⁨5⁩ ⁨weeks⁩ ago
  Evidence or speculation?
  
  source
  - iglou@programming.dev ⁨5⁩ ⁨weeks⁩ ago
    Obviousness? If you mass layoff your tech staff, you take the risk of more technical failures.
    
    A smaller staff cannot do the same work as a larger one, and I guarantee you they’re being asked to progress at the same speed. So, the tradeoff is on the quality of the product and the testing, not on the speed of development.
    
    source
DaMummy@lemmy.world ⁨5⁩ ⁨weeks⁩ ago
Why’s he saying it’s not an attack? Sounds like he’s protesting too much.

source
- tonytins@pawb.social ⁨5⁩ ⁨weeks⁩ ago
  It’s not the first time Cloudflare shot themselves in the foot.
  
  source
- grumpasaurusrex@lemmy.world ⁨5⁩ ⁨weeks⁩ ago
  There’s nothing to be gained from Cloudflare lying about this. It honestly makes them look worse if the outage was caused internally vs if it had been due to an attack
  
  source
  - DaMummy@lemmy.world ⁨5⁩ ⁨weeks⁩ ago
    Unless it’s from a government they’re not allowed to criticize.
    
    source
herseycokguzelolacak@lemmy.ml ⁨5⁩ ⁨weeks⁩ ago
If you website or service was disrupted by the Cloudflare outage, then you only have yourself to blame.

The Crowdstrike incident from last year was a wakeup call to abandon centralized services like Cloudflare. If you missed that lesson, then it is on you.

source
- groet@feddit.org ⁨5⁩ ⁨weeks⁩ ago
  Yes but no. If you use a different service for the same purpose as you would use cloudflare you will be just as offline if they make a mistake. The difference is just that with a centralized player, everyone is offline at the same time. For the individual websites that does not matter.
  
  source
- MonkRome@lemmy.world ⁨5⁩ ⁨weeks⁩ ago
  Anyone using any technology can miss something and end up in the same spot. I think the real takeaway is that there is way too much consolidation of our technology.
  
  source
halfapage@lemmy.world ⁨5⁩ ⁨weeks⁩ ago
[deleted]
source
- foo@feddit.uk ⁨5⁩ ⁨weeks⁩ ago
  They’re laying off testers because they think AI can do it all now.
  
  source
A_norny_mousse@feddit.org ⁨5⁩ ⁨weeks⁩ ago

a routine configuration change

Honest question (I don’t work in IT): this sounds like a contradiction or at the very least deliberately placating choice of words. Isn’t a config change the opposite of routine?

source
- monkeyslikebananas2@lemmy.world ⁨5⁩ ⁨weeks⁩ ago
  Not really. Sometimes there are processes designed where engineers will make a change as a reaction or in preparation for something. They could have easily made a mistake when making a change like that.
  
  source
  - 123@programming.dev ⁨5⁩ ⁨weeks⁩ ago
    E.g.: companies that advertise on a large sporting event might preemptively scale up (maybe warm up depending on language) their servers in preparation for a large load increase following some ad or mention of a coupon or promo code. Failure to capture the market it could generate would be seen as wasted $$$
    
    source
  - NotMyOldRedditName@lemmy.world ⁨5⁩ ⁨weeks⁩ ago
    I don’t think it was a bug making the configuration change, I think there was a bug as a result of that change.
    
    That specific combination of changes may not have been tested, or applied in production for months, and it just happened to happen today, hence the latent part.
    
    source
    -> View More Comments
- floquant@lemmy.dbzer0.com ⁨5⁩ ⁨weeks⁩ ago
  No, in “DevOps” environments “configuration changes” is most of what you do every day
  
  source
- fushuan@lemmy.blahaj.zone ⁨5⁩ ⁨weeks⁩ ago
  They probably mean that they did a change in a config file that is uploaded in their weekly or bi-weekly change window, and that that file was malformed for whichever reason that made the process that reads it crash. The main process depends on said process, and all the chain failed.
  
  Things to improve:
  
  make the pipeline more resilient, if you have a “bot detection module” that expects a file,and that file is malformed, it shouldn’t crash the whole thing: if the bot detection module crahses, control it, fire an alert but accept the request until fixed.
  
  Have a control of updated files to ensure that nothing outside of expected values and form is uploaded: this file does not comply with the expected format, upload fails and prod environment doesn’t crash.
  
  Have proper validation of updated config files to ensure that if something is amiss, nothing crashes and the program makes a controlled decision: if file is wrong, instead of crashing the module return an informed value and let the main program decide if keep going or not.
  
  I’m sure they have several of these and sometimes shit happens, but for something as critical as CloudFlare to not have automated integration tests in a testing environment before anything touches prod is pretty bad.
  
  source
  - groet@feddit.org ⁨5⁩ ⁨weeks⁩ ago
    
    it shouldn’t crash the whole thing: if the bot detection module crahses, control it, fire an alert but accept the request until fixed.
    
    Fail open vs fail closed. Bot detection is a security feature. If the security feature fails, do you disable it and allow unchecked access to the client data? Or do you value Integrity over Availability
    
    Imagine the opposite: they disable the feature and during that timeframe some customers get hacked. The hacks could have been prevented by the Bot detection (that the customer is paying for).
    
    Yes, bot detection is not the most critical security feature and probably not the reason someone gets hacked but having “fail closed” as the default for all security features is absolutely a valid policy. Changing this policy should not be the lesson from this disasters.
    
    source
    -> View More Comments
floquant@lemmy.dbzer0.com ⁨5⁩ ⁨weeks⁩ ago
Did they though? Aside from the “every outage is a latent bug” angle, from their postmortem it doesn’t seem to me like they tried to blame it on anything but their failure to contain the spread of (and timely diagnose) the issue

source