Comment - FBXL Lotide

Comment on Hexbear has been temporarily defederated - UPDATED now federated

Thanks so much for doing the legwork on this. I was going nuts trying to figure out where seemingly random downtimes were coming from. It felt like a DOS and this cause explains why.

Out of curiosity, how did you trace this root cause?

original

Sort:hotnew top

Demigodrick@lemmy.zip ⁨2⁩ ⁨months⁩ ago
I noticed in the logs before every timeout there were lots of “verify” words appearing, and in each iteration of that statement there were more and more verify words. Honestly had no idea what it meant at the point, only that I didn’t recognise it from looking at lemmy logs previously, it always appeared before a crash, and it felt suspicious.

Here’s an example from some logs before a crash:

2026-03-15T21:47:22.670586Z INFO HTTP request{http.method=POST http.scheme="https" http.host=lemmy.zip http.target=/inbox otel.kind="server" request_id=2cc6dc65-571d-4a69-9733-5e80e455c00b}:receive:community: verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify: verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify: verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify: verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify: verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify: verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify: verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify:verify: verify:verify:verify:verify: activitypub_federation::fetch: Fetching remote object https://hexbear.net/comment/7004776 thread 'actix-server worker 18' has overflowed its stack fatal runtime error: stack overflow

I pinged some logs over to Nutomic on matrix, who thought it might have been related to nested comments, and then I noticed Dessalines had made the linked thread, which matched pretty much with what I was seeing behaviour and logs-wise.

Usefully the logs link the object it’s fetching, and 9 times out of 10 its a deeply nested hexbear thread! Or someone from another instance commenting on a nested hexbear thread. Nutomic confirmed the behaviour based on the logs in the issue, and I’m pulling the logs when I get chance to see what other threads are causing it to crash, although hopefully the fix will make it’s way into 0.19.18 beta 3 so I can stop worrying about it!
original
- db0@lemmy.dbzer0.com ⁨2⁩ ⁨months⁩ ago
  Did you also see the db cpu spiking during this period?
  
  original
  - Demigodrick@lemmy.zip ⁨2⁩ ⁨months⁩ ago
    No, no meaningful cpu spikes I could make out anywhere, although admittedly I was focusing on the lemmy server container mostly
    
    original
- mathemachristian@lemmy.blahaj.zone ⁨2⁩ ⁨months⁩ ago
  There has to be a better way to gain visibility for mutual_aid posts because good god
  
  original
  - frongt@lemmy.zip ⁨2⁩ ⁨months⁩ ago
    Voting and sorting by top.
    
    original