This happens after 3-4 days of running the server, then I have to restart it manually.

Submitted ⁨⁨3⁩ ⁨months⁩ ago⁩ by ⁨nutbutter@discuss.tchncs.de⁩ to ⁨selfhosted@lemmy.world⁩

https://discuss.tchncs.de/pictrs/image/95f6de43-dd19-4f48-9909-ccbfed2cbc57.jpeg

I bought an Optiplex 5040, with an i5-6500T, and 8 GB DDR3L RAM.

When I bought it, I installed Fedora Server on it. It got stuck every few days but I could never see the error. The services just stopped working, I couldn’t ssh into it, and connecting it to a monitor showed a black screen.

So, I thought let’s install Ubuntu Server, maybe Fedora isn’t compatible with all of its hardware. The same thing is happening, now, but I can see this error. Even when there’s nothing installed on it, no containers, nothing other than base packages, this happens.

I have updated the bios. I have tried setting nouveau.modeset=0 in the grub config file. I have tried disabling and enabling c-states. No luck till now.

Would really appreciate if anyone helps me with this.

source

Comments

Sort:hotnew top

just_another_person@lemmy.world ⁨3⁩ ⁨months⁩ ago
Seen this before, and almost always has to do with hardware failure or bad hardware config.

Reset the BIOS/CMOS jumper on the board, go back into BIOS setup and set the proper time. Do not touch the CPU or Memory timings. Boot with the defaults and see if it still happens.

Next longer steps: test memory, then stress test the CPU. I’d be shocked if it was a storage issue as I haven’t seen that be the culprit, but might was well run the long SMART tests.

source
- seaQueue@lemmy.world ⁨3⁩ ⁨months⁩ ago
  Make sure the microcode package is up to date as well
  
  source
  - 4am@lemm.ee ⁨3⁩ ⁨months⁩ ago
    Yeah, always check all of this stuff. Server hardware gets a lot more updates than like gamer board BIOS, companies invest high millions, even low billions in this stuff and they expect problems to be address promptly for that kind of cash.
    
    Check for any peripherals or cards, too. RAID, backplanes, networking cards; drivers, firmware, anything.
    
    source
- possiblylinux127@lemmy.zip ⁨3⁩ ⁨months⁩ ago
  Poorly supported hardware will also do this
  
  source
  - seaQueue@lemmy.world ⁨3⁩ ⁨months⁩ ago
    An Optiplex 5040 should be well and thoroughly supported for 6+y now
    
    source
- nutbutter@discuss.tchncs.de ⁨3⁩ ⁨months⁩ ago
  I did reset it. It did not help. I ran memtest86 for over 2 hours and did a CPU stress test for over 15 hours. Nothing crashed during the testing.
  
  source
Shimitar@feddit.it ⁨3⁩ ⁨months⁩ ago
Had the same issues, it was heat.

Cool down your server, add a fan or a cooler…

I added a usb-powered fan sucking cooler air from outside the server area directly blowing it on the chassis.

That fixed for me.

source
- nutbutter@discuss.tchncs.de ⁨3⁩ ⁨months⁩ ago
  I cleaned everything and reapplied the thermal paste. That did not solve the problem. Also, the CPU is only of 35 watts and never goes over 55°C.
  
  source
nialv7@lemmy.world ⁨3⁩ ⁨months⁩ ago
Time to add a cron job to auto reboot it once a day

source
tenchiken@lemmy.dbzer0.com ⁨3⁩ ⁨months⁩ ago
That server sounds a bit older in the teeth… Has new thermal paste been applied to the cpu? Even if the reported temps are under 90c, you might be getting hot spots causing glitches inside the package.

Worth trying a couple of different generations of kernel as well, both newer and older. You might be hitting a regression somewhere.

source
- nutbutter@discuss.tchncs.de ⁨3⁩ ⁨months⁩ ago
  I cleaned everything and reapplied the thermal paste. That did not solve the problem. Also, the CPU is only of 35 watts and never goes over 55°C.
  
  source
solrize@lemmy.world ⁨3⁩ ⁨months⁩ ago
Try running memtest86 for a few days to test memory. That is fairly easy to do though it involves booting from a flash drive. Web search should find info.

source
- nutbutter@discuss.tchncs.de ⁨3⁩ ⁨months⁩ ago
  I just ran it. It took over 2 hours to finish. Showed no errors. Is there a benefit of running it for a few days?
  
  source
  - solrize@lemmy.world ⁨3⁩ ⁨months⁩ ago
    If the problem is intermittent then longer run has better chance of catching it, but 2 hours with no errors is a good sign, with regard to the memory.
    
    source
ikidd@lemmy.world ⁨3⁩ ⁨months⁩ ago
Can you sudo dmesg | grep microcode and see if you have any errors?

If so, I’d be inclined to sudo dnf reinstall microcode_ctl then do a sudo dracut -f to regenerate the initramfs, and reboot. Be sure to have a working fallback kernel like LTS installed so you can recover if need be.

source
lolonaut@discuss.tchncs.de ⁨3⁩ ⁨months⁩ ago
I had problems with soft locks because somehow the PSU was in corrupt state, maybe through a black out or something. The problem persisted through reboots and power offs, only after cutting power helped.

source
Decronym@lemmy.decronym.xyz ⁨3⁩ ⁨months⁩ ago
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:

Fewer Letters More Letters

LTS Long Term Support software version

PSU Power Supply Unit

RAID Redundant Array of Independent Disks for mass storage

[Thread #907 for this sub, first seen 6th Aug 2024, 10:15] [FAQ] [Full list] [Contact] [Source code]

source
rookbrood@lemmy.world ⁨3⁩ ⁨months⁩ ago
It’s a long shot, but I had something similar on one of mine servers once. It was fixed by installing irqbalance and starting that daemon at startup.

source
bulwark@lemmy.world ⁨3⁩ ⁨months⁩ ago
I’ve never seen this particular error, but CPU stall warnings seem like a fairly common thing. I wouldn’t jump straight to hardware fault, but it’s a possibility.

docs.kernel.org/RCU/stallwarn.html

source
- catloaf@lemm.ee ⁨3⁩ ⁨months⁩ ago
  I’d lean toward bad hardware.
  
  Try stress testing the CPU and RAM. See if you can get it to happen more frequently. Also see if you can disable that CPU core, either in the BIOS or in the OS, to see if the problem goes away.
  
  source
  - loganb@lemmy.world ⁨3⁩ ⁨months⁩ ago
    I’m with catloaf. Consistent CPU soft locks point to a possible bad memory module or CPU.
    
    Clear CMOS.
    
    Try removing one memory module at a time.
    
    See if there is an option to disable hyperthreading in bios.
    
    Another thing to try is to remove the CPU, careful not to damage the LGA pins on the motherboard, and clean the CPU contacts with alcohol. Take care to ground yourself out and the case before handling the CPU out of socket.
    
    source
    -> View More Comments
possiblylinux127@lemmy.zip ⁨3⁩ ⁨months⁩ ago
Does it have a Nvidia GPU? If it doesn’t then the nouveau modest does nothing.

My guess it that you are using a badly support realtek device. However I would need to see the full dmesg

source
TheBigBrother@lemmy.world ⁨3⁩ ⁨months⁩ ago
Set watchdog to reboot every day and juice it until the last drop before it definetly crashes.

source
thesporkeffect@lemmy.world ⁨3⁩ ⁨months⁩ ago
I’d start here: askubuntu.com/…/watchdog-bug-soft-lockup-cpu6-stu…

source
slacktoid@lemmy.ml ⁨3⁩ ⁨months⁩ ago
What kind of drives do you have in your RAID? Is it SMR?

source
- nutbutter@discuss.tchncs.de ⁨3⁩ ⁨months⁩ ago
  The boot drive is an SSD, which is not in any RAID. I have another HDD connected via SATA. Another HDD connected via USB.
  
  source
  - slacktoid@lemmy.ml ⁨3⁩ ⁨months⁩ ago
    So 2 HDDs one SATA and one via USB in RAID? Can you remove the RAID drives and test it out?
    
    source
    -> View More Comments

Fewer Letters	More Letters
LTS	Long Term Support software version
PSU	Power Supply Unit
RAID	Redundant Array of Independent Disks for mass storage