Local homeserver suddenly not responsive to SSH but can be pinged

Submitted ⁨⁨11⁩ ⁨hours⁩ ago⁩ by ⁨gedfromgont@piefed.ca⁩ to ⁨selfhosting@slrpnk.net⁩

cross-posted from: https://piefed.ca/c/selfhosted@lemmy.world/p/957754/local-homeserver-suddenly-not-responsive-to-ssh-but-can-be-pinged

Cross-posting this from Lemmy world where it seems my post coming from piefed.ca is not federating.

This is something that seems to happen randomly to my little RaspberryPi 4B homeserver, like once every few months maybe.

What happens is that out of nowhere (meaning I didn’t make any change to config or installation) I cannot reach any services I host on it (i.e. Jellyfin, Anchor notes…). I cannot access it via SSH at all, connection keeps timing out.

Ping however gives a signal. I can see/hear its running and doing something, since I hear the external HDD working. It is running headless and I have no monitor and/or keyboard I could connect.

So far my only way out is to just switch its power off, which is obviously not good for the hard drive. It has happened now like half a dozen times over the past few years, so I am wondering what a better way is to handle this. This was on RPI OS and now its happening on DietPi, so seems not related to the OS version.

I tried sending a shutdown signal via SSH but that also just times out, so how do I get it to shutdown properly?

Secondly, what kind of logs could I look at or start collecting to make sense of this?

original

Comments

Sort:hotnew top

dhork@lemmy.world ⁨10⁩ ⁨hours⁩ ago
This can sometimes happen if the filesystem has failed or is otherwise inaccessible. SSH needs to read config files and if it can’t, it fails to connect. It would try to write a failure to the log, but can’t.

Things that don’t require filesystem access, and can run totally from processes in memory interacting with hardware, would be fine. So your Ethernet stack works, and so does ping. A serial port would probably work. SNMP would work, too, for anything that doesn’t need to get a metric from the filesystem. I bet if you had a remote syslog server, you would get the logs that are falling out of your filesystem.

original
- gedfromgont@piefed.ca ⁨7⁩ ⁨hours⁩ ago
  I think you might onto something with the filesystem. I am using Hd-idle to switch off the HDD when not in use. It was of course in use the entire time during this, but I noticed in the logs that it came up multiple times as “assuming disk spun up after long sleep” even though it hadn’t been off at all.
  
  Only saw this after I made my post and I cannot make sense of it, but I have never seen this message otherwise, it only happened during this period of not being able to access the server.
  
  original
  - Zwuzelmaus@feddit.org ⁨7⁩ ⁨hours⁩ ago
    I have also seen a situation where ssh did not accept a connection anymore when the disk was full.
    
    original
lefaucet@slrpnk.net ⁨8⁩ ⁨hours⁩ ago
I had a similar situation. Turned out to be a grounding issue.

Kept randomly losing the root m.2 drive and when I’d plug in monitor/keyboard I could type in the terminal, but most executables would not work.

Hard reboot and it’d be good for a few weeks or few months. Then it’d die again. Really annoying losing it because my calendar and email gui is on it.

Once I noticed a small static shock when I touched the case.

I taped a wire to the aluminum case and taped the other end to a grounded chassis of another computer and all was good since then

original
- gedfromgont@piefed.ca ⁨8⁩ ⁨hours⁩ ago
  Interesting. Not sure this is it, definitely didn’t feel a shock the last few times I handled the case during this. But thanks for sharing! I’ll keep it in mind.
  
  original
Bane_Killgrind@lemmy.dbzer0.com ⁨10⁩ ⁨hours⁩ ago
Like a spinning disc drive? I always found those separately powered drives flakey on a PI, the drive was fine, the pi is fine, the dock I used was solid on my PC and laptop, but not the pi.

Is it your OS drive? Could you get a tiny USB SSD and try that out as primary?

original
- gedfromgont@piefed.ca ⁨8⁩ ⁨hours⁩ ago
  The OS is on a SD card, the HDD is a spinning disc connected via USB and I actually never had any issues with its connectivity.
  
  There are however issues sometimes if I connect a second drive, where I assume it does have something to do with not getting enough power for 2 external HDDs, but I only ever connect a second drive for backing up.
  
  Don’t think this is connected, but if more people here feel like yes then I am happy to check in that direction.
  
  original
  - Bane_Killgrind@lemmy.dbzer0.com ⁨7⁩ ⁨hours⁩ ago
    Are you powering the spinning disc off the pi? High draw can cause funky stuff too.
    
    Any under voltage warnings?
    
    original
povoq@slrpnk.net@slrpnk.net ⁨11⁩ ⁨hours⁩ ago
You might be somehow locking your self out with fail2ban.

original
- gedfromgont@piefed.ca ⁨8⁩ ⁨hours⁩ ago
  I am not aware I have that installed/configured, not sure if this comes with a DietPi installation?
  
  If we assume yes, how would I fix this?
  
  original
- frisbird@lemmy.ml ⁨10⁩ ⁨hours⁩ ago
  This was my first thought
  
  original
ianhclark510@lemmy.blahaj.zone ⁨10⁩ ⁨hours⁩ ago
last time that happened I had updated SSHD and caused a mismatch in the list of provided/accepted ciphers and I ended up needing to restart

original
UnpledgedCatnapTipper@piefed.blahaj.zone ⁨11⁩ ⁨hours⁩ ago
Do you have a display and keyboard you can connect to it? If you can get video out, and it accepts keyboard input, you can just log in locally and reboot.

I have a desktop system that crashes similarly, though pings also fail. Unfortunately for me, local access also doesn’t work so I simply hard power cycle it like what you’ve been doing for your Pi.

original
- gedfromgont@piefed.ca ⁨8⁩ ⁨hours⁩ ago
  
  It is running headless and I have no monitor and/or keyboard I could connect.
  
  From my post. But also it sits in a shelf where anyway don’t have much space to put anything else. It also happens too rarely that it wouldn’t make sense to get specifically for this a display and keyboard in place.
  
  original
  - UnpledgedCatnapTipper@piefed.blahaj.zone ⁨7⁩ ⁨hours⁩ ago
    Oh apologies, I missed that line in your post.
    
    You wouldn’t need to keep the monitor/keyboard there the whole time. Just connect them temporarily when it has the issue. This obviously doesn’t work if you don’t have a monitor or a keyboard you could borrow from a different system.
    
    original
Zwuzelmaus@feddit.org ⁨11⁩ ⁨hours⁩ ago
From what machine? Pc? Mobile? If a PC, how is it connected to the Pi? Direct cable, a switch, a WiFi…? Does the Pi use a fixed address in your home network?

original
- gedfromgont@piefed.ca ⁨8⁩ ⁨hours⁩ ago
  From phone or laptop, doesn’t matter. The Pi is in my network via Wifi, well technically its connected to the router via cable but then everything else is connecting via Wifi through the router to the Pi. I have given the Pi a fixed IP.
  
  original