emergency remote access

Submitted ⁨⁨6⁩ ⁨months⁩ ago⁩ by ⁨smashing3606@feddit.online⁩ to ⁨selfhosted@lemmy.world⁩

I just wanted to see what other selfhosters backup emergency plan is if the primary internet router goes offline, leaving you without access to your stuff even via vpn.

the options I've considered so far:
1) cellular smart plug

I tried a ubibot smart plug (SP1) that is supposed to work with cellular, but the device or sim is bad. I'm currently troubleshooting. The problem with this one is it requires a proprietary cloud service, it's supposedly self hostable, but it's a pia to setup and their app port can't be changed easily allowing for a reverse proxy setup on VPS.

2) the other option I am considering is cellular wifi router and a wifi smart plug connected to that device.

what other options have I overlooked? Also, specific models of devices would be helpful info from others doing this already.

TIA!

source

Comments

Sort:hotnew top

Brkdncr@lemmy.world ⁨6⁩ ⁨months⁩ ago
At the enterprise level an Opengear appliance fixes this.

Tripplite PDUs have an option to perform a ping test against an ip, and if it stops responding it can power cycle the outlet of your choice.

If you want to get fancy you can advertise routes from two routers into your L3 switch and if a route goes down your switch will use the backup.

source
philpo@feddit.org ⁨6⁩ ⁨months⁩ ago
I use a cheap Mikrotik LTE Router as a second route. It has the smallest data plan my provider offers - but it’s enough for maintenance and if I need more due to the main line being faulty it’s the same provider’s fault and they pay the bill anyway.

It mainly goes into the OPNsense as a second gateway,but it also allows me to VPN in and reboot the OPN if needed.

If the OPN would be fucked totally in theory I could run the network directly over it,but that would be nasty.

source
- smashing3606@feddit.online ⁨6⁩ ⁨months⁩ ago
  would you mind sharing what model of mikrotik router you use?
  
  source
  - philpo@feddit.org ⁨6⁩ ⁨months⁩ ago
    I use an SXT, as I got it cheap, but the wap LTE kits, the LTAPs mini or the hap AX lite should do as well - softwarewise they are all the same anyway.
    
    Sometimes you find decent older ones on eBay as well.
    
    source
lemming741@lemmy.world ⁨6⁩ ⁨months⁩ ago
I have a Shelly plug that is programmed internally to turn itself on if off for 20 seconds. Home assistant turns the plug off for 10 seconds if curl ip.me fails for 15 minutes.

My modem is plugged into that.

I’ve never had a router or firewall crash if I wasn’t fucking with it and did something ~~stupid~~ ill-advised, so I don’t try that kind of stuff unless I’m home.

source
False@lemmy.world ⁨6⁩ ⁨months⁩ ago
The term to look for is out of band management. Typically this will provide serial/console access to a device, and can often perform actions like power cycling. A lot of server hardware has this built in (eg idrac for Dell, IPMI generically). Some users will have a separate oobm network for remotely accessing/managing everything else.

source
fmstrat@lemmy.nowsci.com ⁨6⁩ ⁨months⁩ ago
I made an 8 outlet box with relays connected to each outlet (might post a how to). That’s connected to a Pi via GPIO.

The Pi runs PiKVM, but also has a service that:

Checks if the router can be pinged

Checks if the internet can be pinged

Checks if the router webUI is up

If any of those fail, it toggles the plugs for modem and router.

I run OpnSense on a 5V miniPC. I have a second one and will be setting up CARP, too.
source
utjebe@reddthat.com ⁨6⁩ ⁨months⁩ ago
If my lab goes down, it sucks, but that’s it. I have no critical service running there.

I have some recoverability, but it requires for the main router to run. If it isn’t running it’s either a HW failure, which I will not fix remotely anyway or power is down. In which case, not much I can do about it neither.

I have router with OpenWRT with Wireguard and main server (NUC) on a smart plug. If the router runs and server is mishaving to the point where I cannot reboot it, I can power cycle it via the smart plug connected to the router.

You mentioned your brother lives 30mins away - well put some tiny server in his house. Having everything at your home is not build for redundancy at all. That’s just the risk management, if you absolutely need access to your server, then 1 site is not going to cut it.

source
- filcuk@lemmy.zip ⁨6⁩ ⁨months⁩ ago
  I have my ‘incident recovery’ docs on my server.
  It went down once, and when that connected, my single thought was ‘fuck’ haha.
  
  source
cecilkorik@lemmy.ca ⁨6⁩ ⁨months⁩ ago
Redundancy. I have two independent firewalls, each separately routing traffic out through two totally independent multi-homed network connections (one cable, one DSL, please god somebody give me fiber someday) that both firewalls have access to. For awhile was thinking of replacing the DSL with starlink until Elon turned out to be such a pile of nazi garbage, so for now DSL remains the backup link.

To make things as transparent as possible, the firewalls manage their IPs with CARP. Obviously there’s no way to have a single public IP that ports itself magically from one ISP to another, but on the LAN side it works great and on the WAN side it at least smooths out a lot of possible failure scenarios. Some useful discussions of this setup are here.

source
Decipher0771@lemmy.ca ⁨6⁩ ⁨months⁩ ago
I buy better gear that doesn’t regularly require a reboot

My mikrotik has not NEEDED a reboot ever, except when I run upgrades. Everything is set up to auto recover when disconnects happen, and power up properly if there’s an extended power failure that causes UPS shutdowns.

I will never understand why people think rebooting their router regularly is a normal thing. That just means your gear or setup is crap.

source
- SteveTech@programming.dev ⁨6⁩ ⁨months⁩ ago
  My Mikrotik routers and switches also reboot in seconds (even for upgrades), which I’ve never seen consumer gear do!
  
  Even my Ubiquiti switches seem to take a minute or so to start forwarding traffic after a reboot; whilst my Mikrotik switches reboot faster than any of my unmanaged switches start up.
  
  source
  - eclipse@lemmy.world ⁨6⁩ ⁨months⁩ ago
    Cisco, HP, and many other “Enterprise” switches will take a minute or two to start forwarding frames after boot.
    
    Doesn’t really excuse Ubiquiti but that’s what they’re trying for.
    
    source
- interdimensionalmeme@lemmy.ml ⁨6⁩ ⁨months⁩ ago
  I get what you mean, I only use L3 top-of-rack data center switches, what a bunch of amateur peasants !
  
  source
  - Decipher0771@lemmy.ca ⁨6⁩ ⁨months⁩ ago
    That’s called unnecessary overkill and you’ll introduce failures from excess complexity.
    
    source
    -> View More Comments
harmbugler@piefed.social ⁨6⁩ ⁨months⁩ ago
On holiday, I'll turn on my router's auto-reboot option to reboot daily.

source
- smashing3606@feddit.online ⁨6⁩ ⁨months⁩ ago
  That's not a bad idea, I def didn't think of that lol. Thanks!
  
  source
hendrik@palaver.p3x.de ⁨6⁩ ⁨months⁩ ago
I've moved my instant messenger onto a VPS and that has a good uptime. And I'm somewhat okay if my Nextcloud and calendars don't sync. Most important data is synced anyway.

Other than that I've called my ISP a bunch of times to give me a new router, they refused, I canceled the subscription and made a new one and got a new router. And that one is better. And in doubt I'll call a family member.

source
doogstar@lemmy.100010101.xyz ⁨6⁩ ⁨months⁩ ago
If you expect it to be flaky you could get one of those old school mechanical time switches with the clicky pegs (or a more modern digital equivalent) and just have it set to power down for 1 click, normally 15 mins, at 4am or whenever suits you - minimally technically complicated and guaranteed stability through planned instability!

source
- Static_Rocket@lemmy.world ⁨6⁩ ⁨months⁩ ago
  You have the potential to run into issues if the device is externally managed. At&t likes to push firmware updates at early hours. Cutting power during one of those would be problematic.
  
  source
brickfrog@lemmy.dbzer0.com ⁨6⁩ ⁨months⁩ ago

if the primary internet router goes offline but the internet isn’t out (ie a router reboot would fix the problem)

Maybe you just need to give it a simple power cycle remotely? There are devices that do that sort of thing, I have a Digital Loggers Web Power Switch Pro that I’ve used on-and-off over the years for this purpose.

www.digital-loggers.com

At one point I had to relocate for half a year while having a slightly unstable desktop that wouldn’t always reboot cleanly and get stuck at the BIOS, it sometimes needed a couple of power cycles to come back online. The Power Switch was perfect for that, I’d log into it remotely and power cycle anything that was plugged into it.

It should work for routers too e.g. it can automatically power cycle something plugged into it based on different conditions like maybe it stops responding to pings or whatever. Or I guess if you had multiple IPs / multiple internet connections the switch itself can stay online and accessed remotely without needing to schedule anything automatic.

Pretty sure there are more pro-level (and more expensive) types of devices to do this sort of thing if you look around

source
- smashing3606@feddit.online ⁨6⁩ ⁨months⁩ ago
  Can you use these with a NUT server? I've been unable to find that info. It looks like they have ones with batteries, which makes them a sort of UPS as well, but would be nice to be able to gracefully shut stuff down when the power fails too.
  
  source
- dysprosium@lemmy.dbzer0.com ⁨6⁩ ⁨months⁩ ago
  How’d you send the command remotely? Radio? Via internet would seem… ironic
  
  source
  - brickfrog@lemmy.dbzer0.com ⁨6⁩ ⁨months⁩ ago
    OP’s example use case in the post was with the internet still being up. Building off of that yes, I’d log into the power switch remotely via the internet where I can then power cycle anything plugged into it - for me it was just to restart unresponsive desktops or whatever was plugged into it.
    
    But you wouldn’t need internet to power cycle the internet router itself by using scheduled tasks. e.g. the power switch can check that the internet router is responding to pings every x seconds/minutes and power cycle it if stops responding. (it has other checks/conditions it can use besides simple pings)
    
    That said my own equipment rarely/never needs a reboot so in the case my network loses internet access it usually means the internet is actually down, nothing I can do about that aside from maintaining backup internet if I needed.
    
    source
    -> View More Comments
solrize@lemmy.ml ⁨6⁩ ⁨months⁩ ago
You have to say what your installation is like. If it’s typical consumer cable modem crap that locks up and needs a power cycle now and then, the simplest approach might just be to add a remote power cycle mechanism:

www.adafruit.com/product/2935

isn’t the cheapest but it’s nicely packaged. That’s just a switchable power strip, so yes you’d need some kind of cellular internet or meshtastic or something to operate it.

In the more serious case where your box is at a data center, you generally open a ticket with the data center and ask them to reboot the box (“remote hands”). Sometimes they will do that for free, other times they charge you.

source
sznowicki@lemmy.world ⁨6⁩ ⁨months⁩ ago
So far it never happened but just in case I always leave one of my keys at some neighbors. I do it anyway because if something happens like broken pipe, it’s good when someone can just enter home without destroying our doors.

source
CompactFlax@discuss.tchncs.de ⁨6⁩ ⁨months⁩ ago
Old laptop with SIM card and some outbound remote access (eg logmein, Tailscale) is what some people I know use. But it’s an expense. I just have a reboot script.

source
PetteriPano@lemmy.world ⁨6⁩ ⁨months⁩ ago
I had a 4G modem with a web interface many years ago. It was flaky and would often hang. I just had a raspberry pi on my network pinging some known address, if it failed for long enough it’d replay the commands to restart the web interface.

If I’d have the same problem today I’d probably have home assistant power cycle the router with a smart plug.

source
- smashing3606@feddit.online ⁨6⁩ ⁨months⁩ ago
  homeassitant access would require internet wouldn't it?
  
  source
  - emuspawn@geostationary.orbiting.observer ⁨6⁩ ⁨months⁩ ago
    Nah, you can use an HA Ping (Settings>Add Integration-> Ping) trigger against 9.9.9.9 or whatever and run a script if it comes back false for X minutes
    
    source
    -> View More Comments
tal@lemmy.today ⁨6⁩ ⁨months⁩ ago
If your problem is brief brownouts or similar — my experience is that some consumer broadband routers have cheap power supplies that leaves them in bad states when PCs will pull through — you could put them on a UPS.

If your problem is that your router is unstable, you could just replace your router. Like, if you need remote access and you have a flaky router, that seems like a prime choice.

You could have a power control device or something and have another machine on your network set up so that if it loses Internet connectivity for some sustained period of time, it power-cycles the router.

If this is for when you’re a long ways away, do you have a friend who you’d trust with a key and flipping a switch?

I expect that there are business-oriented routers that will have integrated watchdog features that will auto-reboot if they hang. I have not gone looking, though.

Possibly, if it’s compatible with your use case, and uptime is critical enough here, having a second, backup server elsewhere, possibly not self-hosted. I mean, your connectivity is always going to be bounded by the reliability of your residential Internet connection otherwise.
source
- smashing3606@feddit.online ⁨6⁩ ⁨months⁩ ago
  I am looking into why it locks up, that's a fair point. good read, and useful tips. Thanks!
  
  source
eco_game@discuss.tchncs.de ⁨6⁩ ⁨months⁩ ago
For me I’d just say oh well, gotta fix it when I’m home again.

Otherwise I’d probably write some script on the server, which reboots my router when the server either doesn’t have internet anymore or can’t ping itself.

source
- smashing3606@feddit.online ⁨6⁩ ⁨months⁩ ago
  This only works if you're planning on being home within a reasonable time. The situation that got me thinking about it in the first place was, I was out of state for several weeks and my router went down a couple days into my trip and had no access to jellyfin (mind you at the time this was really the only service I really wanted). So I had to call my brother who lives 30 mins away to go reboot my router.
  
  source
mesamunefire@piefed.social ⁨6⁩ ⁨months⁩ ago
Its my house so ill just walk over and fix it.

If im out, nothing is too important enough that it cant be fixed in a couple of days.

source
- smashing3606@feddit.online ⁨6⁩ ⁨months⁩ ago
  see 1st reply, not really helpful in this situation
  
  source
avidamoeba@lemmy.ca ⁨6⁩ ⁨months⁩ ago
0 4 * * * /usr/sbin/reboot

Adjust interval as needed.
source
- smashing3606@feddit.online ⁨6⁩ ⁨months⁩ ago
  I will give this a shot. it hasn't happened in a couple weeks, So I can't remember if the device freezes completely or an interface reboot would do the trick.
  
  source
- BrianTheeBiscuiteer@lemmy.world ⁨6⁩ ⁨months⁩ ago
  Even if it isn’t an OpenWRT router if you have a hardwired server it can probably do a soft reset of the router or even modem (most modems I’ve used have had a web interface). If your router is in such a bad state it only responds to a hard reset it’s probably reaching EoL.
  
  source
thelittleblackbird@lemmy.world ⁨6⁩ ⁨months⁩ ago
It didn’t happen in more than 5 years…

For critical equipment you need to spend the extra dollar to minimize this kind of stuff

source
- smashing3606@feddit.online ⁨6⁩ ⁨months⁩ ago
  I don't understand this comment? You're router hasn't gone done in 5 years, is that what you're saying?
  
  source
  - thelittleblackbird@lemmy.world ⁨6⁩ ⁨months⁩ ago
    Yep, at least for their own internal mistake. I rebooted them a few times in that time because updates and important config updates. Even for power outages (2 iirc) they are resilient, they are set to automatic boot up when the power is back.
    
    source
  - FapFlop@lemmy.world ⁨6⁩ ⁨months⁩ ago
    Grammar aside, not having to reboot a router in 5 years isn’t unreasonable.
    
    source
poVoq@slrpnk.net ⁨6⁩ ⁨months⁩ ago
If you use a regular PC as the router you can attach a KVM to remote control and reboot it.

source
- tofu@lemmy.nocturnal.garden ⁨6⁩ ⁨months⁩ ago
  I think this is about the case where you can’t connect to that KVM
  
  source
  - poVoq@slrpnk.net ⁨6⁩ ⁨months⁩ ago
    That KVM can be on a separate connection.
    
    source
    -> View More Comments