confusedpuppy

I use Alpine Linux so yeah, I am using something by Linux

It’s not ideal but I don’t have the time to find alternatives for everything all at once, just one thing at a time which is the pace I am going. I’ll deal with it when the time comes but it’s in the back of my mind

I don’t expect perfection out of anyone. I would not make friends if I held them to such high standards or expectations. Everyone learns, adapts and grows at their own pace and I would do better to support them along the way through their journey

Myself and the people around me are doing our best in this situation we are in with the tools that are currently available to us

I don’t believe trying to reduce the complex human experience to 100% right or 100% wrong will lead to any useful discussion when there’s so much in the middle we could be discussing

That’s true, it CertBot seems to be quite mature

It was a bit overwhelming for me when I first started into self hosting, there were a lot of concepts that I had to wrap my head around first before I could look into using CertBot itself

Awesome, thanks so much, this is a big head start for me

I have a good idea how I want to organize things now

I definitely expect this path to be a bit of a challenge

I was looking at lighttpd as well. That was the only one on the list that sounded familiar to me. I may check it out and see if it can fit me needs. I do run a very minimal setup so I might just be enough.

I also make use of Caddy’s built in file server to serve static pages so this is good to know about HAProxy

Thank you, I’ll have a look at lego-acme, it looks interestingly. I prefer to use cron so this is could be something I would consider using.

This is great, thank you for taking the time for this write up :) The provided scripts are a huge help to me

So far my only question I have is about the directories you use. I was wondering if you could provide the directories you use or even just an example so I could better understand the file tree. I’m very particular with my files and have a whole system dedicated to maintaining neat and organized files

I agree about not using /etc for server related stuff. I keep all my server/container related stuff in /srv so it’s easier for me to manage

I’ve used NPM before and it is quite simple and easy to use, however I do have a preference for CLI tools

Impossible? There are people who still write code by hand. There are people who oppose AI, some more actively (or destructively) than others.

Harder to avoid seems like a more reasonable take.

However I am coping, by actively seeking, talking about and supporting alternatives with the hope of spreading that knowledge to those who would like to avoid the use of AI or enable those who support AI.

Human creativity has a much longer and far more interesting history when compared to AI or machine learning. AI hasn’t always existed and does not need to have complete influence over our future.

I personally feel like taking money from an AI sponsor is enabling behaviour. From that perspective, I do not want to support that type of behaviour.

I don’t think you are in the wrong here. You clearly stated your wants and so many replies here have simply ignored what you have said. The response to your post has been strange

How would trace the echo reply in the output chain? I tried adding meta nftrace set 1 directly to the ICMPv4 rule as well as making a chain postrouting rule but I can’t seem to figure it out.

The default policy for output is accept so I’m assuming I don’t need to explicitly add ICMPv4 rules to the output chain. As a test I did add the ICMPv4 rules to the output chain and I still have the same results as before.

I can confirm that the server is receiving the ping requests but my computer sending the pings loses the ping packets.

Aah you did mention that, my eyes just decided to skip that when I read your post.

This reminds me of another issue I ran into but I use Alpine Linux so I don’t know if it’s a distribution specific issue. I’ll share the issue and workaround solution anyways as something to consider.

Networking on Alpine Linux is controlled by a process called networking and for reasons I don’t understand and can’t see by any logs, it just stops working. I can’t ssh or access the reverse proxy port. I don’t remember if ping was working or not as it’s been a while since I dealt with it now.

My work around was to have a script on my server ping a known location and restart networking if it couldn’t ping out. If a second ping after restarting the process failed, it would then restart the device. This script would run every 15 minutes.

It’s a bandage solution that doesn’t solve the problem but it does keep my server running. However it seems like pings still work with your server so you might need to get creative in how you test your server’s connectivity.

It sounds to me like ssdh may have stopped working. That may explain why you can’t ssh into your server but pings still respond. I have a Raspberry Pi4 and a Pi5 and have had similar issues in the past.

I would probably approach this issue by writing a small script that checks every so often if the process sshd is still alive and if not restart sshd. Maybe SystemD can so something similar but I am not familiar with SystemD.

I didn’t think to try ping6. It looks like I can reliably get responses from ping6 but not ping4

The server can ping other devices on the same network just fine

Also, I’m glad you mentioned syslog. I couldn’t figure out logging and it turns out I had to add the syslog package to my server to get logging working

I tried what you said. I sent a ping from my computer to the server and this was the output of nft monitor trace:

trace id 1d01c81e ip ping_trace prerouting packet: iif "eth0" ether saddr b0:7d:64:e8:8f:3c ether daddr d8:3a:dd:de:28:99 ip saddr 192.168.40.201 ip daddr 192.168.40.203 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 65074 ip length 84 icmp type echo-request icmp code 0 icmp id 35586 icmp sequence 0 
trace id 1d01c81e ip ping_trace prerouting rule icmp type { echo-reply, echo-request } meta nftrace set 1 (verdict continue)
trace id 1d01c81e ip ping_trace prerouting policy accept 
trace id 1d01c81e inet filter input conntrack: ct direction original ct state new ct id 271120081 
trace id 1d01c81e inet filter input packet: iif "eth0" ether saddr b0:7d:64:e8:8f:3c ether daddr d8:3a:dd:de:28:99 ip saddr 192.168.40.201 ip daddr 192.168.40.203 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 65074 ip protocol icmp ip length 84 icmp type echo-request icmp code 0 icmp id 35586 icmp sequence 0 
trace id 1d01c81e inet filter input rule ip protocol icmp icmp type { echo-reply, destination-unreachable, echo-request, time-exceeded, parameter-problem } accept comment "Accept ICMP" (verdict accept)

I sort of get what’s happening and it looks like the ping request has been accepted.

From my computer when I send a ping it shows:

15:55 dell:/tmp/ $ ping -c1 192.168.40.203
PING 192.168.40.203 (192.168.40.203): 56 data bytes

--- 192.168.40.203 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss

So even though it’s being accepted, I still get nothing going back to my computer, at least that’s how I understand it.

I’ve been checking my rules with nft -c -f /etc/nftables.d/firewall.nft as well as checking the ruleset after every change and every change appears as it should. I’m stumped. Even more stumped because just allowing all traffic still doesn’t allow me to ping my server but I can access ssh, wireguard and my reverse proxy just fine. I would have assumed allowing all inbound traffic would also accept ping requests too…

I tried your suggested rules and still nothing

I went a step further and simply enabled all incoming connections with:

table inet filter {
	chain input {
		type filter hook input priority 0; policy allow;
	}
}

Again I can connect with SSH and WireGuard but I still can’t ping my server. If I restore to my last backup with iptables, I can get a response from ping again.

I also tried directly translating the rules from iptables with:

iptables-save > /tmp/iptables.dump
iptables-restore-translate -f /tmp/iptables.dump > nftables.dump

and adding the rules:

#!/usr/sbin/nft -f

define WIREGUARD_PORT = 51820
define WIREGUARD_ADDRESS = 10.0.0.0/24
define SSH_PORT = 5025
define SSH_ADDRESSES = { $WIREGUARD_ADDRESS . $SSH_PORT, 192.168.40.204 . $SSH_PORT }
define PUBLIC_PORTS = { 5050 }

table inet filter {
        chain input {
                udp dport $WIREGUARD_PORT accept \
                comment "Accept WireGuard connections"

                ip saddr . tcp dport $SSH_ADDRESSES accept \
                comment "Accept SSH connections from known devices or WireGuard"

                tcp dport $PUBLIC_PORTS accept \
                comment "Accept public connections"

                icmp type echo-request limit rate 5/second burst 10 packets counter accept
                icmp type echo-request limit rate 30/minute burst 120 packets counter accept
                icmp type echo-request limit rate 1/minute burst 2 packets counter log prefix " PING-PONG-FLOOD "
                icmp type echo-request counter drop

                icmp type destination-unreachable counter accept
                icmp type time-exceeded counter accept
                icmp type parameter-problem counter accept
                icmp type echo-request counter accept
        }
        chain forward {
                icmp type destination-unreachable counter accept
                icmp type time-exceeded counter accept
                icmp type parameter-problem counter accept
                icmp type echo-request counter accept
        }
}

and still no ping from my server…

I will agree, the documentation for nftables is just not as accessible or consistent as iptables. It’s a bit frustrating.

That’s fair

I have a very different view on data, physical property and familial relationships. Everything is temporary to me and I’ve prefer my stuff to be reused rather than act as an archive to my own life.

My pictures, music and technology related projects are just for myself. Anything I wish to share after my death is stored on an unencrypted drive connected to a Raspberry Pi that acts as my web facing server that serves only static data. It’s mainly a bunch of wikis, linux/shell scripting references, some of my git repositories, some survival type ebooks and some other random stuff.

I’ve lost data multiple times throughout my life so I know I’d be disappointed but not sad if I lost all my data one more time. I do have multiple backups now so I at least have some data resilience compared to the past.

I am the same way. My backup is on the other side of the room. If my house goes up in flames, I’ll have bigger issues to deal with, like my house going up in flames.

Data is just data. It wasn’t there when I was born and it’s all encrypted now so when I die, I want the next person to wipe it clean and use the hardware for themselves.

Having a backup is convenient but it’s not the end of the world if I lose it.

I was just a cable runner, I just happened to work with a lot of different trades. I learned a little bit from every trade

I used to work as an electrician in the automation industry (robots that welded the frame of automobiles) but I was only an apprentice. I worked alongside robot and PLC programmers and absorbed information through them. I lost the motivation to finish my apprenticeship due to a changed perspective on the harmful affects of too much progress. That was as close as I got to any coding.

I decided to enjoy a mid-life retirement after protesting my way into getting fired which gave me time to explore hobbies. Along the way I ended up buying a used Raspberry Pi 4 and 5 and found some joy in both self hosting and shell scripting. Fortunately shell scripting supplements self hosting. I have been slowly crafting a low resource, low maintenance, minimal server.

Both Pi’s run Alpine Linux, the Pi 4 is dedicated to HomeAssistant which controls a handful of lights and switches. The Pi 5 runs Caddy and Kiwix. Right now it just hosts a bunch of wikis and a static file server with Caddy. Eventually I plan to run a blog created only by a single Bash script.

This is all completely outside of any workplace skill and I think I’d like to keep it that way. Programming for money would likely kill the wonder I still have for computers.

Everything I have currently is pretty much working and maintaining itself. My logs are quiet and predictable. It’s been nice.

I do have two things to do. One is it switch from iptables to nftables because podman is depreciating support for iptables. I absolutely hate working with firewalls, the syntax is always awful. For some reason ufw isn’t working with nftables like I hoped so now I’m working up the energy to learn nftables itself.

The other thing to do is to update Alpine Linux to the latest release version. I’m trying to figure out how to properly update the cache of installed packages on my main computer. After that I should be able to run my upgrade script and things should take care of itself. I hope. With Alpine I usually wait a month or two after the initial release of a new version, things always seem to break if I update immediately and I don’t want to trouble shoot too many things.

Other than that, I’ve gone back to making functional scripts again. Currently I am working on a new backup script that handles logging and automated rotation of backup snapshots. It’ll be POSIX portable and easy to integrate with crontab. It’s essentially the culmination of all my POSIX shell scripting knowledge. I just have to figure out one last and huge hurdle and I’ll be happy with the script.

I too was upset with the use of Claude/AI/LLM’s in the rsync project. The maintainer received a lot of public backlash and as a result people have offered their time to help the maintainer with this project.

Since the backlash, the use of agents has been drastically reduced. That seems like a very human and overall positive response. It sucks that it happened. It’s nice that people are now doing the work again. The response could have been a lot worse. I can be forgiving in this specific situation.

I also haven’t seen any bot activity after I started using wildcard sub domains. My ISP blocks all incoming on common ports so I also use uncommon ports. I assume the combination of the two makes it too time consuming to find me.

I hid my ssh port with a wireguard connection so I also don’t see any attempts on my ssh port anymore either. My logs, including fail2ban, are quiet and boring.

It’s nice to have a quiet corner of the internet for myself.

I’m going to post all the commands I use because I think that may be easier to follow. All the commands I’m posting will include the –dry-run option so if anyone tries to copy/paste this into their terminal, no actions will be taken. Instead it will show you what is going to happen if you ran the command without any changes.

As I mentioned before, each partition will require it’s own command. The easiest way is using lsblk. Below is my current setup and here you can see I have 5 partitions. One partition is a swap so I will only be working with 4 partitions, /, /boot, /boot/efi and /home:

dell:~ $ lsblk
NAME                  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
nvme0n1               259:0    0 953.9G  0 disk  
├─nvme0n1p1           259:1    0   500M  0 part  /boot/efi
├─nvme0n1p2           259:2    0  62.5G  0 part  
│ └─luks1-dell        253:0    0  62.5G  0 crypt 
│   ├─vg_dell-lv_boot 253:1    0   500M  0 lvm   /boot
│   ├─vg_dell-lv_swap 253:2    0     8G  0 lvm   [SWAP]
│   ├─vg_dell-lv_root 253:3    0    38G  0 lvm   /
│   └─vg_dell-lv_home 253:4    0    16G  0 lvm   /home
├─nvme0n1p3           259:3    0   600G  0 part  
├─nvme0n1p4           259:4    0 270.9G  0 part  
└─nvme0n1p5           259:5    0    20G  0 part 

It’s good to first check what partitions you are using. My Raspberry Pi’s (ARM) only have / and /boot for example.

The following rsync commands are what I use to make a complete backup of my system. I do exclude a number of directories because they are for temporary stuff like ram, processes or even devices/drives. It’s also important to exclude the specified backup directory to avoid recursing into the backup directory and filling up your storage space.

I have a manual backup location and automated backup location. The following is for my manual backup location in /backup/main on my system. This location can be changed to wherever you want your backup.

# Backup
# /
rsync --dry-run --archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --numeric-ids --delete --exclude=/backup/* --exclude=/boot/* --exclude=home/* --exclude=proc/* --exclude=sys/* --exclude=dev/* --exclude=tmp/* --exclude=run/* --exclude=mnt/* --exclude=media/* '/' '/backup/main/'

# /boot/
rsync --dry-run --archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --numeric-ids --delete --exclude=lost+found '/boot/' '/backup/main/boot/'

# /boot/efi/
rsync --dry-run --archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --numeric-ids --delete --exclude=lost+found '/boot/efi/' '/backup/main/boot/efi/'

# /home/
rsync --dry-run --archive --acls --one-file-system --xattrs --hard-links --sparse --verbose --human-readable --partial --progress --numeric-ids --delete --exclude=lost+found --exclude=.cache/* '/home/' '/backup/main/home/'

It’s been a while since I last researched these options so I’ll give a brief explanation of the types of options I used. I’d suggest having a look online or at the man page to get a better idea of what each option does.

Options:

–dry-run Only displays what rsync will do, remove this once you are ready to commit any syncs/changes

archive --acls --one-file-system --xattrs --hard-links --sparse Helps preserve file attributes and other information. I think hard-links is also used to reduce backup size. There are many rsync guides that will give a better explanation of how hard-links work

verbose --human-readable --partial --progress will display visual data about what rsync will do

–numeric-ids I use this because I store multiple device backups on a single drive which gets copied to other storage devices. This stores file ownership information as numeric values to prevent ownership issues when restoring

–delete this will force the destination directory to match the source directory completely. If you delete a file from the source directory, when you perform a sync, it will delete the same fie in the destination directory. This can be dangerous if you are not prepared for it. This is why –dry-run is so important and useful.

Extra options: My automated scripts use 2 additional options. I keep a rolling set of 4 backups (One month of weekly backups). I create a new directory /backup/updating and use a symlink from/backup/latest that points to the most recent automated backup. After the backup is created, I rename /backup/updating to something with a timestamp like /backup/backup_2026-07-01_1782882013

–mkpath will create any non existing directories specified in the command

–link-dest=/backup/latest/ will use the unchanged files from this directory to help reduce backup sizes. I think this is called an incremental backup

This has been the most reliable way to handle backups for myself. I do run into issues with docker/podman containers sometimes and will have to manually delete those directories. I haven’t figured out how to deal with that issue yet but fortunately it’s easy to find those directories. Running the command will give errors about what directories can’t be removed which makes it easy to hand delete them in another terminal window.