glizzyguzzler

I approach it the same as I did with Docker, one user per container.

I started with rootless but networking within Podman is moot with multiple users per container. And one user for all containers to get networking has to lead to subUID clashes (and thus escape vectors) - unless someone can explain how not…

But root Podman is just as secure anyway, and easier, so I just roll with UserNS=auto and use idmap on the volumes to enable writing as the specified user for the container. And networking in Podman works because it’s one user space. By default UserNS=auto gives 1024 subUIDs to a container. I had to up that to 65534 or whatever the max is for Frigate to work. Every other container is cool with the default 1024. The subUIDs are pulled from a user named container that you need to enable for Podman root to work with UserNS, and it has like 2 million or something with their recommended setup, so it’s good.

It was containers:2147483647:2147483648 into subUID and subgid files

And I do have a fuckton of users; Debian once complained it ran out of numbers or something after like 20 users, so I just ran the first thing I found to make the UID limit some really big number, and I never thought about it again!

I have this setup. Upfront, I would not recommend Proxmox, the update methods are annoying. The better way is straight Debian with Incus installed, then you get straightforward stable Debian updates automatically - they won’t break anything and you’re secure. Sometime I’ll redo it - I haven’t because, of course, it is my router and when its down I don’t have internet! So foreboding and on the back burner.

Also also Proxmox’s GUI leaves a lot to be desired (for me, it looks like ass and is confusing), Incus is nicer for VM control and Cockpit is nicer for host control. After typing all that I realize I’m a hater at this point

I haven’t really noticed downtime issues cause of Proxmox updates cause I just do it when nothing is happening. And Proxmox hasn’t bricked itself, though I am wary of it because that has happened to others due to their rolling release update style.

I’ve got a Dell Wyse 5070 Extended with a 2 port Intel NIC in it. I pass both ports through leaving the built-in port for managing Proxmox.

Here are my notes:

Set NIC PCIe Passthrough for Network Card

nano /etc/default/grub

• Edit this line by adding intel_iommu=on to get

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

update-grub

nano /etc/modules

• Add these lines

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

update-initramfs -u -k all

reboot

Click on 2nd level thing named router on the left side vertical bar hierarchy thing and then click in the top right the blue Create VM button.

• General tab

  • Name: OPNsense
  • Start at boot: checked
  • Start/Shutdown order: 1
  • Startup delay: 15

• OS tab

  • Use media: DVD version (usb might work) of OPNsense.iso

• System tab

  • Machine: q35
  • Bios: OVMF (UEFI)
    • Storage: local-lvm
    • UNCHECK Pre-enroll Keys (HATE)

• Hard Disk tab

  • Disk size (GiB): 15
  • Discard: checked
  • SSD emulation: checked

• CPU tab

  • Cores: 4
  • Type: host {makes it not moveable between diff CPU types but will theoretically allow for more speed}

• Memory tab

  • Memory (MiB): 2048
  • Minimum memory (MiB): 512

• Network tab

  • No network device: checked

• Confirm tab

  • Do not start on creation

• After creation, go to Hardware tab in the 2nd left vertical list on the browser page and click add

• Click PCI Device
  • Device: ...01:00.0 I350 Gigabit... & ...01:00.1 I350 Gigabit... (1st & 2nd ones)
  • PCI-Express: checked

Go to the Console tab in the 2nd left vertical list on the browser page and hit enter to get to a command line in the OPNsense VM

!Add expand storage via command line!

And lastly, during setup I have these notes

It will choose wrong (WAN gets igb1 and LAN gets igb0 -> we want WAN gets igb0 and LAN gets igb1)
Default User: root, PW: opnsense (they don't tell you anywhere, you don't have internet b/c this is your new router, fuck em)
**Access at 192.168.1.1 via pluging an ethernet cable into the 1st port in a set of forwarded ports**
*Note that we will move it so the 1st port is the WAN (can't access OPNsense from the WAN port for safety), so after following this you access via 2nd port*

So watch out for those things. Not sure quite what I mean by the 1st and 2nd port things, may be related to on setup it had the order of the ports I wanted wrong so they’re switched till setup is complete and it reboots.

I don’t remember doing this at this point, but maybe this info dump will help!

Pangolin has a lot more going on than I expected, I thought it was just a mesh system. That might be a one-stop shop, thanks for sharing!!

I do see Authentik can apparently act as a reverse proxy, so it can sit at the very front. But I’d lose out on caddy + crowdsec then…

I’ll have to do some reading if caddy can actually just forward to caddy, and where the TLS is terminated and all that.

And I do use the internal setting now, but I need it off if I want to publish the port on the LAN so that the VM can see the ports on the LAN. But if I can have WAN caddy do the auth check and forward along good stuff to the LAN caddy, then I’ll only need to publish LAN caddy’s port and that’s not the worst at all.

Thanks for the ideas, I’ll try to cook “caddy (DMZ) -> auth OIDC (DMZ) -> caddy (LAN) -> services (LAN)”!

Rootful to get the cross-user networking

Containers lower the bar since the developer doesn’t need to make their program work on every system - just the container’s system.

Price we pay for more programs. And they bring boons like read-only, rootless, limited capabilities, and constrained perf limits (esp if you use Podman with Quadlets).

And don’t feel trapped - the Dockerfile is a recipe to build that program. Probably want to do it in an LXC container since it’ll want to use /data for its data or something. But the LXC container can also be run as a user but the program thinks it’s root. Plenty of security abounds!

I think it’s worth the price and you’re not trapped. They’re trapped with you and your robust Quadlet files

King, simply neg your collaborators into using overleaf