except that all requests’ IP addresses are set to the router’s IP address (192.168.3.1), so I am unable to use proper rate limiting and especially fail2ban.

I’d guess that however the network is configured, you have the router NATting traffic going from the LAN to the Internet (typical for a home broadband router) as well as from the home LAN to the server.

That does provide security benefits in that you’ve basically “put the server on the Internet side of things”, and the server can’t just reach into the LAN, same as anything else on the Internet. The NAT table has to have someone on the LAN side opening a connection to establish a new entry.

But…then all of those hosts on the LAN are going to have the same IP address from the server’s standpoint. That’s the experience that hosts on the Internet have towards the same hosts on your LAN.

It sounds like you also want to use DHCP:

Getting the router to actually assign an IP address to the server was quite a headache

I’ve never used VLANs on Linux (or OpenWRT, and don’t know how it interacts with the router’s hardware).

I guess what you want to do is to not NAT traffic going from the LAN (where most of your hardware lives) and the DMZ (where the server lives), but still to disallow the DMZ from communicating with the LAN.

considers

So, I don’t know whether the VLAN stuff is necessary on your hardware to prevent the router hardware from acting like a switch, moving Ethernet broadcast packets directly, without them going to Linux. Might be the case.

I suppose what you might do — from a network standpoint, don’t know off-the-cuff how to do it on OpenWRT, though if you’re just using it as a generic Linux machine, without using any OpenWRT-specific stuff, I’m pretty sure that it’s possible — is to give the OpenWRT machine two non-routable IP addresses, something like:

192.168.1.1 for the LAN

and

192.168.2.1 for the DMZ

The DHCP server serves DHCP responses for the LAN that tell it to use 192.168.1.1 as the default route, and 192.168.2.1 for hosts in the DMZ. So, for example, the server in the DMZ would maybe be assigned 192.168.2.2.

Then it should be possible to have a routing table entry to route 192.168.1.1 to 192.168.2.1 and vice versa, just as as if the two addresses were actually physical routers.

When a LAN host initiates a TCP connection to a DMZ host, it’ll look up its IP address in its routing table, say “hey, that isn’t on the same network as me, send it to the default route”. That’ll go to 192.168.1.1, with a destination address of 192.168.2.2. The OpenWRT box forwards it, doing IP routing, to 192.168.2.1, and then that box says “ah, that’s on my network, send it out the network port with VLAN tag whatever” and the switch fabric is configured to segregate the ports based on VLAN tag, and only sends the packet out the port associated with the DMZ.

The problem is that the reason that you want to use NAT is to disallow incoming connections from the server to the LAN. This will make that go away — the LAN hosts and DMZ hosts will be on separate “networks”, so things like ARP requests and other stuff at the purely-Ethernet level won’t reach each other, but they can freely communicate with each other at the IP level, because the two 192.168.X.1 virtual addresses will forward packets between each other. You’re going to need to firewall off incoming TCP connections (and maybe UDP and ICMP and whatever else you want to block) inbound on the 192.168.1.0/24 network from the 192.168.2.0/24 network. You can probably do that with iptables at the Linux level. I think that all the traffic should be reaching the Linux kernel in this scenario.

If you get that set up, hosts at 192.168.2.2, on the DMZ, should be able to see connections from 192.168.1.2, on the LAN, using its original IP address.

That should work if what you had was a Linux box with two Ethernet cards and the VLAN stuff wasn’t in the picture. I’m not 100% certain that any VLAN switching fabric stuff might muck that up — I’ve only very rarely touched VLANs myself, and never tried to do this. But I can believe that it’d work.

If you do set it up, I’d also fire up tcpdump on the server. If things are working correctly, sudo ping -b 192.168.1.255 on the LAN shouldn’t show up as reaching the server. However, ping 192.168.2.1 should.