I have one more doubt.

(Nothing setup as of now, just checking all possible angles)

Since, as per your suggestion, I have passed the NIC to OPNsense, so Proxmox won’t see the dual NIC card. The onboard ethernet port eth3 is connected to a switch and Proxmox will use eth3.

Image Here’s a fresh diagram

You can see that I have a Ubuntu VM. How will the PC communicate with the VM? (Links E and D are hypothetical. I don’t know which one is better given my scenario). Will it be C -> B -> D or C -> A -> E?

Case #1: C -> B -> D

The switch must differentiate between uplink traffic/proxmox traffic/other physical devices connected to the switch (phone/TV). I prefer this solution because even if OPNsense is down, I can still access the Ubuntu server. But the switch must handle this complexity. I will be getting a managed switch anyway.

Case #2: C -> A -> E

The interface eth2, which OPNsense is using, must have some sort of routing table which redirects to links E or F depending upon destination IP (Probably implemented using virtual bridge or virtual switch). This is simpler for the switch but if OPNsense is down, i lose direct network access to the VM. I can probably access it via Proxmox web-interface.

Can you tell me which one is preferred or which one you would have done?

Sorry for my ramblings.