Never point your DNS at two different IP addresses like this. It will only cause you pain and unexpected behaviour.

Why?

I have a similar setup, but to add to the problem, I’m also behind CGNAT. Here’s my setup:

• LAN - 192.168… addresses
• WAN - 10… address from ISP
• VPS - public address

To access my LAN from outside, I have a WireGuard tunnel to my VPS.

The address my DNS resolves to is absolutely unrelated to any addresses my router understands. So to prevent traffic to my locally hosted resources from leaving my LAN, I need my DNS to resolve to local addresses. So I configured static DNS entries on my router to point to local addresses, and I have DHCP provide my router as the primary DNS source and something else as a backup.

This works really well, and TLS works as expected both on my LAN and from outside my LAN. The issue OP is seeing is probably with a non-configured device somewhere that’s not querying the local DNS server.