Try a different browser, or the curl command in another comment. Your understanding so far is correct, though unusual, typically it’s not recommended to put LAN records in WAN DNS.

But if you’ve ever run HTTPS there before, Firefox might remember that and try to use it automatically. I think there’s a setting in Firefox. You might also try the function to forget site information, both for the name and IP. I assume you haven’t turned on any HTTP-to-HTTPS redirect in nginx.

Also verify that nginx is set up with a site for that name, or has a default site. If it doesn’t, then it doesn’t know what to do and will fail.