Now for the other weird bit. If I enable secret mode, the website works fine.

most likely the bad 404 response was also cached in your web browser. try clearing your cache or doing a reload with ctrl+f5 or cmd+shift+r.