Put the exhaust fan on separate switch and don’t use it while you’re in there. That’s what I’ve been doing, most Japanese baths are wired that way.

Older Japanese construction can have a small window to the outside to help with ventilation. The only reason it would be drafty is if that window is open/won’t seal properly or if the fan is pulling the heat out too quickly.