I’ve thought about doing this too.

Larger FOV for your screen than anything else portable. Potentially more power efficient too, as you don’t blast light everywhere to get a tiny bit into an eye.

The privacy is nice – and I don’t think that that reasonably can be reduced to looking at porn, which I assume most people aren’t going to want to do in public anyway, for obvious reasons. I can throw a password list up onscreen, don’t need to deal with those inane “hide your password as you type it” things that try to mitigate privacy issues with people using computers in public places.

Problem is that at least today, wearing HMDs is not as comfortable and sharp as looking at a display. Also, VR goggles and headphones tend to compete for the same spot around the ears — circumaural headphones need to seal there, so you may need to accept whatever sound, if any, the HMD can provide. You have less awareness of your surroundings, which may or may not matter in a given situation.