Yeah, as said, that’s going to be due to however your TV upscales things.

Getting a dedicated scaler will help, but bear in mind that you’re taking an image that’s at least 1/4 smaller (might be less than that, can’t remember the math off hand) than your screen’s native resolution and zooming in on it.

Using most scalers gives you a ton of options for how to zoom in. Straight pixels, bilinear (or trilinear) filtering, and some have various shader effects as well to emulate the style of TVs these consoles were made for.

By using HDMI/VGA you are getting the clearest digital signal version of that image possible, but it’s still tiny, and any way you choose to expand it will have pros and cons.