If you aren't transcoding, and the player is taking too long to cache the video before starting, you might be having some sort of storage issue. You would need to try a couple of different things to figure out what, specifically, is taking so long to send the video out.

The first thing that comes to mind is that your storage is on an SSD, and it is nearly full. An SSD that is nearly full will usually perform much much worse than it would if it had more space to work with. https://pureinfotech.com/why-solid-state-drive-ssd-performance-slows-down/

The next thing that comes to mind is that your files are stored on the same drive that jellyfin transcodes onto, and it is not using an SSD. If you have jellyfin reading from a single drive, jellyfin encoding to that same drive, and also everything else also running, you might be causing your hard drive to seek a lot in order to get everything up and running. You could test this by changing the jellyfin transcode location to a different storage device.

I've also found that page and video loading times tend to be directly affected by the storage medium's seek times. If you had jellyfin installed on the same hard drive as your videos, it will be slower than if you had installed jellyfin on a ssd separate from the drive you store your videos on. This one wouldn't likely result in minute loading times though.