Some suggestions for solid state alternatives as old mechanical magnetic storage has such a high failure rate. But retro whatever way you want, of course!

You can get ide to compact flash or ide to sata adapters and get some reasonably modern solid state hard drive storage in there, if that interests you. I understand (haven’t tried personally) that compatibility can be kind of a crapshoot though.

You can also get a gotek which has a floppy interface and can load floppy images from a USB thumbdrive. Which might be a more functional option than getting a USB floppy drive for a modern machine.