I occasionally find 3.5" floppy disks at the local thrift stores. There are usually new old stock disks on ebay too.

The USB floppy drives usually only support IBM formatted disks and are useless for data recovery. For any other formats, a Greaseweazle will come in very handy.