I’ve just been down this exact journey, and ended up settling on Kavita. It has all the browse, search and library stuff you’d expect. You can download or read things in the web interface. I’m only using it for epub and PDF books, but its focus is comics and manga so I expect it to shine there.

I don’t think it does mobi, but since I use Calibre on my laptop to neaten up covers and metadata before I drop books on to the server it’s a simple matter to convert the odd mobi I end up with. Installation (using docker inside an LXC) was simple.

It’s been a really straightforward, good experience. Highly recommend. I like it better than AudioBookshelf (which I’m already hosting for audio books) which I also tried, but didn’t like as much for inexplicable reasons. I also considered Calibre-Web, but that seemed a bit messy since I guess I’d use Calibre on my laptop to manage my books on a NAS share then serve it headless from the server with Calibre-Web? I might have that completely wrong, I didn’t spend any time looking into it because Kavita was the second thing I tried and it did exactly what I wanted.