I’m mystified how it’s any different than downloading off someone’s server

A huge difference is you actually are downloading off someone’s server - unlike torrents where you’re actively participating in the distribution of pirated material. If you ever do end up in court… that difference will be important.