QuantumEraser

Do you think a passive heatsink like this would work? As I understand it 1U servers usually use a passive heatsink with fans pushing air through the case horizontally, rather than an active heatsink with a fan. Noise isn't a problem since it's in a server room so I can cram in as many fans as is necessary.

Yes, hosting providers do run servers on desktop hardware. See for example Bisect's dedicated server options which are all desktop Ryzen processors.

Smaller Minecraft servers can run just fine on standard server hardware, and I'm sure that's what most hosting providers use for their lower tier server plans, since you can cram dozens of Minecraft servers onto one machine. But the main limitation for larger servers is single-threaded performance, which desktop CPUs are just far better at.

For small playercounts a server CPU works fine, but my server regularly has 20+ concurrent players and sometimes has up to 50 so we kinda need the performance.

A typical server unfortunately won't work for my use case, as Minecraft servers are highly single threaded workloads and desktop CPUs are just far better than server CPUs in that regard.

That heatsink says it's rated for up to 125W, which I'm not sure will be enough for the 265K (Intel says 250W at max boost). I don't have a specific chassis in mind; the one I linked was just one I found with a quick search. I haven't purchased server hardware before so I'm not really sure what to look for. Do you think buying a generic microATX compatible chassis and a similar heatsink (higher rated power if I can find one), then adding a ton of high RPM fans and possibly some custom airflow guides to get very high airflow through the heatsink would work?

I can't go with a server grade mobo/CPU as Minecraft servers need very high single threaded performance which you can only really get with desktop CPUs (at least at non-insane prices).

What are you doing that mainly needs single threaded performance, running a Minecraft server?