Comment on Is anyone self-hosting their own AI?

ProperlyProperTea@lemmy.ml ⁨1⁩ ⁨week⁩ ago

Last night I managed to run Qwen Flash Next iq2 XS using Strata in docker on a VM

I passed through my 3090 64GB of DDR4 12 vcores of my 5960x

Managed 90t/s

I’ll probably try a larger version soon but QFN already seems a lot more capable than 3.6 35b A3B, and runs better than Qwen 3.8 dense in llama.cpp

original
Sort:hotnewtop