Comment on Any external NPU that can be hooked to a proxmox miniPC?
dan@upvote.au 3 days ago
Do you want to run TensorFlow Lite / LiteRT models? PyTorch Mobile? onnx? YOLO? Something else?
Google Coral was decent for TensorFlow Lite, but it’s EOL (end of life) now. I’ve got the dual TPU Mini PCIe version in my home server, via a PCIe adapter board. I use it for object detection with Blue Iris + CodeProject AI and it works pretty well for that use case.
iturnedintoanewt@lemmy.world 3 days ago
Thanks! I was considering using it perhaps to have a diffusion model? Or running ollama or similar without a RAM hit on my laptop or phone. Also whisper comes to mind, for Bazarr or other tools to use.
yellowbadbeast@lemmy.blahaj.zone 3 days ago
Any TPU you can buy is going to be essentially useless for either image diffusion or LLMs. The onboard RAM is both far too small and far too slow (LLM text generation speed relies on RAM speed first and foremost, and both LLMs and image models tend to be, you know, big), and USB isn’t nearly fast enough to help with that, Not to mention that software support is pretty much nonexistent. You’d be better off upgrading the GPU to a 3060ti 12gb or something.
P.S. A word of advice, consider using something other than Ollama. Llama.cpp in router mode or llama-swap support pretty much all of the functionality that Ollama does without being crap. Ik_llama.cpp is also nice if you have a CPU/Nvidia setup.
dan@upvote.au 2 days ago
I wish more modern TPUs were available for purchase. Google’s TPU v8i is very powerful, but you can only rent it via Google Cloud. (that and they’re most useful in big clusters, and the pricing for such a cluster is definitely out of reach for anyone that’s not running their own data center)
echo@lemmy.ml 2 days ago
Qwen3.8 is out now and blows Qwen3.6 out of the water in pretty much every metric
yellowbadbeast@lemmy.blahaj.zone 2 days ago
AFAIK there isn’t a sparse 35B variant of 3.8, and I would not recommend running a dense 27B model on dual-channel DDR4 unless you are exceptionally patient.
hendrik@palaver.p3x.de 3 days ago
Aren’t diffusion models and LLMs (ollama) too big for a NPU? As far as I know something like a Coral runs specific models only. And it’s limited to the 1 or 2GB of memory on it. It’d do tasks like voice recognition, or image classification. But not generate images or text.
frongt@lemmy.zip 3 days ago
Agreed. OP should probably upgrade to a bigger case and discrete graphics card.
iturnedintoanewt@lemmy.world 3 days ago
Thanks! I was seeing if there was a way to do it while avoiding a full blown desktop PC case :(