Depends strongly on what ops the NPU supports IMO. I don’t do any local gen AI stuff but I do use ML tools for image processing in photography (e.g. lightroom’s denoise feature, GraXpert denoise and gradient extraction for astrophotography). These tools are horribly slow on CPU. If the NPU supports the right software frameworks and data types then it might be nice here.
humanspiral@lemmy.ca 1 week ago
NPUs are very scammy, with all use vendor specific proprietary, often undocumented, implementations that are often incompatible with previous vendor architectures. Microsoft is makeing DirectML, but AMD/Intel (different NPUs that keep changing) aren’t fully supported. Copilot does manage to do some minimal AI use. Their small LLM is snapdragon elite only. but 27 tokens/s for 1.6gb ram (4 bit int quantized) is much lower than x86 (or gpu) performance on similar sized models. ultra low power use is the benefit, but so far, any chip die space given to NPU is, IMO, a waste of money, partly because it is a dark black box that only Microsoft has the key to.
KingRandomGuy@lemmy.world 1 week ago
Yeah I agree on these fronts. The hardware might be good but software frameworks need to support it, which historically has been very hit or miss.