Comment on Is possible to run a LLM on a mini-pc like the GMKtec K8 and K9?
MasterNerd@lemm.ee 4 months ago
Look into ollama. It shouldn’t be an issue if you stick to 7b parameter models
Comment on Is possible to run a LLM on a mini-pc like the GMKtec K8 and K9?
MasterNerd@lemm.ee 4 months ago
Look into ollama. It shouldn’t be an issue if you stick to 7b parameter models
TheBigBrother@lemmy.world 4 months ago
Yeah, I did see something related to what you mentioned and I was quite interested. What about quantized models?
entropicdrift@lemmy.sdf.org 4 months ago
Quantized with more parameters is generally better than floating point with fewer parameters. If you can squeeze a 14b parameter model down to a 4-bit int quantization it’ll still generally outperform a 16-bit Floating Point 7b parameter equivalent.
TheBigBrother@lemmy.world 4 months ago
Interesting information mate, I’m documenting myself into the subject, thx for the help 👍👍
MasterNerd@lemm.ee 4 months ago
I don’t have any experience with them honestly so I can’t help you there
TheBigBrother@lemmy.world 4 months ago
Appreciate you 👍👍