Comment on Llama 3.1 is Meta's latest salvo in the battle for AI dominance
brucethemoose@lemmy.world 2 months ago
IMO the more interesting models are 70B and 8B, aka the first models you can run yourself and (for basically the first time) the first open models distilled from such a large “parent” model.
But the release is a total dud among testers because they’re bugged with llama.cpp, lol.
tonyn@lemmy.ml 2 months ago
I’ve got llama 3.1 8b running locally in open webui. What do you mean it’s bugged with llama.cpp?
brucethemoose@lemmy.world 2 months ago
llama.cpp, the underlying engine, doesn’t support extended RoPE yet. Basically this means long context doesnt work and short context could be messed up too.
I am also hearing rumblings of a messed up chat template?
Basically with any LLM in any UI that uses a GGUF, you have to be very careful of bugs you wouldn’t get in the huggingface-based backends.
FaceDeer@fedia.io 2 months ago
I wouldn't call it a "dud" on that basis. Lots of models come out with lagging support on the various inference engines, it's a fast-movibg field.
brucethemoose@lemmy.world 2 months ago
Yeah, but it leaves a bad initial impression when all the frontends ship it and the users aren’t aware its bugged.