Comment on Llama 3.1 is Meta's latest salvo in the battle for AI dominance
tonyn@lemmy.ml 2 months agoI’ve got llama 3.1 8b running locally in open webui. What do you mean it’s bugged with llama.cpp?
Comment on Llama 3.1 is Meta's latest salvo in the battle for AI dominance
tonyn@lemmy.ml 2 months agoI’ve got llama 3.1 8b running locally in open webui. What do you mean it’s bugged with llama.cpp?
brucethemoose@lemmy.world 2 months ago
llama.cpp, the underlying engine, doesn’t support extended RoPE yet. Basically this means long context doesnt work and short context could be messed up too.
I am also hearing rumblings of a messed up chat template?
Basically with any LLM in any UI that uses a GGUF, you have to be very careful of bugs you wouldn’t get in the huggingface-based backends.
FaceDeer@fedia.io 2 months ago
I wouldn't call it a "dud" on that basis. Lots of models come out with lagging support on the various inference engines, it's a fast-movibg field.
brucethemoose@lemmy.world 2 months ago
Yeah, but it leaves a bad initial impression when all the frontends ship it and the users aren’t aware its bugged.