Comment on Llama 3.1 AI Models Have Officially Released
admin@lemmy.my-box.dev 2 months agoYeah, there’s a massive negative circlejerk going on, but mostly with parroted arguments. Being able to locally run a model with this kind of context is huge. Can’t wait for the finetunes that will result from this (*cough* NeverSleep’s *-maid models come to mind).
brucethemoose@lemmy.world 2 months ago
I am looking into doing it on the 12B for myself TBH, not so much for RP but novel style prose.
admin@lemmy.my-box.dev 2 months ago
Ah, that’s a wonderful use case. One of my favourite models has a storytelling lora applied to it, maybe that would be useful to you too?
At any rate, if you’d end up publishing your model, I’d love to hear about it.
brucethemoose@lemmy.world 2 months ago
[Oh, my friend, you have to switch to this: huggingface.co/BeaverAI/mistral-doryV2-12b
It’s so much smarter than llama 13B. And it goes all the way out to 128K!
admin@lemmy.my-box.dev 2 months ago
Oof - not on my 12gb 3060 it doesn’t :/ Even at 48k context and the Q4_K quantization, it’s ollama its doing a lot of offloading to the cpu. What kind of hardware are you running it on?