Mixtral GPTQ can run on a 3090
Mistral 7b can run on most modern gpus
Comment on Everybody’s talking about Mistral, an upstart French challenger to OpenAI
Ashyr@sh.itjust.works 11 months ago
It’s neat, but I hear you need a really beefy system to make it work.
It may be an insurmountable hurdle to bring such capabilities to lesser systems, so I’m not necessarily complaining, I just wish it was more accessible.
Mixtral GPTQ can run on a 3090
Mistral 7b can run on most modern gpus
Oh boy, I missed Mixtral GPTQ and only tried Mistral 7b
Currently downloading mixtral-8x7b-v0.1.Q4_K_M.gguf
Thank you!
Be warned, prompt processing is slow
It is indeed. I’m switching to the instruct model to see if I can get better results for code and documentation.
I'm looking forward to the day where these tools will be more accessible, too. I've tried playing with some of these models in the past, but my setup can't handle them yet.
You should definitely try Mistral. It runs on a potato
I'll give it a shot later today, thanks
Are you running llama.cpp and a gguf format of the model?
I believe I was when I tried it before, but it's possible I may have misconfigured things
Have you checked out llama-cpp-python? The API is very simple, from the readme
For this one, you should be able to run it on anything with 8GB of VRAM. That said, it may not be fast. You will probably want a Turing or newer card with as much VRAM bandwidth as possible.
That's good to know. I do have 8GB VRAM, so maybe I'll look into it eventually.
joneskind@lemmy.world 11 months ago
I run it fine on a base model MacBook Air with 8Gb of RAM and absolutely crazy on a 30 GPU cores M2 Max. Didn’t try on my company’s M1 Pro but I will tomorrow.
I use the LMStudio app and download Mistral from there. The heavier model for my beefy Mac and a 3Gb one for the Air.
I tried a lot of models for development purposes and this one blew my mind.
cheese_greater@lemmy.world 11 months ago
Seriously? Might have to try it
joneskind@lemmy.world 11 months ago
You download the model and it’s on your computer for as long as you want. The whole point is to be able to use it locally.
cheese_greater@lemmy.world 11 months ago
So it is entirely local? Schweet! How large is it?