Comment on Everybody’s talking about Mistral, an upstart French challenger to OpenAI

Ashyr@sh.itjust.works ⁨9⁩ ⁨months⁩ ago

• joneskind@lemmy.world ⁨9⁩ ⁨months⁩ ago

  I run it fine on a base model MacBook Air with 8Gb of RAM and absolutely crazy on a 30 GPU cores M2 Max. Didn’t try on my company’s M1 Pro but I will tomorrow.

  I use the LMStudio app and download Mistral from there. The heavier model for my beefy Mac and a 3Gb one for the Air.

  I tried a lot of models for development purposes and this one blew my mind.

  source

  • cheese_greater@lemmy.world ⁨9⁩ ⁨months⁩ ago

    Seriously? Might have to try it

    source

    • joneskind@lemmy.world ⁨9⁩ ⁨months⁩ ago

      You download the model and it’s on your computer for as long as you want. The whole point is to be able to use it locally.

      source

      • cheese_greater@lemmy.world ⁨9⁩ ⁨months⁩ ago

        So it is entirely local? Schweet! How large is it?

        source

        • -> View More Comments
• bioemerl@kbin.social ⁨9⁩ ⁨months⁩ ago

  Mixtral GPTQ can run on a 3090

  Mistral 7b can run on most modern gpus

  source

  • joneskind@lemmy.world ⁨9⁩ ⁨months⁩ ago

    Oh boy, I missed Mixtral GPTQ and only tried Mistral 7b

    Currently downloading mixtral-8x7b-v0.1.Q4_K_M.gguf

    Thank you!

    source

    • bioemerl@kbin.social ⁨9⁩ ⁨months⁩ ago

      Be warned, prompt processing is slow

      source

      • joneskind@lemmy.world ⁨9⁩ ⁨months⁩ ago

        It is indeed. I’m switching to the instruct model to see if I can get better results for code and documentation.

        source
• daredevil@kbin.social ⁨9⁩ ⁨months⁩ ago

  I'm looking forward to the day where these tools will be more accessible, too. I've tried playing with some of these models in the past, but my setup can't handle them yet.

  source

  • joneskind@lemmy.world ⁨9⁩ ⁨months⁩ ago

    You should definitely try Mistral. It runs on a potato

    source

    • daredevil@kbin.social ⁨9⁩ ⁨months⁩ ago

      I'll give it a shot later today, thanks

      source
  • GBU_28@lemm.ee ⁨9⁩ ⁨months⁩ ago

    Are you running llama.cpp and a gguf format of the model?

    source

    • daredevil@kbin.social ⁨9⁩ ⁨months⁩ ago

      I believe I was when I tried it before, but it's possible I may have misconfigured things

      source

      • GBU_28@lemm.ee ⁨9⁩ ⁨months⁩ ago

        Have you checked out llama-cpp-python? The API is very simple, from the readme

        source

        • -> View More Comments
• iopq@lemmy.world ⁨9⁩ ⁨months⁩ ago

  For this one, you should be able to run it on anything with 8GB of VRAM. That said, it may not be fast. You will probably want a Turing or newer card with as much VRAM bandwidth as possible.

  source

  • daredevil@kbin.social ⁨9⁩ ⁨months⁩ ago

    That's good to know. I do have 8GB VRAM, so maybe I'll look into it eventually.

    source