Local AI is one step closer through Mistral-NeMo 12B

Submitted ⁨⁨3⁩ ⁨months⁩ ago⁩ by ⁨IndustryStandard@lemmy.world⁩ to ⁨technology@lemmy.world⁩

https://www.techzine.eu/news/applications/122568/local-ai-is-one-step-closer-through-mistral-nemo-12b/

Mistral NeMo 12B is the name of the new AI model, presented this week by Nvidia and Mistral. “We are fortunate to collaborate with the NVIDIA team, leveraging their top-tier hardware and software,” said Guillaume Lample, cofounder and chief scientist of Mistral AI. “Together, we have developed a model with unprecedented accuracy, flexibility, high-efficiency and enterprise-grade support and security thanks to NVIDIA AI Enterprise deployment.”

The promise of the new AI model is significant. Whereas previous LLMs were tied to datacenters, Mistral NeMo 12B moves to workstations. And it does this without sacrificing performance, or well, that’s the promise.

source

Comments

Sort:hotnew top

bamboo@lemm.ee ⁨3⁩ ⁨months⁩ ago
There are already lots of models in the 7B and 14B ranges that are quite capable and run on commodity hardware. What makes this one so special?

source
- i_like_water@feddit.org ⁨3⁩ ⁨months⁩ ago
  From the top of my head: the context size is way higher. 128k tokens vs 8k usually.
  
  source
  - bamboo@lemm.ee ⁨3⁩ ⁨months⁩ ago
    Oh wow. Yeah a large context size is a significant improvement, doesn’t seem like the article included that detail.
    
    source
- MudMan@fedia.io ⁨3⁩ ⁨months⁩ ago
  Yeah, it seems more interesting to reverse engineer why they chose this line of marketing. They are clearly misrepresenting the challenge and cost of running a LLM locally, so... why?
  
  source
- makingStuffForFun@lemmy.ml ⁨3⁩ ⁨months⁩ ago
  Was wondering the same thing
  
  source
j4k3@lemmy.world ⁨3⁩ ⁨months⁩ ago
::: spoiler I don’t agree with the initial premise about model size and practicality. I run much larger models on my own hardware. 12th gen Intel laptops had a common enthusiast level build that came with a GTX 3080Ti. The Ti is very important here. The 3080 was a 8 GB GPU, while the Ti is a 16 GB version. It is loud as far as fans, and pretty much junk for battery life with the GPU running. However, for a second hand very AI capable all in one setup, you can find one well under $2k. The gigabyte aorus is an example. The ROQ model has more addressable system memory, and that is a nice bonus. Use Linux hardware probe’s website to determine compatibility.

I run quantized 8×7b models which is actually equivalent to ~47b IIRC, (models math accounting for redundancy is not 8×7=56). I also run a quantized 70b and 72b. These are massive.

There is a major assumption people/articles make regarding models. If you’re a developer looking to train models, and limited on hardware resources, you’re going to be playing with stuff like 7b models because you need the full size model to accurately train and that is quite large, but you don’t need this to run the model in most cases. A quantized 7b will fit natively on a flagship phone.

If you just want to run the best model you can for your hardware without training, run the largest model you can while testing quantization. Also look for people that know how to quantize a model well. It involves different types of bit reduction techniques in different areas so that emergent behavior is preserved in the tensors. It is not just float(16) to int(4) across the board.

Models are far more complex than they first seem. Small models are very capable, but they tend to have compounding issues that are difficult to identity if you’re not familiar with them. Larger models tend to have one problem at a time and can self diagnose many of these. Problems are almost always related to safety alignment which is a form of overtraining, and are primarily needed to avoid offending people for our nonsensical cultural biases and inner conflicts we all have just under the surface of our awareness. Prompt momentum can overcome most of the issues. Smaller models need a lot more momentum to be useful. This momentum is hard to create in practice. Larger models require far less momentum to access advanced information.

People tend to use and write articles based on what they can run for free using Google’s data mining stalkerware in exchange for a free (limited) cloud GPU program. This is why few people poor enough to attempt writing for a living do not write articles based on what real consumer hardware can run. There is a 16 GB GPU in the google thing, but it doesn’t have the system memory required to initially load the model or the threads needed to split the model and run like I can. That available setup is not usable.

The best offline AI setup is to run it on a tower as a service to your local network. Get the biggest GPU memory you can afford.

source
- PriorityMotif@lemmy.world ⁨3⁩ ⁨months⁩ ago
  The best GPU to buy right now would be an Intel arc a770. You can get them for under $300 with 16gb vram.
  
  game.intel.com/…/wield-the-power-of-llms-on-intel…
  
  pcpartpicker.com/products/video-card/#P=171798691…
  
  source
  - tau@lemmings.world ⁨3⁩ ⁨months⁩ ago
    Just beware that like AMD, Intel GPUs suffer a performance hit when using LLMs because of the CUDA specific optimizations in frameworks like llama.cpp
    
    source
  - j4k3@lemmy.world ⁨3⁩ ⁨months⁩ ago
    Isn’t there an AMD 16 GB card too, like a 7600 or something?
    
    source
    -> View More Comments
- mememuseum@lemmy.world ⁨3⁩ ⁨months⁩ ago
  The laptop 3080tis have 16 gigs? The desktop ones only got 12.
  
  source
  - j4k3@lemmy.world ⁨3⁩ ⁨months⁩ ago
    Yeah. It is different for laptops. The 3080 market nomenclature is all over the place. Look up the model number to confirm the manufacturer’s specs. Linux hardware probe should list them too. It has been a year since I did all of my research but IIRC the mobile version only came with “3080” 8 GB and “3080Ti” 16 GB. I have the 16 GB and can confirm it is a thing. That is the largest GPU in a laptop for the last generation GTX 3xxx stuff. The largest AMD sold at the same time is a 12 GB 6850 IIRC but the AMD 6k series doesn’t (did not at the time) have the same HIPS ROCM support as the AMD 7k series new stuff (not sure if it has changed).
    
    The following is an old pic from a year ago. The left side is tiled into 3 terminals, top is the running model inference, middle is htop, bottom is my GPU monitoring script that also shows the total memory available. This was also a 70b Llama 2 model (GGML and 4bit quantization)
    
    Image
    
    source
Fisch@discuss.tchncs.de ⁨3⁩ ⁨months⁩ ago
This is only intereating to me if it’s better than Llama 3 8B

source
- General_Effort@lemmy.world ⁨3⁩ ⁨months⁩ ago
  Llama 3.1 is out today. The new 8B should be noticeably better as it’s distilled from the 405B.
  
  source
  - Fisch@discuss.tchncs.de ⁨3⁩ ⁨months⁩ ago
    Thanks for telling me, that’s good to know. Really cool that they also increased context size, especially to such a huge number.
    
    source
- aodhsishaj@lemmy.world ⁨3⁩ ⁨months⁩ ago
  That depends on what you run it on.
  
  source
  - Fisch@discuss.tchncs.de ⁨3⁩ ⁨months⁩ ago
    I’m talking about quality here, not speed
    
    source
fubarx@lemmy.ml ⁨3⁩ ⁨months⁩ ago
Tried running it on a Mac via LM-Studio. Wouldn’t run. Said llama.c needed updating.

source
- realharo@lemm.ee ⁨3⁩ ⁨months⁩ ago
  You can run the model locally in ollama ollama.com/library/mistral-nemo
  
  source