Comment on Smaug-72B-v0.1: The New Open-Source LLM Roaring to the Top of the Leaderboard

• nicetriangle@kbin.social ⁨9⁩ ⁨months⁩ ago

  Ok but will this run on my TI-83? It's a + model.

  source

  • Rai@lemmy.dbzer0.com ⁨9⁩ ⁨months⁩ ago

    Only if it’s silver.

    source

    • 01189998819991197253@infosec.pub ⁨9⁩ ⁨months⁩ ago

      Dang. So close.

      source

      • Rai@lemmy.dbzer0.com ⁨9⁩ ⁨months⁩ ago

        My 83 was ganked by some kid I knew so my folks bought me a silver. He denied it. I learned that day to write my name in secret spots.

        source

        • -> View More Comments
  • whoelectroplateuntil@sh.itjust.works ⁨9⁩ ⁨months⁩ ago

    no. but put this clustering software i wrote in ti-basic on 40 million of them? still no

    source
• FaceDeer@kbin.social ⁨9⁩ ⁨months⁩ ago

  It's been discovered that you can reduce the bits per parameter down to 4 or 5 and still get good results. Just saw a paper this morning describing a technique to get down to 2.5 bits per parameter, even, and apparently it 's fine. We'll see if that works out in practice I guess

  source

  • Corngood@lemmy.ml ⁨9⁩ ⁨months⁩ ago

    I’m more experienced with graphics than LM, but wouldn’t that cause a significant increase in computation time, since those aren’t native types for arithmetic? Maybe that’s not a big problem?

    If you have a link for the paper I’d like to check it out.

    source

    • FaceDeer@kbin.social ⁨9⁩ ⁨months⁩ ago

      My understanding is that the bottleneck for the GPU is moving data into and out of it, not the processing of the data once it's in there. So if you can get the whole model crammed into VRAM it's still faster even if you have to do some extra work unpacking and repacking it during processing time.

      The paper was posted on /r/localLLaMA.

      source
    • L_Acacia@lemmy.one ⁨9⁩ ⁨months⁩ ago

      You can take a look at exllama and llama.cpp source code on github if you want to see how it is implemented.

      source
• Amaltheamannen@lemmy.ml ⁨9⁩ ⁨months⁩ ago

  Though with quantisatiom you can get it down to like 30GB of vram or less.

  source
• rs137@lemmy.world ⁨9⁩ ⁨months⁩ ago

  Llama 2 70B with 8b quantization takes around 80GB VRAM if I remember correctly. I’ve tested it a while ago.

  source
• OutrageousUmpire@lemmy.world ⁨9⁩ ⁨months⁩ ago

  Any idea what 8Q requirements would be? Or 4 or 5?

  source

  • General_Effort@lemmy.world ⁨9⁩ ⁨months⁩ ago

    huggingface.co/senseable/…/main

    About 44GB and 50GB for the Q4 and 5. You’d need quite some extra to fully use the 32k context length.

    source