Comment on The first GPT-4-class AI model anyone can download has arrived: Llama 405B

• cheddar@programming.dev ⁨1⁩ ⁨month⁩ ago

  Typically you need about 1GB graphics RAM for each billion parameters (i.e. one byte per parameter). This is a 405B parameter model.

  Image

  source
• Deceptichum@quokk.au ⁨1⁩ ⁨month⁩ ago

  Or you could run it via cpu and ram at a much slower rate.

  source

  • errer@lemmy.world ⁨1⁩ ⁨month⁩ ago

    Yeah uh let me just put in my 512GB ram stick…

    source

    • Deceptichum@quokk.au ⁨1⁩ ⁨month⁩ ago

      Samsung do make them.

      Goodluck finding 512gb of VRAM.

      source
    • bruhduh@lemmy.world ⁨1⁩ ⁨month⁩ ago

      www.ebay.com/p/116332559 lga2011 motherboards quite cheap, insert 2 xeon 2696v4 44 threads each totalling at 88 threads and 8 ddr4 32gb sticks, it comes quite cheap actually, you can also install Nvidia p40 with 24gb each, you can max out this build for ai for under 2000$

      source
  • chiisana@lemmy.chiisana.net ⁨1⁩ ⁨month⁩ ago

    Finally! My dumb dumb 1TB ram server (4x E5-4640 + 32x32GB DDR3 ECC) can shine.

    source
• Siegfried@lemmy.world ⁨1⁩ ⁨month⁩ ago

  At work we habe a small cluster totalling around 4TB of RAM.

  source
• TipRing@lemmy.world ⁨1⁩ ⁨month⁩ ago

  When the 8 bit quants hit, you could probably lease a 128GB system on runpod.

  source
• 1984@lemmy.today ⁨1⁩ ⁨month⁩ ago

  Can you run this in a distributed manner, like with kubernetes and lots of smaller machines?

  source
• obbeel@lemmy.eco.br ⁨1⁩ ⁨month⁩ ago

  According to huggingface, you can run a 34B model using 22.4GBs of RAM max. That’s a RTX 3090 Ti.

  source
• Longpork3@lemmy.nz ⁨1⁩ ⁨month⁩ ago

  Hmm, I probably have that much distributed across my network… maybe I should look into some way of distributing it across multiple gpu

  source
• arefx@lemmy.ml ⁨1⁩ ⁨month⁩ ago

  Ypu mean my 4090 isn’t good enough 🤣😂

  source