Comment on Liking an OS isn't a personality trait ❌

• Linkerbaan@lemmy.world ⁨7⁩ ⁨months⁩ ago

  Huggingface usually Mixtral recently released a pretty good model that’s not very big

  source
• FiniteBanjo@lemmy.today ⁨7⁩ ⁨months⁩ ago

  Technically the technology is open to the public but regular people cannot afford to implement it.

  The thing that makes Large Language Models hardly functional is scaling up their databases and processing power of one of several of their small models with specialized tasks. One model creates output from input, another model checks it for accuracy, a third model polices it for things that are not allowed.

  So unless you’ve got a datacenter and three high powered servers with top-grade cooling systems and a military grade power supply, fat fucking chance.

  source

  • AdrianTheFrog@lemmy.world ⁨7⁩ ⁨months⁩ ago

    I can run a small LLM on my 3060, but most of those models were originally trained on a cluster of a100s (maybe as few as 10, so more like one largish server than one datacenter)

    Bitnet came out recently and is looking like it will lower these requirements significantly (essentially training a model using ternary numbers instead of floats to reduce requirements, which turns out to not lower the quality that significantly)

    source
  • Kolanaki@yiffit.net ⁨7⁩ ⁨months⁩ ago

    They should do to AI what they make me do at work: More with less.

    source
  • Simon@lemmy.dbzer0.com ⁨7⁩ ⁨months⁩ ago

    Haha this really shows how much you know about the subject. Try the entire datacenter.

    If LLM was practical on three servers everyone and their mum would have an AI assistant product.

    source
  • OozingPositron@feddit.cl ⁨7⁩ ⁨months⁩ ago

    Basically Mistral, check /lmg/ in /g/, if you have a GPU newer than 2 years you can probably run a 32B quantised model.

    source