Comment on Nvidia delivers first Vera Rubin AI GPU samples to customers — 88-core Vera CPU paired with Rubin GPUs with 288 GB of HBM4 memory apiece

• AliasAKA@lemmy.world ⁨1⁩ ⁨day⁩ ago

  Current models are speculated at 700 billion parameters plus. At 32 bit precision (half float), that’s 2.8TB of RAM per model, or about 10 of these units. There are ways to lower it, but if you’re trying to run full precision (say for training) you’d use over 2x this, something like maybe 4x depending on how you store gradients and updates, and then running full precision I’d reckon at 32bit probably. Possible I suppose they train at 32bit but I’d be kind of surprised.

  source

  • in_my_honest_opinion@piefed.social ⁨1⁩ ⁨day⁩ ago

    Sure, but giant context models are still more prone to hallucination and reinforcing confidence loops where they keep spitting out the same wrong result a different way.

    source

    • AliasAKA@lemmy.world ⁨21⁩ ⁨hours⁩ ago

      Sorry, I’m not saying that’s a good thing. It’s not just the context that’s expanding, but the parameter of the base model. I’m saying at some point you just have saved a compressed version of the majority of the content (we’re already kind of there) and you’d be able to decompress it even more losslessly. This doesn’t make it more useful for anything other than recreating copyrighted works.

      source

      • in_my_honest_opinion@piefed.social ⁨18⁩ ⁨hours⁩ ago

        Ah I see, however you do bring up another point. I really think we need a true collection of experts able to communicate without the need for natural language and then a “translation” layer to output natural language or images to the user. The larger parameters would allow the injection of experts into the pipeline.

        Thanks for the clarification, and also for the idea. I think one thing we can all agree on is that the field is expanding faster than any billionaire or company understands.

        source

        • -> View More Comments
• in_my_honest_opinion@piefed.social ⁨1⁩ ⁨day⁩ ago

  Fundamentally no, linear progress requires exponential resources. The below article is about AGI but transformer based models will not benefit from just more grunt. We’re at the software stage of the problem now. But that doesn’t sign fat checks, so the big companies are incentivized to print money by developing more hardware.

  https://timdettmers.com/2025/12/10/why-agi-will-not-happen/

  Also the industry is running out of training data

  https://arxiv.org/html/2602.21462v1

  What we need are more efficient models, and better harnessing. Or a different approach, reinforced learning applied to RNNs that use transformers has been showing promise.

  source

  • TropicalDingdong@lemmy.world ⁨1⁩ ⁨day⁩ ago

    Yeah I’ve read that before. I don’t necessarily agree with their framework. And even working within their framework, this article is about a challenge to their third bullet.

    I’m just not quite ready to rule out the idea that if you can scale single models above a certain boundary, you’ll get a fundamentally different/ novel behavior. This is consistent with other networked systems, and somewhat consistent with the original performance leaps we saw (the ones I think really matter are ones from 2019-2023, its really plateaued since and is mostly engineering tittering at the edges). It genuinely could be that 8 in a MoE configuration with single models maxing out each one could actually show a very different level of performance. We just don’t know because we just can’t test that with the current generation of hardware.

    Its possible there really is something “just around the corner”; possible and unlikely.

    source

    • in_my_honest_opinion@piefed.social ⁨16⁩ ⁨hours⁩ ago

      I mean what you’re proposing was the initial push of gpt3. All the experts said, these GPTs will only hallucinate more with more resources and they’ll never do anything more than repeat their training data as a word salad posing as novelty.

      https://gwern.net/scaling-hypothesis

      source
• boonhet@sopuli.xyz ⁨1⁩ ⁨day⁩ ago

  LLMs can already use way more I believe, they don’t really run them a single one of these things.

  The HBM4 would likely be great for speed though.

  source
• Cocodapuf@lemmy.world ⁨22⁩ ⁨hours⁩ ago

  Lol, this was literally my exact response

  lemmy.world/comment/22356808

  I feel you man.

  source
• panda_abyss@lemmy.ca ⁨1⁩ ⁨day⁩ ago

  Yeah they’re going to cost as much as a house.

  I think we’ll see much larger active portions of larger MOEs, and larger context windows, which would be useful.

  The non LLM models I run would benefit a lot from this, but I don’t know of I’ll ever be able to justify the cost of how much they’ll be.

  source