Comment on Apple officially unveils M3, M3 Pro, and M3 Max: 3 nanometer, Dynamic Caching GPU, more
NotMyOldRedditName@lemmy.world 11 months agoThe memory maximums are going to be more and more important when it comes to local AI applications.
Take language models for an example
To run a 30b model, you need 24gb of video ram to do it fully on the video card. That’s a nvidia 3090 or 4090 today. But in the grand scheme of things, 30b is small. They are going to get much bigger, especially when you want larger contexts which allow the AI to remember more.
Apples memory is unified, so it can be system ram, or video ram. You’ll be able to easily load a 70b model into it for example, where you’d need 2 3090s or 4080s and a hefty PSU on a current Gen non Mac PC.
For the moment, things are better optimized for windows and nvidia hardware, but Apple is encroaching on this space, and their huge amounts of video memory will begin to unlock using and training larger and larger models.
Expect to see nvidia starting to offer higher video ram cards as well for this exact reason.
BetaDoggo_@lemmy.world 11 months ago
I can’t see local models or hardware needing to scale much past the sizes we already have. Recent models like mistral have shown that we are still far from saturation at current model sizes.
NotMyOldRedditName@lemmy.world 11 months ago
And we only ever needed 64kb of ram.
Even if we have a lot of room to optimize and grow within what we have, we still have so much more to do.
Fully coherent audio and video synthesis for a scene for example.
And these models are being trained on server farms, but thats just because video memory is so expensive to come by.
We’re just starting to crawl, we haven’t even started walking yet on where this is going.
BetaDoggo_@lemmy.world 11 months ago
I was mainly referring to language models which have somewhat predictable scaling laws. It doesn’t make sense to continue scaling the parameters when you can scale the data instead.
Diffusion models are a completely different domain which is less established. Most advancements made in that space are related to the architecture and training methodology. In terms of scale they haven’t changed much.
Large models will always be trained in datacenters because the compute will always be exponentially greater and cheaper than what you could get as an individual. Local finetuning already happens but it’s expensive and limited.