Power corrupts, we’ve know this since the dawn of time
Comment on AI boom could falter without wider adoption, Microsoft chief Satya Nadella warns
Rekall_Incorporated@piefed.social 20 hours ago
Nadella maybe know a lot more than any of us about LLMs/GenAI tech, but one doesn’t need to know anything about LLMs (or even technology) to know that an oligarch like Nadella cannot be trusted (in any context).
Grandwolf319@sh.itjust.works 19 hours ago
tal@lemmy.today 20 hours ago
I’m kind of more-sympathetic to Microsoft than to some of the other companies involved.
Microsoft is trying to leverage the Windows platform that they control to do local LLM use. I’m not at all sure that there’s actually enough memory out there to do that, or that it’s cost-effective to put a ton of memory and compute capacity in everyone’s home rather than time-sharing hardware in datacenters. Nor am I sold that laptops are a fantastic place to be doing a lot of heavyweight parallel compute.
But…from a privacy standpoint, I kind of would like local LLMs to be at least available, even if they aren’t as affordable as cloud-based stuff. And at least Microsoft is at least supporting that route. A lot of companies are going to be oriented towards just doing AI stuff in the cloud.
makyo@lemmy.world 20 hours ago
Is that true? I haven’t heard MS say anything about enabling local LLMs. Genuinely curious and would like to know more.
tal@lemmy.today 20 hours ago
That’s why they have the “Copilot PC” hardware requirement, because they’re using an NPU on the local machine.
searches
learn.microsoft.com/en-us/windows/…/npu-devices/
Copilot+ PCs are a new class of Windows 11 hardware powered by a high-performance Neural Processing Unit (NPU) — a specialized computer chip for AI-intensive processes like real-time translations and image generation—that can perform more than 40 trillion operations per second (TOPS).
It’s not…terribly beefy. Like, I have a Framework Desktop with an APU and 128GB of memory that schlorps down 120W or something, kind of out of what you’re going to do on a laptop. And that in turn is weaker computationally than something like the big Nvidia hardware going into datacenters.
But it is doing local computation.
4am@lemmy.zip 19 hours ago
Microsoft wants developers to have local access to models but end users are 100% corralled into OneDrive and Copilot. I’m not sympathetic to them at all.
Feyd@programming.dev 19 hours ago
They’re trying to leverage their windows platform to seek rent (sell premium cloud services like LLM access) for shit people don’t even want because they aren’t satisfied making very respectable money on licenses.
wonderingwanderer@sopuli.xyz 17 hours ago
If Microsoft cared about privacy then they wouldn’t have made windows practically spyware. Even if they install AI locally in the OS, it’s still proprietary software that constantly sends data back to the mothership, consuming your electricity and RAM to do so. Linux has so many options, there’s really no reason not to switch.
Small LLMs already exist for local self-hosting, and there are open-source options which won’t steal your data and turn you into a product.
huggingface.co/spaces/…/open_llm_leaderboard#/
Bear in mind that the number of parameters your system can handle is limited by how much memory is available, and using a quantized version can increase the number of parameters you can handle with the same amount of memory.
Unless you have some really serious hardware, 24 billion parameters is probably the maximum that would be practical for self-hosting on a reasonable hobbyist set-up. But I’m no expert, so do some research and calculate for yourself what your system can handle.
tal@lemmy.today 16 hours ago
Unless you have some really serious hardware, 24 billion parameters is probably the maximum that would be practical for self-hosting on a reasonable hobbyist set-up.
Eh…I don’t know if you’d call it “really serious hardware”, but when I picked up my 128GB Framework Desktop, it was $2k (without storage), and that box is often described as being aimed at the hobbyist AI market. That’s pricier more than most video cards, but an AMD Radeon RX 7900 XTX GPU was north of $1k, an NVidia RTX 4090 was about $2k, and it looks like the NVidia RTX 5090 is presently something over $3k (and rising) on EBay, well over MSRP. None of those GPUs are dedicated hardware aimed at doing AI compute, just high-end cards aimed at playing games that people have used to do AI stuff on.
I think that the largest LLM I’ve run on it was a 106 billion parameter GLM model at Q4_K_M quantitization on my Framework Desktop. It was certainly usable, and I wasn’t trying to squeeze as large a model as possible on the thing. I’m sure that one could run substantially-larger models.
wonderingwanderer@sopuli.xyz 13 hours ago
See, you have more experience in the matter than I do, hence the caveat that I’m not an expert. Thanks for sharing your experience.
Then again, I’d consider 128GB of memory to be fairly serious hardware, but if that’s common among hobbyists then I stand corrected. I was operating on the assumption that 64GB of RAM is already a lot
All in all, 106 billion parameters with quantization on 128GB of memory doesn’t surprise me all that much. But again, I’m just going off of the vague notions I’ve gathered from reading about it.
The focus of my original comment was more on the fact that self-hosting is an option, I wasn’t trying to be too precise with the specs. My bad if it came off that way
Rekall_Incorporated@piefed.social 20 hours ago
I wouldn’t trust a local LLM solution from a large American company. Not saying that they would try to “pull a quick one”, but they are unreliable and corrupt.
lime@feddit.nu 20 hours ago
yeah idk about that first part