Comment on Selfhosted & AI
curbstickle@anarchist.nexus 1 day agoI run my local models on a Mac mini m2, but I could also be using AMD (ROCm), Intel with OpenVINO, or just CPU. Simple edge applications I could even use something like an RK3588.
Being tied to NVidia is marketing from NVidia, not the reality of LLMs.
midribbon_action@lemmy.blahaj.zone 1 day ago
Congrats, you are in the 1%. And that is exactly the type of additional context I think is necessary when discussing it.
curbstickle@anarchist.nexus 1 day ago
Sorry, I’m not following here. Are you suggesting I’m in the 1% of income earners? Because if so… LOL not even close, I’m barely in the top 40% by rough math.
Just to note, an apple silicon Mac is one of the most efficient (dollars, wattage, whatever take your pic) ways to run an LLM. Mine was a build target for iOS stuff for a client I’m now repurposing, that was a refurb that cost about $600.
I can’t even buy a GPU for that price these days.
midribbon_action@lemmy.blahaj.zone 1 day ago
You are in the 1% of llm users.
Do you think anybody is choosing m2s over nvidia? Everybody running on cpu dreams of being rich enough to do real gold digging on an ampere. Amd is the only direct competitor, an order of magnitude smaller, and supports fewer models. I don’t think any hobbyist has one of these alternatives at the top of their wishlist, they are substitutes.
curbstickle@anarchist.nexus 1 day ago
I have two clients I just set up on-prem LLM for with a cluster of Mac studios. Another already had about a half dozen custom servers with radeon pros doing a different job which got repurposed for on-prem.
I don’t think the wishlist really matters, honestly. That’d just be pointing to the marketing team at NVidia IMO.
I actually don’t have any really current NVidia hardware myself, I have mostly AMD GPUs, though ive been looking to pick up an Intel for AV1 purposes. I’d also mention that I recommend apple often (for this specific purpose only) due to their efficiency and power use.
In any case, that doesn’t change the reality here - there is no single specific manufacturer that must be used, and all an LLM is, is software. Its not to blame for what companies are doing any more than Linux is to blame because its preferred as a server OS.