Comment on Selfhosted & AI
curbstickle@anarchist.nexus 22 hours agoCongrats, you are in the 1%
Sorry, I’m not following here. Are you suggesting I’m in the 1% of income earners? Because if so… LOL not even close, I’m barely in the top 40% by rough math.
Just to note, an apple silicon Mac is one of the most efficient (dollars, wattage, whatever take your pic) ways to run an LLM. Mine was a build target for iOS stuff for a client I’m now repurposing, that was a refurb that cost about $600.
I can’t even buy a GPU for that price these days.
midribbon_action@lemmy.blahaj.zone 22 hours ago
You are in the 1% of llm users.
Do you think anybody is choosing m2s over nvidia? Everybody running on cpu dreams of being rich enough to do real gold digging on an ampere. Amd is the only direct competitor, an order of magnitude smaller, and supports fewer models. I don’t think any hobbyist has one of these alternatives at the top of their wishlist, they are substitutes.
curbstickle@anarchist.nexus 21 hours ago
I have two clients I just set up on-prem LLM for with a cluster of Mac studios. Another already had about a half dozen custom servers with radeon pros doing a different job which got repurposed for on-prem.
I don’t think the wishlist really matters, honestly. That’d just be pointing to the marketing team at NVidia IMO.
I actually don’t have any really current NVidia hardware myself, I have mostly AMD GPUs, though ive been looking to pick up an Intel for AV1 purposes. I’d also mention that I recommend apple often (for this specific purpose only) due to their efficiency and power use.
In any case, that doesn’t change the reality here - there is no single specific manufacturer that must be used, and all an LLM is, is software. Its not to blame for what companies are doing any more than Linux is to blame because its preferred as a server OS.
midribbon_action@lemmy.blahaj.zone 20 hours ago
OK, but still, just a clarification like “I’m not buying nor will I ever buy nvidia chips, now here’s a thing I made with ai…” is enough context, in my opinion. Just blindly saying “ai is good for this” is promoting the bubble, as the vast vast majority of llm users are running nvidia chips.
Also, whatever financials you showed to convince them it was a good investment, whatever type of business it is, I think it was bullshit. I haven’t seen any proof of efficiency gains at any company rollout of ai, certainly not on macs. Most companies are currently pulling back on token usage is my understanding. ROI has been unmeasurable so far. At best, you informed your clients explicitly this was a highly speculative purchase and may not benefit them.
curbstickle@anarchist.nexus 20 hours ago
Thats certainly a take.
I don’t show financials or propose these decisions, I get paid to design and sometimes implement.
As far as whether or not its a benefit, I’m going to have to completely disagree. As I previously mentioned - its a tool. They are great at detecting potential security issues in code data extraction and classification (especially with unstructured or poorly structured sources, like PDFs), knowledge base searches (especially where that knowledge may be spread across multiple internal sources like a wiki, memos, miscellaneous docs, etc), doc review for tone to meet standards, etc.
Your statement that it essentially doesn’t pay to use llms is intrinsically tied to the OpenAI/MS/NVidia/Anthropic/etc “everything can be done with AI now!” marketing nonsense just the same as believing it has payoff for all scenarios. You recognize the “all uses are good uses” as being complete bs, but you’re jumping to “that means no uses are good uses”.
And that is decidedly not true.
The best example is that data ingest I mentioned. I had a client looking to bring in a bunch of differently formatted forms to a database. What they had been doing was taking their regular employees who handle these forms and using them for data entry - a pretty poor use of their time.
Instead, these scans were evaluated by a tuned model specific to their needs. Each form has a unique ID (though the way it could be numbered was very different), which then gets assigned to one of these folks for review at ingest. They are given a new unique number, and a verification flag (3 stages - first employee review, second employee review, and final import acceptance) which was basically the same flow as the previous setup.
The difference is that each person didnt need to hunt across the form to find the details. When the comparison comes up for approval at each stage, they get the snippet being brought in and the field its being applied to. It can be approved for that field, sent back for reevaluation, or sent for human only review (often this is because the scan sucked).
The project took less than 10% of the original timeframe, and the people handling the forms (and previously assigned for ingest) didn’t end up with the stupidly increased workload that originally got assigned.
Again, using a tool at what its good for is what’s important. Using it for what you think that it can do (ie: the executive method) is just piss poor practice due to easily convinced c suite who gobble up marketing nonsense.