if you have a single commodity machine, the cited solutions are useful for deploying small models. If you have several commodity machines, you can’t combine them efficiently with the cited solutions to deploy a large model, and even if you could, it would require a team to manage and maintain the system.
You’re wrong and OP is right. Llama.cpp has the ability to do exactly what “Anyway System” claim to do without the bullshit. Like this claim of " even if you could, without a team to manage it" is so stupid.
There are several frameworks that allow this beside Llama.cpp that are open source, that have been for a while, is extremely maintained.
Also, this seems like a stunt to get VC money. Good on the founders I guess. But the solution from what I read is a nothing burger. My money on llama.cpp or ktransformers or ik_llama.cpp
afk_strats@lemmy.world 1 week ago
Let me see if I can clarify
The article talks about running models on consumer hardware. I am making the point that this is not a new concept. The GUI is optional but, as I mentioned, llama.cpp and other open source tools provide an OpenAI-compatible api just like the product described in the article.
No. LLMs, as we know them, aren’t that old, were a harder to run and required some coding knowledge and environment setup until 3ish years ago, give or take when these more polished tools started coming out.
Ollama matches that description. Llama is a model family from Facebook. Llama.cpp, which is what I was talking about, is an inference and quantization tool suite made for efficient deployment on a variety of hardware including consumer hardware.
Map reduce, in very simplified terms, means spreading out compute work to highly pararelized compute workers. This is, conceptually, how all LLMs are run at scale. You can’t map reduce or parallelize LLMs any more than they already are. The article doent imply map reduce other than taking about using multiple computers.
They don’t talk about how the models are run in the article. But I know a tiny bit about how they’re run. LLMs require very simple and consistent math computations on extremely large matrixes of numbers. The bottleneck is almost always data transfer, not compute. Basically, every LLM deployment tool is already tries to use as much parallelism as possible while reducing data transfer as much as possible.
The article talks about gpt-oss120, so were aren’t talking about novel approaches to how the data is laid out or how the models are used. We’re talking about tranformer models and how they’re huge and require a lot of data transfer. So, the preference is try to keep your model on the fastest-transfer part of your machine. On consumer hardware, which was the key point of the article, you are best off keeping your model in your GPU’s memory. If you can’t, you’ll run into bottlenecks with PCIe, RAM and network transfer speed. But consumers don’t have GPUs with 63+ GB of VRAM, which is how big GPT-OSS 120b is, so they MUST contend with these speed bottlenecks. This article doesn’t address that. That’s what I’m talking about.