Comment on What exactly is a self-hosted small LLM actually good for (<= 3B)
herseycokguzelolacak@lemmy.ml 2 days ago
for coding tasks you need web search and RAG. It’s not the size of the model that matters, since even the largest models find solutions online.
catty@lemmy.world 2 days ago
Any suggestions for solutions?
herseycokguzelolacak@lemmy.ml 1 day ago
Not on top of my head, but there must be something. llama.cpp and vllm have basically solved the inference problem for LLMs. What you need is a RAG solution on top that also combines it with web search.