Comment on What exactly is a self-hosted small LLM actually good for (<= 3B)

herseycokguzelolacak@lemmy.ml ⁨4⁩ ⁨weeks⁩ ago

• catty@lemmy.world ⁨4⁩ ⁨weeks⁩ ago

  Any suggestions for solutions?

  source

  • wise_pancake@lemmy.ca ⁨3⁩ ⁨weeks⁩ ago

    Open webui lets you install a ton of different search providers out of the box, but you do need sn API key for most and I haven’t vetted them

    I’m trying to get Kagi to work with Phi4 and not having success.

    source

    • catty@lemmy.world ⁨3⁩ ⁨weeks⁩ ago

      Thanks, when I get some time soon, I’ll have another look at it and cherry ai with a local install of ollama

      source
  • herseycokguzelolacak@lemmy.ml ⁨4⁩ ⁨weeks⁩ ago

    Not on top of my head, but there must be something. llama.cpp and vllm have basically solved the inference problem for LLMs. What you need is a RAG solution on top that also combines it with web search.

    source