Comment on Please suggest some good self-hostable RAG for my LLM.
scrubbles@poptalk.scrubbles.tech 4 months ago
I’m not 100% what you’re asking for, but I use text-generation-webui for all of my local generation needs.
Comment on Please suggest some good self-hostable RAG for my LLM.
scrubbles@poptalk.scrubbles.tech 4 months ago
I’m not 100% what you’re asking for, but I use text-generation-webui for all of my local generation needs.
brucethemoose@lemmy.world 4 months ago
Text-generation-webui is cool, but also kinda crufty. Honestly a lot of the stuff is holdovers from what’s now ancient history in LLM land, and it has (for me) major performance issues at longer context.
scrubbles@poptalk.scrubbles.tech 4 months ago
Anything better you know of? Most of my usage now with it is through its api
brucethemoose@lemmy.world 4 months ago
Uh, depends on your hardware and model, but probably TabbyAPI?
scrubbles@poptalk.scrubbles.tech 4 months ago
Neat! I’ll check it out!