Comment on [AIT] what is your selfhosted local LLM stack?

mlg@lemmy.world ⁨1⁩ ⁨week⁩ ago

vLLM to host and openwebui for a frontend that I really only use to demo models to friends.

harness

Opencode running with full access inside a rootless podman container with only the directories I want mounted (usually a single project, sometimes read only).

So far though, I’ve only used it to document and report on project repos because Qwen is not that good at actually coding.

Also useful for parsing pita data dump type files that I’m too lazy to construct multiline awk commands for.

original
Sort:hotnewtop