Not llamacpp?
Comment on Where to start with setting up my own server?
undrwater@lemmy.world 2 days ago
You can run an LLM, but it will be limited. Look into Open Vino Model Server.
I’d recommend a minimal install distro (Ubuntu server, Arch, Gentoo, etc…), then add the things you need from there.
Document every step. <— important
surewhynotlem@lemmy.world 2 days ago
undrwater@lemmy.world 2 days ago
OVMS + a model that was built for it is far more responsive than llamacpp.
Cyber@feddit.uk 2 days ago
@cerebralhawks@lemmy.dbzer0.com this is the very important bit:
Whatever you do will make sense for about a day, you’ll have a rough idea of what you did the following weekend, after that… all gone.
And, whatever you do won’t be right first time, so you will be editing those notes - all part of the journey
cerebralhawks@lemmy.dbzer0.com 1 day ago
Thanks. I took note the first time I read that, but I appreciate the emphasis.