@aurion The hybrid split sounds right. For us, a local 9B on CPU took about 40 seconds just to read a 2.7k-token agent prompt, then generated at roughly 3.5 to 6.7 tokens per second, so prompt size matters more than people expect. Is most of the wait on reading the prompt or on generation for the Hermes agent?
Comment on Is anyone self-hosting their own AI?
aurion@piefed.aurionblack.com 3 weeks ago
Yes! I’ve got quite a few projects going right now. Probably too many of them to keep track
I am the only user, so it’s relatively easy to run multiple models for what ever task. Ollama is pretty good at balancing which models are working at what time to not bog down they system
I’m running gpt-oss 120 for a general purpose Hermes agent. Out of the box, it was ok. Not very snappy, but it was one of the best ones for orchestrating other agents.
Glm flash (4.something?) does pretty good as well.
I’ve found a hybrid of cloud and local is the best solution. Use local for smaller or more defined tasks. Use cloud for orchestrating, planning, and…fixing the local agents mistakes lol
I’ve got an LLM maintained Obsidian wiki that is written and edited by a few Ollama agents based on chats via my own self hosted matrix instance. I’ve taken it to a point where I’m pretty happy with the results. But I think if I had enough time and energy, I could really dial it in.
One of my favorite use cases for this is when I’m researching topics like UFOs/high strangeness/conspiracy theories. I have my very own LLM maintained Obsidian vault filled with curated notes and links to sources.
Literally, no one cares about this and they think it’s stupid and I’m stupid and I should feel bad for spending time on this or having the audacity of sharing it… they just don’t appreciate the amount of work and knowledge it took to get what I have set up.
I think it’s cool as fuck. I geek out every time that it works.
benchsmoke@mastodon.social 3 weeks ago aurion@piefed.aurionblack.com 2 weeks ago
Hey there! Sorry for the delay, I didn’t forget about you.
I didn’t know how to answer your questions, but they are very good questions that I wanted to know myself.
I’ve been working on a testing plan to figure this out. But life is got super busy and it’s been hard to focus on it.
Hopefully I’ll complete it before the weekend is over and will share the results.
zer0squar3d@lemmy.dbzer0.com 3 weeks ago
Do what makes you happy mate! Tell the naysayers to fk off.