Comment on Is anyone self-hosting their own AI?
benchsmoke@mastodon.social 2 weeks ago @aurion The hybrid split sounds right. For us, a local 9B on CPU took about 40 seconds just to read a 2.7k-token agent prompt, then generated at roughly 3.5 to 6.7 tokens per second, so prompt size matters more than people expect. Is most of the wait on reading the prompt or on generation for the Hermes agent?
aurion@piefed.aurionblack.com 1 week ago
Hey there! Sorry for the delay, I didn’t forget about you.
I didn’t know how to answer your questions, but they are very good questions that I wanted to know myself.
I’ve been working on a testing plan to figure this out. But life is got super busy and it’s been hard to focus on it.
Hopefully I’ll complete it before the weekend is over and will share the results.