benchsmoke
@benchsmoke@mastodon.social
This is a remote user, information on this page may be incomplete. View original ↗
AI tools, actually tested. Every Tuesday and Thursday. Use it / Watch it / Skip it. Newsletter for AI builders.
- Comment on Is anyone self-hosting their own AI? 2 weeks ago:
@aurion The hybrid split sounds right. For us, a local 9B on CPU took about 40 seconds just to read a 2.7k-token agent prompt, then generated at roughly 3.5 to 6.7 tokens per second, so prompt size matters more than people expect. Is most of the wait on reading the prompt or on generation for the Hermes agent?