Tokasaurus: An LLM Inference Engine for High-Throughput Workloads

Submitted ⁨⁨1⁩ ⁨day⁩ ago⁩ by ⁨cm0002@lemmy.world⁩ to ⁨technology@lemmy.zip⁩

https://scalingintelligence.stanford.edu/blogs/tokasaurus/