From their linked study:
“Filling this utilization gap requires us to better saturate each GPU by enabling it to serve requests from multiple models. As such, we aim to conduct effective GPU pooling. By sharing a GPU between as many models as possible with- out violating the service-level objective (SLO), GPU pooling promises great reductions in operational expenses (OPEX) for concurrent LLM serving.”
The “saturate each gpu” part seems to support your idea.
Rentlar@lemmy.ca 22 hours ago
I do expect operational savings from this optimization, but my guesstimate would be a 2-5x savings rather than the reported 9x savings when looked at over a fixed time period.
BCOVertigo@lemmy.world 20 hours ago
I mean if short term GPU supply is the bottleneck it will help that, but if it cooks the GPUs even faster and doesn’t result in less overall compute being done it just lets smaller players with less hardware compete with hyperscalers and may not actually be savings in the long term. The unit economics being negative are still the giant elephant in the room preparing to trample them.