The ecological costs don’t need to be very high. We host our own LLM models at my company on a Mac Mini, which doesn’t use a ton of power and works pretty well.
FWIW I did try few LLMs locally too (cf my notes on the topic …benetou.fr/…/SelfHostingArtificialIntelligence ) but AFAIK that is only the top of the iceberg, that LLM has been trained before and that’s a significant part of the cost.
sugar_in_your_tea@sh.itjust.works 6 days ago
The ecological costs don’t need to be very high. We host our own LLM models at my company on a Mac Mini, which doesn’t use a ton of power and works pretty well.
utopiah@lemmy.world 6 days ago
FWIW I did try few LLMs locally too (cf my notes on the topic …benetou.fr/…/SelfHostingArtificialIntelligence ) but AFAIK that is only the top of the iceberg, that LLM has been trained before and that’s a significant part of the cost.