Comment on Smaug-72B-v0.1: The New Open-Source LLM Roaring to the Top of the Leaderboard
kakes@sh.itjust.works 10 months agoAfaik you can substitute VRAM with RAM at the cost of speed. Not exactly sure how that speed loss correlates to the sheer size of these models, though. I have to imagine it would run insanely slow on a CPU.
Infiltrated_ad8271@kbin.social 10 months ago
I tested it with a 16GB model and barely got 1 token per second. I don't want to imagine how long it would take if I used swap instead.