Comment on Smaug-72B-v0.1: The New Open-Source LLM Roaring to the Top of the Leaderboard
kakes@sh.itjust.works 7 months agoAfaik you can substitute VRAM with RAM at the cost of speed. Not exactly sure how that speed loss correlates to the sheer size of these models, though. I have to imagine it would run insanely slow on a CPU.
Infiltrated_ad8271@kbin.social 7 months ago
I tested it with a 16GB model and barely got 1 token per second. I don't want to imagine how long it would take if I used swap instead.