it’s distilled so it’s going to be smaller than any non distilled of the same quality
Comment on DeepSeek's distilled new R1 AI model can run on a single GPU | TechCrunch
blarth@thelemmy.club 4 weeks ago
7b trash model?
knighthawk0811@lemmy.world 4 weeks ago
LainTrain@lemmy.dbzer0.com 4 weeks ago
I’m genuinely curious what you do that a 7b model is “trash” to you? Like yeah sure a gippity now tends to beat out a mistral 7b but I’m pretty happy with my mistral most of the time if I ever even need ai at all.
TropicalDingdong@lemmy.world 4 weeks ago
Yeah idk. I did some work with deepseek early on. I wasn’t impressed.
HOWEVER…
Some other things they’ve developed like deepsite, holy shit impressive.
double_quack@lemm.ee 4 weeks ago
Save me the search, please. What’s deepsite?
TropicalDingdong@lemmy.world 4 weeks ago
Above is what I can do with deepsite by pasting in the first page of your lemmy profile and the prompt:
“This is double_quack, a lemmy user on Lemmy, a new social media platform. Create a cool profile page in a style that they’ll like based on the front page of their lemmy account (pasted in a ctrl + a, ctrl + c, ctrl + v of your profile).”
double_quack@lemm.ee 4 weeks ago
Excuse me… what? Ok, that’s something…
vhstape@lemmy.sdf.org 4 weeks ago
Most models come in 1B, 7-8B, 12-14B, and 27+B parameter variants. According to the docs, they benchmarked the 8B model using an NVIDIA H20 (96 GB VRAM) and got between 144-1198 tokens/sec. Most consumer GPUs probably aren’t going to be able to keep up with
avidamoeba@lemmy.ca 4 weeks ago
It proved sqrt(2) irrational with 40tps on a 3090 here. The 32b R1 did it with 32tps but it thought a lot longer.
vhstape@lemmy.sdf.org 4 weeks ago
On my Mac mini running LM Studio, it managed 1702 tokens at 17.19 tok/sec and thought for 1 minute
brucethemoose@lemmy.world 4 weeks ago
Depends on the quantization.
7B is small enough to run it in FP8 or a Marlin quant with SGLang/VLLM/TensorRT, so you can probably get very close to the H20 on a 3090 or 4090.