Comment

Comment on Microsoft Bans Employees From Using DeepSeek App

brucethemoose@lemmy.world ⁨21⁩ ⁨hours⁩ ago

True, though there’s a big output difference between the 7B distil (or even 32B/70B) and the full model.

And Microsoft does host R1 already, heh. Again, this headline is a big nothingburger.

Also (random aside here), you should consider switching from ollama. They’re making some FOSS unfriendly moves, and depending on your hardware, better backends could host 14B models at longer context, and similar or better speeds.

source

Sort:hotnew top

Sabata11792@ani.social ⁨21⁩ ⁨hours⁩ ago
What other back ends are good?

source
- brucethemoose@lemmy.world ⁨21⁩ ⁨hours⁩ ago
  Completely depends on your laptop hardware, but generally:
  
  TabbyAPI (exllamav2/exllamav3)
  
  ik_llama.cpp, and its openai server
  
  An MLX host with one of the new distillation quantizations
  
  Text-gen-web-ui (slow, but supports a lot of samplers and some exotic quantizations well)
  
  SGLang (extremely fast for parallel calls if thats what you want).
  
  source