Comment on Is anyone self-hosting their own AI?
xylogx@lemmy.world 1 week ago
With just my NVidia Geforce 3070 card with 8GB of VRAM I can get 40 tokens per second on smallish models such as llama3.1 or qwen 3.8
ollama list
NAME ID SIZE MODIFIED
qwen3:8b 500a1f067a9f 5.2 GB 5 months ago
llama3.1:8b 46e0c10c039e 4.9 GB 5 months ago
Schilling2304@thelemmy.club 1 week ago
What game servers ? Have you looked into qwen3.5 9b ? Gemma e4b ? There is plenty of new small models after yours. There is r/lowendlocalai, a community of people with low computing resources but it is in reddit.
MiniCPM5 may be good.