Comment on What exactly is a self-hosted small LLM actually good for (<= 3B)
swelter_spark@reddthat.com 1 week ago
7b is the smallest I’ve found useful. I’d try a smaller quant before going lower, if I had super small vram.
Comment on What exactly is a self-hosted small LLM actually good for (<= 3B)
swelter_spark@reddthat.com 1 week ago
7b is the smallest I’ve found useful. I’d try a smaller quant before going lower, if I had super small vram.