Comment on Where to start with setting up my own server?
frongt@lemmy.zip 5 days agoYou can run stuff on a little GPU, IGPU, or even just a CPU. You’ll be limited by RAM/VRAM, and obviously it’ll be slower, but it’ll work.
Comment on Where to start with setting up my own server?
frongt@lemmy.zip 5 days agoYou can run stuff on a little GPU, IGPU, or even just a CPU. You’ll be limited by RAM/VRAM, and obviously it’ll be slower, but it’ll work.
I have a server that I wanted to do this with, and cpu-only interference was several minutes for pretty simple tasks (all text, couple of sentences). I shelled out the cash for a 5060 Ti 16GB earlier this summer, and even with a pcie 3 x16 slot running at x4, I’m down to 5-30 second responses and rapid output speeds, even with large conversations. But the initial prompt/loading is still a couple of minutes by itself, so I leave the model in vram always.