Comment on What type of computer setup would one need to run ai locally?
ShellMonkey@piefed.socdojo.com 2 weeks ago
I was using a Nvidia 3060 for a while, then had 2 in one box, then switched to a 3090.
The amount of vram is a big factor for decent performance. Getting it to not sound like a predictably repetitive bot though is a whole separate thing that is still kind of elusive.
surewhynotlem@lemmy.world 1 week ago
Does multiple GPU help? I could get a cheap 970 to toss in my rig
ShellMonkey@piefed.socdojo.com 1 week ago
My go to for messing with chat bots is Kobold that’ll let you split the work between multiple GPUs. I get the impression the actual processing is only done on one but it lets you load larger models with the extra memory.