Comment on Where to start with setting up my own server?

breadsmasher@lemmy.world ⁨6⁩ ⁨days⁩ ago

• BartyDeCanter@piefed.social ⁨5⁩ ⁨days⁩ ago

  Or you can go straight to the source and use the Trash Guides

  original
• frongt@lemmy.zip ⁨6⁩ ⁨days⁩ ago

  You can run stuff on a little GPU, IGPU, or even just a CPU. You’ll be limited by RAM/VRAM, and obviously it’ll be slower, but it’ll work.

  original

  • wreckedcarzz@lemmy.world ⁨5⁩ ⁨days⁩ ago

    I have a server that I wanted to do this with, and cpu-only interference was several minutes for pretty simple tasks (all text, couple of sentences). I shelled out the cash for a 5060 Ti 16GB earlier this summer, and even with a pcie 3 x16 slot running at x4, I’m down to 5-30 second responses and rapid output speeds, even with large conversations. But the initial prompt/loading is still a couple of minutes by itself, so I leave the model in vram always.

    original