Comment on Self-Hosted AI is pretty darn cool

dataprolet@lemmy.dbzer0.com ⁨3⁩ ⁨months⁩ ago

• MangoPenguin@lemmy.blahaj.zone ⁨3⁩ ⁨months⁩ ago

  Not really, it uses some GPU power when it’s actively generating a response, but otherwise it just sits idle.

  source
• Toribor@corndog.social ⁨3⁩ ⁨months⁩ ago

  I’ve been testing Ollama in Docker/WSL with the idea that if I like it I’ll eventually move my GPU into my home server and get an upgrade for gaming. When you run a model it has to load the whole thing into VRAM. I use the 8gb models so it takes 20-40 seconds to load the model and then each response is really fast after that and the GPU hit is pretty small. After I think five minutes by default it will unload the model to free up VRAM.

  Basically this means that you either need to wait a bit for the model to warm up or you need to extend that timeout so that it stays warm longer (or just get used to waiting on it). That also means that I cannot really use my GPU for anything else while the LLM is loaded.

  I haven’t tracked power usage, but besides the VRAM requirements it doesn’t seem too intensive on resources, but maybe I just haven’t done anything complex enough yet.

  source
• Swedneck@discuss.tchncs.de ⁨2⁩ ⁨months⁩ ago

  you hear that said about AI because companies are desperately throwing more and more resources at it to get 0.3% better results, and people are collectively running an insane amount of prompts all the time.

  but on a personal level it’s not really any different from any other computations, people render videos all the time and no one complains about the resource usage from that, because companies aren’t trying to sell bloated video rendering services to gardening firms.

  source