Thx in advice.
“Bang for Buck”
Good luck. I would wait for the AI phase to crash
What's the bang for the buck go to for AI image generation and LLM models?
Submitted 3 months ago by TheBigBrother@lemmy.world to selfhosted@lemmy.world
Comments
possiblylinux127@lemmy.zip 3 months ago
maxwellfire@lemmy.world 3 months ago
I feel like this really depends on what hardware you have access too. What are you interested in doing?How long are you willing to wait for it to generate, and how good do you want it to be?
You can pull off like 0.5 word per second of one of the mistral models on the CPU with 32GB of RAM. The stabediffusion image models work okay with like 8-16GB of vram.
istanbullu@lemmy.ml 3 months ago
Automatic1111
kata1yst@sh.itjust.works 3 months ago
KobaldCPP will probably be the easiest way out of the box that has both image generation and LLMs.
I personally use vllm and HuggingChat, mostly because of vllm’s efficiency and speed increase.
DarkThoughts@fedia.io 3 months ago
It is probably dead but Easy Diffusion is imo the easiest for image generation.
KoboldCPP can be a bit weird here and there but was the first thing that worked for me for local text gen + gpu support.
hendrik@palaver.p3x.de 3 months ago
Buy the cheapest graphics card with 16 or 24GB of VRAM. In the past people bought used NVidia 3090 cards. You can also buy a GPU from AMD, they're cheaper but ROCm is a bit more difficult to work with. Or if you own a MacBook or any Apple device with a M2 or M3, use that. And hopefully you paid for enough RAM in it.
thirdBreakfast@lemmy.world 3 months ago
An M1 MacBook with 16GB cheerfully runs llama3:8b outputting about 5 words a second. FA second hand MacBook like that probably costs half to a third of a secondhand RTX3090.
It must suck to be a bargain hunting gamer. First bitcoin, and now AI.
Damage@feddit.it 3 months ago
Patient gamers at least have the steam deck option now
Fisch@discuss.tchncs.de 3 months ago
I actually use an AMD card for running image generation and LLMs on my PC on Linux. It’s actually not hard to set up.
fhein@lemmy.world 3 months ago
For LLMs it entirely depends on what size models you want to use and how fast you want it to run. Since there’s diminishing returns to increasing model sizes, i.e. a 14B model isn’t twice as good as a 7B model, the best bang for the buck will be achieved with the smallest model you think has acceptable quality. And if you think generation speeds of around 1 token/second are acceptable, you’ll probably get more value for money using partial offloading.
If your answer is “I don’t know what models I want to run” then a second-hand RTX3090 is probably your best bet.