Comment on Is there any actual standalone AI software?
CaptDust@sh.itjust.works 3 months ago
Local LLMs can be compressed to fit on consumer hardware. Model formats like GUFF and Exl2 can be loaded up with a offline hosted API like KobaldCPP or Oobabooga. These formats lose resolution and become “dumber” but it’s good enough for many uses.
FaceDeer@fedia.io 3 months ago
Though bear in mind that parameter count alone is not the only measure of a model's quality. There's been a lot of work done over the past year or two on getting better results from the same or smaller parameter counts, lots of discoveries have been made on how to train better and run inferencing better. The old ChatGPT3 from back at the dawn of all this was really big and was trained on a huge number of tokens but nowadays the small downloadable models fine-tuned by hobbyists would compete with it handily.
CaptDust@sh.itjust.works 3 months ago
Agreed, especially true with Llama3 their 7b model is extremely competitive.
FaceDeer@fedia.io 3 months ago
Makes it all the more amusing how OpenAI staff were fretting about how GPT-2 was "too dangerous to release" back in the day. Nowadays that class of LLM is a mere toy.
tyler@programming.dev 3 months ago
They were fretting about it until their morals went out the door for money.
lung@lemmy.world 3 months ago
Whenever these corps talk up the danger of AI, all I think is “nice marketing dept bro”