Is there any actual standalone AI software?

Submitted ⁨⁨3⁩ ⁨months⁩ ago⁩ by ⁨Ad4mWayn3@lemmy.world⁩ to ⁨nostupidquestions@lemmy.world⁩

Is there any computer program with AI capabilities (the generative ones seen in ChatGPT; onlineText-to-Picture generators, etc.) that is actually standalone? i.e. able to run in a fully offline environment.

As far as I understand, the most popular AI technology rn consists of a bunch of matrix algebra, convolutions and parallel processing of many low-precision floating-point numbers, which works because statistics and absurdly huge datasets. So if any such program existed, how would it even have a reasonable storage size if it needs the dataset?

source

Comments

Sort:hotnew top

Ziggurat@sh.itjust.works ⁨3⁩ ⁨months⁩ ago
There is tons of “standalone” software that you can run on your own PC

For Text generation, the easiest way is to get GPT4All package which allows you to run text generation model in CPU on your own PC

For image generation, you can try to get Easy difusion package which is an easy to use stable diffusion package, then if you like-it, time to try the “comfyUI”

You can check !localllama@sh.itjust.works and !imageai@sh.itjust.works for some more information
source
- deranger@sh.itjust.works ⁨3⁩ ⁨months⁩ ago
  I’ve wanted to try these out for shits and giggles - what would I expect with a 3090, is it going to take a long time to make some shitposts?
  
  source
  - chicken@lemmy.dbzer0.com ⁨3⁩ ⁨months⁩ ago
    3090s are ideal because the most important factor is vram, and those are at the top of the plateau for vram until you get into absurdly expensive server hardware. Expect around 3 seconds for generating a 512x512 image or 4 words per second generating text at around GPT 3.5 quality.
    
    source
  - Ziggurat@sh.itjust.works ⁨3⁩ ⁨months⁩ ago
    With SD 1.5 my old GTX 970was doing fine (30 second per image) I upgraded to a Radeon 7060 and with SDXL get like 4 images in these 30 seconds (but sometimes crash my Pac when loading a model)
    
    source
  - tyler@programming.dev ⁨3⁩ ⁨months⁩ ago
    I did a bunch of image generation on my 3080 and it felt extremely fast. Enough that I was able to set it up as a shared node in one of those image generation nets and it outperformed most other people in the net.
    
    source
    -> View More Comments
KoboldCoterie@pawb.social ⁨3⁩ ⁨months⁩ ago
Stable Diffusion (AI image generation) runs fully locally. The models (the datasets you’re referring to) are generally around 3GB in size. It’s more about the processing power needed for it to run (it’s very GPU-intensive) than the storage size on disk.

source
Wahots@pawb.social ⁨3⁩ ⁨months⁩ ago
lmstudio.ai

You can load up your own datasets, has some of its own, too. Most of these are pretty good, but run on synthetic data. Storing and processing something the size of chatgpt would bankrupt most people.

This program can use significant amounts of computer resources if you let her eat. I recommend closing other programs and games.

source
ichbinjasokreativ@lemmy.world ⁨3⁩ ⁨months⁩ ago
Stable diffusion and ollama for image and text generation locally. Super easy to do on linux and support gpu acceleration out of the box

source
CaptDust@sh.itjust.works ⁨3⁩ ⁨months⁩ ago
Local LLMs can be compressed to fit on consumer hardware. Model formats like GUFF and Exl2 can be loaded up with a offline hosted API like KobaldCPP or Oobabooga. These formats lose resolution and become “dumber” but it’s good enough for many uses.

source
- FaceDeer@fedia.io ⁨3⁩ ⁨months⁩ ago
  Though bear in mind that parameter count alone is not the only measure of a model's quality. There's been a lot of work done over the past year or two on getting better results from the same or smaller parameter counts, lots of discoveries have been made on how to train better and run inferencing better. The old ChatGPT3 from back at the dawn of all this was really big and was trained on a huge number of tokens but nowadays the small downloadable models fine-tuned by hobbyists would compete with it handily.
  
  source
  - CaptDust@sh.itjust.works ⁨3⁩ ⁨months⁩ ago
    Agreed, especially true with Llama3 their 7b model is extremely competitive.
    
    source
    -> View More Comments
Rhaedas@fedia.io ⁨3⁩ ⁨months⁩ ago
The AI, image, and audio models that can run on a typical PC have all been broken down from originally larger models. How this is done affects what the models can do and the quality, but the open source community has come a long way in making impressive stuff. First question is more hardware - do you have an Nvidia GPU that can support these types of generations? They can be done through CPU alone, but it's painfully much slower.

If so, then I would highly recommend looking into Ollama for running AI models (using WSL if you're using Windows) and ComfyUI for graphical generation. Don't let the workflow of complicated ComfyUI scare you, starting from the basics with plenty of Youtube help out there it will make sense. As for TTS, there's a lot of constant "new stuff" out there, but for actual local processing in "real time" (still takes a bit) I have yet to find anything to replace my Coqui TTS copy with Jenny as the model voice. It may take some digging and work to get that together, it's older and not supported anymore.

source
- hendrik@palaver.p3x.de ⁨3⁩ ⁨months⁩ ago
  I don't think they break them down. For most models the math requires to start at the beginning and train each model individually from ground up.
  
  But sure, a smaller model generally isn't as capable as a bigger one. And you can't train them indefinitly. So for a model series you'll maybe use the same dataset but feed more into the super big variant and not so much into the tiny one.
  
  source
  - Rhaedas@fedia.io ⁨3⁩ ⁨months⁩ ago
    The breaking down I mentioned is the quantization that forms a smaller model from the larger one. I didn't want to get technical because I don't understand the math details myself past how to use them. :)
    
    source
    -> View More Comments
MacNCheezus@lemmy.today ⁨3⁩ ⁨months⁩ ago
For LLMs, the already mentioned LM Studio does a good job as far as beginner friendliness goes.

For text-to-image, I like Fooocus, which is a custom Stable Diffusion setup with automatic prompt enhancement, which can comfortably compete with Midjourney.

Here’s a setup guide for first time users. There’s also an online version to try it out.

source
- iturnedintoanewt@lemm.ee ⁨3⁩ ⁨months⁩ ago
  Just wanted to thank you, as I hadn’t had any luck running any other SD software on my AMD setup with Nobara. But after a couple of fixes to get rocm running, this one runs, and runs pretty fast. Thanks!
  
  source
bjoern_tantau@swg-empire.de ⁨3⁩ ⁨months⁩ ago
Krita has an AI plugin that’s pretty painless to setup if you’ve got an nVidia card. AMD has to be done manually or you can fall back to slow CPU generation. It uses ComfyUI in the background.

source
Grimy@lemmy.world ⁨3⁩ ⁨months⁩ ago
You need a GPU for any kind of performance.

For text I suggest: Ollama backend - command line interface, very easy to download models with one line of code. Supports most models and you can talk with the model inside the terminal so it’s stand alone OpenWebUI - easy install with docker and is meant to work easily with ollama. Comes with web search features and uploading pdfs. A bunch of different community tools and modules are available.

For img I suggest either: Automatic1111 - Traditional UI using gradio. Lots of extras you can download through the UI to do different things. ComfyUI - Node based UI, a bit more complicated but more powerful than automatic1111

For models, you can go on civitai and just download whatever you need and drop it into their respective folders for both auto and comfy.

For text, there’s also LMStudio which is very user friendly. It is closed source and much slower than ollama from my experience though. I have a 4060 in my laptop (8gb VRAM) and I’m getting an image every 2 secs about using stable diffusion 1.5 models and text speed is on par with chatgpt with the smaller 8b-9b model. For text I suggest gemma2 which is probably the best small model out right now.

source
JackGreenEarth@lemm.ee ⁨3⁩ ⁨months⁩ ago
I use Krita with the AI Diffusion plugin for Image Generation, which is working great, and Jan for text Generation, using the Llama 3 8B Q4 model. I have a NVIDIA GTX 1660 Ti with 6GB of VRAM and both are reasonably fast.

source
Starbuck@lemmy.world ⁨3⁩ ⁨months⁩ ago
If you are into development, the setup I use is ollama running codegemma:7b along with the Continue.dev plugin for vscode.

source
Sabata11792@ani.social ⁨3⁩ ⁨months⁩ ago
If you have a good GPU, you should be able to run a model without issue. The big ones are technically usable with tweaking but so slow enough to be useless on normal hardware. A small model may be 4-8 gb, but a larger one could be 100+gb. You don’t need the training data(if its even public) to run them, only if your building or retraining the model. There’s a crap ton of different software to run AI on.

To get started assuming you got a beefy PC, you need a model and software to interact with it. I started with mistral7b and textGenWebUi and been trying out different software and models. Text gen has the basics to load and chat to a model and is a good starting point.

Model-https://mistral.ai/technology/#models Software-https://github.com/…/text-generation-web…

For Images, you can choose models at based on what the sample images look like, they tend to be specialized for certain styles or content. You can add LORAs to further change how the output looks(think specific characters or poses). It’s very much trial and error getting good images.

Models-https://civitai.com (potentially NSFW) Software-https://github.com/vladmandic/automatic

There’s more models and software out their than I can keep track of, so if something is crap you should be able to find an alternative. Youtube guides are you friend.

source
Evotech@lemmy.world ⁨3⁩ ⁨months⁩ ago
ComfyUI is the best for image AI

source
random72guy@lemmy.world ⁨3⁩ ⁨months⁩ ago
For LLMs, I’ve had really good results running Llama 3 in the Open Web UI docker container on a Nvidia Titan X (12GB VRAM).

For image generation tho, I agree more VRAM is better, but the algorithms still struggle with large image dimensions, ao you wind up needing to start small and iterarively upscale, which afaik works ok on weaker GPUs, but will gake problems.

source
bobburger@fedia.io ⁨3⁩ ⁨months⁩ ago
Llamafile is a pretty good option for 100% local LLMs. The smaller models are pretty good for basic applications. They run at a reasonable speed on my Samsung laptop and really fast on my M2 macbook.

source