Comment on Have you tried LocalGPT PrivateGPT or other similar alternatives to ChatGPT?
Assman@sh.itjust.works 10 months ago
There’s a project called Tabby that your can host as a server on a machine that has a GPU, and has a VSCode extension that connects to the server.
The default model is called starcoder, and it’s the small version, 1B parameters. The downside is that it’s not super smart (but still an improvement over built in tools), but since it’s such a small model, I’m getting sub-second processing times.
You should make sure you are running a model that fits in your vram, for me it runs faster than any online LLM I’ve tried.
I’ve found it’s pretty good for translating between steps so to speak.
Converted some bash to python relatively quickly by giving it snippets and fixing errors as it made them.
I also had success generating an ansible playbook based on my own previously written install instructions for SillyTavern and llama.cpp.
I could do both of those tasks myself, but thar would be more difficult than having a mostly correct translation and fixing some errors.
SoleInvictus@lemmy.world 10 months ago
It’s good for me because I’m piss poor at programming. In my defense, I’m not a programmer or even programmer adjacent. I do see how it wouldn’t be useful to a pro. It also has occasionally given me garbage advice that an expert would spot right away while I had to figure it out was ‘hallucinating’ again.
bogo@sh.itjust.works 10 months ago
I can absolutely see it getting useful for a pro. It’s already a better version of IDE templates. If you have to write boilerplate code this can already do that. It’s a huge time saver for the things you’d have to go look up to remember how to do and piece together yourself.
Example: today I wanted a quick way to serve my current working directory over HTTP so I could do some quick web work. I asked ChatGPT to write me a bash function I could stick in my profile to do this, and I told it to pick a random unused port. That would have taken me much longer had I went to lookup how to do that all. The only hint I gave it was to use the Python builtin module for serving http.