Comment on What's up, selfhosters? It's self hosting Sunday!
Jakeroxs@sh.itjust.works 3 weeks agoInteresting, I mainly have used text generation webui which has a search support plug in, kinda nifty to use my searxng instance for it. It’s a bit finicky though.
Another thing to keep in mind then (apologies if this is just repeating info you already know), you’d also want to keep in mind your total potential context size in relation to the model size, since both take up VRAM.
Bluefruit@lemmy.world 3 weeks ago
Yea I’m aware but I appreciate the insight :) so far my local ai experience has been lack luster so I’m hoping
If you haven’t tried using RAG in some form, I would recommend giving it a go. Its pretty cool stuff, helps make models answer more accurately based on the documentation you give them though in my case, ive had limited success. Tbh, chatgpt has become my last resort when I just wanna get something done but I don’t like using it due to the privacy concerns, not to mention the ethical issues I have with ai training in general from big tech.
How is searxng BTW? Would you say its good to host or do you use a normal search engine more often? Or do you just use it for the AI search plugin?
Ive actually been thinking about using it rather than duckduckgo but was also hopeful the search index they are working on would be enough to satisfy my needs, or that a self hosted AI enabled search engine would work well enough when I need it.
Jakeroxs@sh.itjust.works 3 weeks ago
I’ve completely replaced my searching with searxng, it is a little slower and ofc if I have an outage or something at home I have to go back to a different search temporarily but overall I like it a lot.
It was one of the first things I set up last year with my homelab because I am attempting to degoogle a fair amount, the Ai search stuff was just a fun test
Bluefruit@lemmy.world 3 weeks ago
Thats rad, thanks for the info. I may follow suit, been trying to degoogle myself lately.
Jakeroxs@sh.itjust.works 3 weeks ago
For sure, good luck and have fun :D