I tried running yacy for a while but it just ran for a bit less than a day then ran out of memory and crashed, over and over. Tried to figure out the problem, but it’s niche enough that I couldn’t get anywhere googling the issue.
Comment on Any “small-web” search engines?
zutto@lemmy.fedi.zutto.fi 2 months ago
Teclis - Includes search results from Marginalia, free to use at the moment. This search index has been in the past closed down due to abuse.
Kagi, whose creation Teclis is, is a paid search engine (metasearch engine to be more precise) also incorporates these search results in their normal searches. I warmly recommend giving Kagi a try, it’s great, I’ve been enjoying it a lot.
–
Other options I can recommend; You could always try to host your own search engine if you have list of small-web sites in mind or don’t mind spending some effort collecting such list. I personally host Yacy [github link] (and Searxng to interface with yacy and several other self-hosted indexes/search engines such as kiwix wiki’s.). Indexing and crawling your own search results surprisingly is not resource heavy at all, and can be run on your personal machine in the background.
phanto@lemmy.ca 2 months ago
zutto@lemmy.fedi.zutto.fi 2 months ago
This is a bit off-topic, but did you try to increase the JVM limits inside Yacy’s administration panel?
Spoilering to hide wall of text related to this topic.
This setting located in
/Performance_p.html-page for example gives the java runtime more memory. Same page also has other settings related to ram, such as setting how much memory Yacy must leave unused for the system. (These settings exist so people who run Yacy on their personal machines can have guaranteed resources for more important stuff) Image Other things that would reduce memory usage is to limit the concurrency of the crawler for example. There’s quite a lot of tunable settings that can affect memory usage. Would recommend trying to hit up one of the Yacy forums is also good place to ask questions. The Matrix channel (and IRC) are a bit dead, but there are couple of people including myself there! Also, theres new docs written by the community, they might help as well! yacy.net/docs/ yacy.net/operation/performance/phanto@lemmy.ca 2 months ago
Yeah, I did try that. Basically, if I doubled the memory I allocated, I gave it half again longer before it crashed, but it still crashed, eventually.
It’s no big deal, this was last year, I may try again one day. Loving Searxng though!
Sl00k@programming.dev 2 months ago
Personally really been enjoying Kagi for the past year.