Comment on I'm creating a curated search engine for web developers. Asking for a feedback
sznowicki@lemmy.world 11 months ago- SO and Reddit are on the TODO list. It even had SO (in the bottom indeed) once but not via crawling, via SO Search API. It has very poor quality results and was super slow so I had to remove it while thinking of a better solution. Crawling entire SO might be little too much of this project at this state tho but if I have enough courage and hours at night I might parse that 20GB stack overflow archive dump and try doing something useful with it.
Same for Reddit but here I have mixed feelings about it in general and hope it’s going to die soon being replaced by amazing Lemmy communities.
I also used to type some question and end with “reddit” in Google to get good quality content, but here with kukei the experiment is whether blogosphere can replace it properly when index is promoting it.
- Why blogs?
This is my main thing. To promote good quality blogs that I tried to follow via RSS but somehow never did. Having them all indexed (and more, some Mastodon community gave me amazing links to index) makes me actually visit them often.
For the “SEO cancer” that where curation comes into play. Before crawling I check unknown blogs to me and decide whether something goes in or not.
DrakeRichards@lemmy.world 11 months ago
That makes sense. I really like that the documentation is right at the top; many times all I want to do is find the right page in the official docs. You might want to look at how results are prioritized though: right now when I search for something simple like “how to center a div”, that result from Mozilla’s docs is included but it’s hidden as the second or third result. I would expect the page that’s explicitly about centering a div to be the top result, followed by the docs page for the element itself and maybe pages for flex or grid or something. That’s a really simple example, so maybe it’s not the target of this project, but I would still hope that simple topics are covered just as well as complex ones.