Comment on I'm creating a curated search engine for web developers. Asking for a feedback
Kissaki@feddit.de 6 months agoI have also a different problem, dev.to has a lot of good resources but also tons of SEO spam and low quality content. It’s also freaking huge and while it was for some time in the index I had to remove it and think about it some more.
Yeah, a public platform is unlikely to provide consistent content. If curation is not an explicit goal, I would not include them for the reasons you mentioned.
If indexing could happen not on domain but with more granular filters - URL base paths - that may be viable. Indexing specific authors on devto.
sznowicki@lemmy.world 6 months ago
Good idea. I had this thought once to do some narrow indexing of websites, e.g. stack overflow is a big issue, indexing all of this is crazy, picking up some specific tags on the other hand feels like tons of work. In the end I adjust the whole project as it grows with hope that after every tuning it gets better.
As long as I have fun with it I’ll continue :D
Kissaki@feddit.de 6 months ago
Of course - cutting scope is a good call to keep it manageable and fun, and not end up with creep and what you wanted to evade in the first place. :)