Comment on I'm creating a curated search engine for web developers. Asking for a feedback
sznowicki@lemmy.world 10 months agoIt’s still in MVP, work in progress, hence the index is not “full”.
For me “web development” is everything that we might need for well, web. Servers, mongo docs all goes into the index (I’m adding it every day basically but also it takes some time to index stuff and I observe how this whole thing works as index grows).
ASP.NET goes into the index of course. If your website has dev resources and blog posts that would go into it as well. Recently one person suggested tons of Haskell blogs and they are being indexed as we speak.
I have also a different problem, dev.to has a lot of good resources but also tons of SEO spam and low quality content. It’s also freaking huge and while it was for some time in the index I had to remove it and think about it some more.
Where would you draw lines on mixed c content or technologies
For now the line is: does this website have anything that web devs would need? Yes? Then it might get in.
If it’s a blog about locomotive CPU programming then maybe not. Although mostly due to infrastructure costs. Indexing cost in the end but having some non related stuff in the index should not hurt the results.
All of what I wrote is the state for today. I’m changing my mind often as it’s still in “having fun” state.
Kissaki@feddit.de 10 months ago
Yeah, a public platform is unlikely to provide consistent content. If curation is not an explicit goal, I would not include them for the reasons you mentioned.
If indexing could happen not on domain but with more granular filters - URL base paths - that may be viable. Indexing specific authors on devto.
sznowicki@lemmy.world 10 months ago
Good idea. I had this thought once to do some narrow indexing of websites, e.g. stack overflow is a big issue, indexing all of this is crazy, picking up some specific tags on the other hand feels like tons of work. In the end I adjust the whole project as it grows with hope that after every tuning it gets better.
As long as I have fun with it I’ll continue :D
Kissaki@feddit.de 10 months ago
Of course - cutting scope is a good call to keep it manageable and fun, and not end up with creep and what you wanted to evade in the first place. :)