I'm Starting A Search Engine For The Fediverse

Submitted ⁨⁨11⁩ ⁨months⁩ ago⁩ by ⁨lautan@lemmy.ca⁩ to ⁨fediverse@lemmy.world⁩

Hey everyone,

This isn’t an announcement, just wanted peoples thoughts on this.

I think everyone knows searching the fediverse can be better. Googling doesn’t work too well, etc. So I wanted to do my part and help out.

Indexing all posts, etc is quite a lot to handle, so I wanted to start small and just focus on video search. I’ve started indexing videos from Peertube and other video websites. (Even YouTube but this could be removed to just focus on independent sites)

I know Peertube has their own search engine for videos. I will be reaching out to them. Compared to my site I’m planning it’ll have other video sources and be easier to use.

So that leads to feedback from you guys.

What do you think about indexing videos posted on the fediverse and other independent platforms?
Are there similar services?
Am I just wasting my time?

source

Comments

Sort:hotnew top

gabe@literature.cafe ⁨11⁩ ⁨months⁩ ago
Well, please make sure it respects post privacy at least but also realize that on the microblogging side of the fediverse, they may not take kindly to this prospect at all. People who start these kinds of projects are often harassed or at least receive passive hostility.

source
- scrubbles@poptalk.scrubbles.tech ⁨11⁩ ⁨months⁩ ago
  I disagree. Post privacy sure, but the drivers is by definition public. Anything you put out there can be used for pretty much everything, the original rules of the internet apply. I’d be happy to see an easy opt out on the engine to remove yourself, but if everything is opt in it’ll never get off the ground.
  
  source
  - TimLovesTech@badatbeing.social ⁨11⁩ ⁨months⁩ ago
    As the fediverse is almost exclusively run by volunteers that are paying server bills and being admins, I could see some larger instances not taking kindly to this, especially depending on how much stress it would be putting on some already at capacity servers.
    
    source
    -> View More Comments
  - gabe@literature.cafe ⁨11⁩ ⁨months⁩ ago
    That’s not how the fediverse functions and approaching it that way is a problem waiting to happen. I’m stating so as a warning to be mindful of the culture of the way the fediverse itself functions. This is not Reddit, we share the fediverse with other software with different uses and features and we need to be mindful of that especially when building these kinds of tools. Making it opt out not only places a burden on smaller instances but presents a potential harassment risk for instances with vulnerable people on other fediverse platforms. As well, it is contrary to the entire way specific other activitypub instances operate. The fediverse is like a city we share with others, if Lemmy is not mindful of that city’s culture then people will promptly give them the boot.
    
    I’m not saying user by user opt in either, but instance by instance. Lemmy needs a tool of archiving especially. There is already cultural clashes I see occurring with the rest of the fediverse. Post like these of potential tools when it seems like the creator doesn’t know the messy history behind previous projects like them in the fediverse make me fearful of the clashes coming to fruition.
    
    source
    -> View More Comments
  - skullgiver@popplesburger.hilciferous.nl ⁨11⁩ ⁨months⁩ ago
    [deleted]
    source
    -> View More Comments
TimLovesTech@badatbeing.social ⁨11⁩ ⁨months⁩ ago
I found FediSearch, and also this post basically saying that a fediverse search engine would just be used as a tool by trolls.

source
- lautan@lemmy.ca ⁨11⁩ ⁨months⁩ ago
  That’s a good point. But those people can be banned? I guess Reddit handles this by moderation and archiving old posts.
  
  source
  - gabe@literature.cafe ⁨11⁩ ⁨months⁩ ago
    People will take the harassment off site especially if they are dedicated enough or use it to scrape for potential personal info to publicly release.
    
    source
    -> View More Comments
  - JustEnoughDucks@feddit.nl ⁨10⁩ ⁨months⁩ ago
    Yes, but moderation teams on the fediverse are very small, and by nature of it, can make hundreds of account of different servers all trailing that would need to be individually sought out and banned.
    
    It is a game of cat & 100 mice
    
    source
- ggsu7@futurology.today ⁨11⁩ ⁨months⁩ ago
  >muh trolls
  
  God shut up please. Why do you have to ruin something amazing like searching the entire fediverse with a meaningless arguments about muh trolls.
  
  source
  - gabe@literature.cafe ⁨11⁩ ⁨months⁩ ago
    But they are correct. There are vulnerable groups of people who have a harassment risk against them. We share the fediverse with others, be mindful of that. Making a search engine or an archiver for lemmy is such a good idea with how it functions! but for the wider fediverse… that’s just directly contradictory to its culture.
    
    source
    -> View More Comments
  - spiderman@ani.social ⁨11⁩ ⁨months⁩ ago
    anything you post on internet can be indexed. if someone wants to post some thing on their little private garden they are options for that too. fediverse has potential to grow and if we try to stop everything that could help to grow it as “no only trolls will use it”, after some point no body except people who complaint won’t use it. do you want fediverse to be your own little echo chamber?
    
    source
  - TimLovesTech@badatbeing.social ⁨11⁩ ⁨months⁩ ago
    Your post is of much privilege, marginalized groups care about trolls very much.
    
    source
- MHLoppy@fedia.io ⁨11⁩ ⁨months⁩ ago
  It's worth noting that since FedSearch, Mastodon has actually natively implemented opt-in search on posts.
  
  source
- 0x1C3B00DA@kbin.social ⁨10⁩ ⁨months⁩ ago
  That post wasn't claiming that a search engine would only be used by trolls; it was explaining that they shut down their project because a chunk of the fediverse thinks that and complain about any search engine projects. Discoverability is one of the network's biggest challenges and a search engine could really help with that.
  
  source
  - TimLovesTech@badatbeing.social ⁨10⁩ ⁨months⁩ ago
    Yes, not only used by trolls, but would be a tool that could be leveraged by trolls. And I think the fediverse makes it easier to establish instances for marginalized groups, but also has more admins that just don’t want trolls because nobody here is making $ off them like the corporate socials are. I think if adding search that is going to try and vacuum up everyone’s posts in the fediverse and make them easily sortable/targetable without instance admins permission, then that isn’t cool. If someone is running a general instance that covers nothing that a troll could latch onto and wants the instance catalogued and searchable then that’s fine by me. I don’t think boys should be doing that to the fediverse as a whole without admin permission though.
    
    source
    -> View More Comments
Treczoks@kbin.social ⁨11⁩ ⁨months⁩ ago
A good search engine would be quite important. One thing that annoyed me back on the site that should not be named was that their search engine was completely useless - It was not even capable to find posts where I entered verbatim text of.

Having a good search engine that can actually find a post I was looking for would be a major plus for the fediverse.

source
- lautan@lemmy.ca ⁨11⁩ ⁨months⁩ ago
  Yep, the idea is to simulate the type of results you get from Google. People trust Lemmy answers more than spam sites now a days.
  
  source
stockRot@lemmy.world ⁨11⁩ ⁨months⁩ ago
You should federate the search engine so that folks can defed from the search as desired.

But then we would need a search engine for all the search engines…

source
- KISSmyOS@lemmy.world ⁨11⁩ ⁨months⁩ ago
  Everything you post on the Fediverse is public.
  If you don’t want to show up in an internet search, post your stuff on your own server and only give access to the people you want to invite.
  
  source
- lautan@lemmy.ca ⁨11⁩ ⁨months⁩ ago
  Yep
  
  source
thisisawayoflife@lemmy.world ⁨11⁩ ⁨months⁩ ago
Is this something you can point yacy at?

source
- lautan@lemmy.ca ⁨11⁩ ⁨months⁩ ago
  I heard it’s not optimized well but I’ll take a look at it.
  
  source
  - thisisawayoflife@lemmy.world ⁨11⁩ ⁨months⁩ ago
    I’d be interested to know what “not optimized” means.
    
    source
    -> View More Comments
Valmond@lemmy.mindoki.com ⁨11⁩ ⁨months⁩ ago
I love the idea, especially from a technical standpoint!

How big is the fediverse today? How many posts are there? What kind of algorithms atmre you using to store the results? Do you scan sites and then their connected sites or do you have a premade list?

More technical information please 😊!

source
- lautan@lemmy.ca ⁨11⁩ ⁨months⁩ ago
  The fediverse is a few thousand servers, from Mastodon, Lemmy, etc. Can’t say the amount of posts but there are a lot.
  
  So on the more technical side, I plan on using a light weight fast search engine called Sonic (It’s written in rust). I have already used it in other projects and it can handle billions of messages / posts. But it has a cost it doesn’t have faceted search, like for example if you want to exclude certain texts from the results. I think this is a fair trade off. The other solution would be to use something more mature like ElasticSearch but it’ll be expensive (I’m assuming not much money will be made from this and I’m talking about donations)
  
  For scanning sites there are premade lists to start with and it’ll be possible to scan new sites from other instances if found. So a bit of both.
  
  source
yuki2501@lemmy.world ⁨11⁩ ⁨months⁩ ago
Just, for the love of god, make sure to make it opt-in. Don’t scan people’s posts without their consent.

Also, if you’re going with this, make sure to respect people’s requests for removal, e.g. deleting their posts from the engine when they delete them from their instances. Otherwise you’d get in trouble with the EU regarding GDPR…

source
- jol@discuss.tchncs.de ⁨11⁩ ⁨months⁩ ago
  Your public posts are not protected from GDPR from indexing…
  
  source
Sensitivezombie@lemmy.zip ⁨10⁩ ⁨months⁩ ago
I support bigger picture. Rather than an independent site, wouldn’t it be more practical to work with current fediverse app developers for lemmy, mastodon, etc to integrate search engine within the app?

source
- lautan@lemmy.ca ⁨10⁩ ⁨months⁩ ago
  I’m reaching out to see their thoughts. But there are limitations to what they can index.
  
  source
jcrabapple@infosec.pub ⁨10⁩ ⁨months⁩ ago
The fedi elite will snipe you from a bell tower.

source
Aopen@discuss.tchncs.de ⁨11⁩ ⁨months⁩ ago
Why wouldnt people want do have search engine? Without it Fediverse stands no chance against non-free internet. Everything posted here would be much more valuable if it was searchable. Now comment posted once is viewed only until post gets less popular. Any other site of this kind displays answers decades old. Privacy isnt issue as everything posted here is available to everyone on internet.

source
JoBo@feddit.uk ⁨10⁩ ⁨months⁩ ago
I don’t know anything about the technical side of this. But I would (possibly naively) think that it would be simpler to have a filter that you could automatically apply to sift bog-standard search engine results for Fediverse instances? Like adding “site:uk” to the end of a normal search, except that your filter term would check a list of Fediverse instances to return the relevant results.

And make it an app/add-on so that people can use it with their usual search strategies.

source
- lautan@lemmy.ca ⁨10⁩ ⁨months⁩ ago
  People been doing that now but it comes with limitations. Thanks for the input.
  
  source