Researchers Scrape 2 Billion Discord Messages and Publish Them Online

Submitted ⁨⁨4⁩ ⁨weeks⁩ ago⁩ by ⁨Mindwolf@lemm.ee⁩ to ⁨technology@lemmy.world⁩

https://www.404media.co/researchers-scrape-2-billion-discord-messages-and-publish-them-online/

Researchers published a massive database of more than 2 billion Discord messages that they say they scraped using Discord’s public API. The data was pulled from 3,167 servers and covers posts made between 2015 and 2024, the entire time Discord has been active.

Though the researchers claim they’ve anonymized the data, it’s hard to imagine anyone is comfortable with almost a decade of their Discord messages sitting in a public JSON file online. Separately, a different programmer released a Discord tool called “Searchcord” based on a different data set that shows non-anonymized chat histories.

source

Comments

Sort:hotnew top

asbestos@lemmy.world ⁨4⁩ ⁨weeks⁩ ago
Probably our only chance to find solutions to problems with open source software that uses Discord as their forum

source
- boatswain@infosec.pub ⁨4⁩ ⁨weeks⁩ ago
  Seriously. It’s beyond painful when some open source project only uses Discord for communication. You have to hope that you post your question at a time when the right people are online, and that there’s not a more interesting conversation going on, otherwise it just gets lost. Index that whole dataset.
  
  source
  - ALostInquirer@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
    Given some similar issues, why is it some projects still use IRC then?
    
    source
    -> View More Comments
  - Peffse@lemmy.world ⁨4⁩ ⁨weeks⁩ ago
    I’ve always wanted to contribute to The Cutting Room Floor wiki but they hide registration behind a Discord server bot that will give the registration code.
    
    source
  - Ulrich@feddit.org ⁨3⁩ ⁨weeks⁩ ago
    
    Index that whole dataset
    
    I’ve seen a few projects doing just that with www.answeroverflow.com and they have come up in my web searches. Not really a solution but at least a stopgap.
    
    source
- Dojan@pawb.social ⁨4⁩ ⁨weeks⁩ ago
  I spent nearly three hours today between discord and matrix trying to figure out how to get these two pieces of software to talk using a certain protocol.
  
  Imagine if there were online indexable platforms where people could publish this information so it’s easily accessible rather than having to scour through message logs hoping to find the right keywords. Such a technology surely doesn’t exist already, right?
  
  I hate discord.
  
  source
  - dual_sport_dork@lemmy.world ⁨4⁩ ⁨weeks⁩ ago
    I don’t hate Discord, I simply hate that so many projects and companies have unanimously decided to use it as the wrong tool for the wrong job.
    
    It’s fine for its intended use case, which is bickering with my friends about video games and fiction, and spamming each other with .gifs and meme images.
    
    source
    -> View More Comments
  - MDCCCLV@lemmy.ca ⁨4⁩ ⁨weeks⁩ ago
    Yeah, but then you have something like when people protest deleted their history on reddit which is fine as a protest tactic but leaves a hole where your specific question came up but now there’s nothing there.
    
    source
  - spiderhamster@lemmy.world ⁨4⁩ ⁨weeks⁩ ago
    you get it to work? i didnt have time to get it working in both directions. matrix to discord worked fine but not the other way.
    
    source
    -> View More Comments
  - interdimensionalmeme@lemmy.ml ⁨4⁩ ⁨weeks⁩ ago
    You mean NNTP ?
    
    source
- nawa@lemmy.world ⁨4⁩ ⁨weeks⁩ ago
  Lol, I’ve read this headline and thought “thank fuck, probably the only option to have Discord’s content readable”, I like how universal this opinion is
  
  source
Metz@lemmy.world ⁨4⁩ ⁨weeks⁩ ago
So basically discord finally got a usable search. I count that as a win.

source
donuts@lemmy.world ⁨4⁩ ⁨weeks⁩ ago
Well yeah, it’s not encrypted. It would be the same as 10 years of Reddit posts or Lemmy posts scraped

source
- simple@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
  This isn’t even them scraping private chats and small servers, they just scraped public servers in the discovery tab. None of that information was ever private, and every user can browse the chat history there.
  
  source
  - donuts@lemmy.world ⁨4⁩ ⁨weeks⁩ ago
    Yeah, exactly. It may sound scary or like a violation of privacy, but there is no privacy when posting to public online areas.
    
    source
  - micka190@lemmy.world ⁨4⁩ ⁨weeks⁩ ago
    “Researchers scrape thousands of hours of news footage from their TVs!” is about as big a deal, honestly.
    
    source
  - taiyang@lemmy.world ⁨4⁩ ⁨weeks⁩ ago
    In other words-- your sexting is safe, friends.
    
    source
  - MoonRaven@feddit.nl ⁨4⁩ ⁨weeks⁩ ago
    It also includes deleted messages. And they refuse to delete things when someone opts out.
    
    source
    -> View More Comments
- Quibblekrust@thelemmy.club ⁨4⁩ ⁨weeks⁩ ago
  There’s literally no difference. Each Discord server is like a tiny chunk of Reddit. If anyone expected any privacy on these servers, they’re nuts.
  
  source
Tattorack@lemmy.world ⁨4⁩ ⁨weeks⁩ ago
I see a lot of drama here in the thread, people decrying data leaks, how Discord is very very bad, and a number of people wanting the “good old days” of forums.

Yes. I like forums too, but, uh…

These researchers scraped publicly posted messages. Keyword here being “public”. How would anything similarly public, like a forum, be better?

I actually remember the times when forums were at their peak. I hung out on BZPower for Bionicle things, and the Relic News Forum for Homeworld modding. You know what they had? Google bots that scraped messages, looked for certain words, and populated websites with advertisements based on what it could scrape from forums.

Pretty sure Lemmy doesn’t do encryption either, unless there’s some very special, private Lemmy server that nobody has access to. So the researchers could’ve just as well scraped the fediverse.

source
- Gibibit@lemmy.world ⁨4⁩ ⁨weeks⁩ ago
  Yeah this being just as easy on bb forums or literally any webpage with a public comment section was my first thought as well…
  
  Isn’t most of the internet scraped anyways, by the internet archive? The concerning part is that this is 100% going to be used to train some coomer brained AI. Scraping, botting, scamming: all those things are going to happen on large public communities.
  
  source
  - Melvin_Ferd@lemmy.world ⁨4⁩ ⁨weeks⁩ ago
    Yea a lot of this stuff is usher in new laws to prevent data scraping.
    
    Propaganda spreads easily by fake accounts. How would we detect these huge operations if they’re creating accounts and deleting them after or trying to hide among us. We would need lots of these huge data dumps so we could mine it and find patterns. So the powers that have the influence spread the message that we must hate all this scraping. We need laws to prevent it. That’s the end game.
    
    source
- FauxLiving@lemmy.world ⁨4⁩ ⁨weeks⁩ ago
  
  How would anything similarly public, like a forum, be better?
  
  Forums were the primary way that groups would talk with one another pre-global scale social media.
  
  They could contain public subforums, but the majority of all of the forums that I’ve been a part of were not viewable without an account, which was manually approved or required a small payment (to make bans have a chance to actually stick).
  
  source
- hansolo@lemm.ee ⁨3⁩ ⁨weeks⁩ ago
  People in general have no idea and just wasn’t to get spun up in drama and manufactured outrage.
  
  Same thing happened when people started scrapping Twitter 10-15 years ago.
  
  source
- mic_check_one_two@lemmy.dbzer0.com ⁨3⁩ ⁨weeks⁩ ago
  People saw “scraped Discord messages” and immediately jumped to “oh shit fuck my private chats have been leaked everybody panic”.
  
  source
CosmoNova@lemmy.world ⁨4⁩ ⁨weeks⁩ ago
That’s good news. Internet archiving is an important endeavor because you never know when they‘ll pull the plug. Now it‘s a little more secured and probably far more useful than in Discord‘s hands alone.

source
- Mustakrakish@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
  Not for messages that are supposed to be private lol. Let me just make a copy of all texts you’ve sent over the last decade, for “archiving”.
  
  source
  - MangoPenguin@lemmy.blahaj.zone ⁨3⁩ ⁨weeks⁩ ago
    This says it was done via the API so they wouldn’t be private messages.
    
    source
  - nomy@lemmy.zip ⁨3⁩ ⁨weeks⁩ ago
    Texts are sent in plain-text and I wouldn’t recommend discussing anything you’d like to keep private via text.
    
    source
  - shaggyb@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
    If you think messages you post anywhere on the internet are private, you’re in for a bad time.
    
    source
sp3ctr4l@lemmy.dbzer0.com ⁨4⁩ ⁨weeks⁩ ago
So this is:

'Uh guys, Discord chats leaked…"

For… what, just literally everyone who used Discord between 2015 and 2017, everyone who was an early adopter?

Dear fucking god.

I used to say ‘someday, people will learn’, but fucking no obviously not, no they won’t, almost everyone is an idiot and/or truly doesn’t care.

… I guess this’ll be fodder for a whole bunch of dramatubers / pedohunters for the next year or so…

source
- interdimensionalmeme@lemmy.ml ⁨4⁩ ⁨weeks⁩ ago
  The disappearance of forum public discussion to unsearchable, unpreserved, discord semi-private discussion chambers is probably the largest informational catastrophe of the internet so far.
  
  source
  - sp3ctr4l@lemmy.dbzer0.com ⁨4⁩ ⁨weeks⁩ ago
    I potentially agree, but as a possible competitor, I submit:
    
    Everything DOGE has done in the last 3 months.
    
    source
    -> View More Comments
  - Tattorack@lemmy.world ⁨4⁩ ⁨weeks⁩ ago
    But I want something like Discord. Just lot corporate owned…
    
    source
    -> View More Comments
Entertain529@lemmy.ml ⁨4⁩ ⁨weeks⁩ ago
Saving this article for the next time someone says “Just message me on discord its easier”.

source
lefixxx@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
“scraped” via API? I don’t think It means what you think it means.

source
snowsuit2654@lemmy.blahaj.zone ⁨3⁩ ⁨weeks⁩ ago
“anonymized” sure. I highly doubt they read every message. I’m sure there is lots of de-anonymizing information in the messages themselves

For example–

Anon1: “hey jeff, wanna play Minecraft?” Anon2: “sure”

Thus we know Jeff’s name. I imagine there’s a lot of this.

source
- moseschrute@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
  Shit. My name is Jeff. Now they know
  
  source
fullsquare@awful.systems ⁨4⁩ ⁨weeks⁩ ago
🚩

marked safe

from Brazilian mass discord message leak

(never used discord)

source
CosmicTurtle0@lemmy.dbzer0.com ⁨4⁩ ⁨weeks⁩ ago
Clickwall Removed

source
Samsy@lemmy.ml ⁨4⁩ ⁨weeks⁩ ago
Meanwhile AI scrapers: This will be a fine addition to my collection.

source
toastmeister@lemmy.ca ⁨4⁩ ⁨weeks⁩ ago
Great news for open source AI.

source
ABetterTomorrow@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
wtf…… going to get worse after IPO!

source
- Jimmycakes@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
  If you don’t want strangers knowing what you say don’t join open servers it’s pretty easy
  
  source
  - ABetterTomorrow@lemm.ee ⁨3⁩ ⁨weeks⁩ ago
    Open or close, going to get worse!
    
    source
    -> View More Comments
gwilikers@lemmy.ml ⁨4⁩ ⁨weeks⁩ ago
So how does this work? Like how did those messages through API calls? Also, is this not something that Discord would dislike since it dilutes the value of their data horde?

source
joyjoy@lemm.ee ⁨4⁩ ⁨weeks⁩ ago
Public data should be accessible anonymously. You can’t change my mind.

source
Reygle@lemmy.world ⁨4⁩ ⁨weeks⁩ ago
Ooh! Do Teams next

source
thesohoriots@lemmy.world ⁨4⁩ ⁨weeks⁩ ago
They just wanted to find new slurs.

source
pelespirit@sh.itjust.works ⁨4⁩ ⁨weeks⁩ ago
Every time you post, you’re posting so that Meta, Google, Reddit and every known retail store like Walmart, Target, Kroger, etc. can see it because they bought that info or harvested it themselves. I think these are great announcements so people can see who sees and manipulates you with your own contributions of data.

source
- dubyakay@lemmy.ca ⁨4⁩ ⁨weeks⁩ ago
  The feedback loop is everywhere in tech.
  
  source
obbeel@lemmy.eco.br ⁨4⁩ ⁨weeks⁩ ago
This is just trolling, at this point.

source
arararagi@ani.social ⁨3⁩ ⁨weeks⁩ ago
If they were on OPEN servers, I doubt they cared that much.

source
reiterationstation@lemm.ee ⁨3⁩ ⁨weeks⁩ ago
I was hoping people would do this!!!

source
MrSoup@lemmy.zip ⁨4⁩ ⁨weeks⁩ ago
I can’t find this “public” json

source
- obbeel@lemmy.eco.br ⁨4⁩ ⁨weeks⁩ ago
  zenodo.org/records/15170676
  
  source
  - MrSoup@lemmy.zip ⁨4⁩ ⁨weeks⁩ ago
    Thanks, but the file seems to be restricted.
    
    source
    -> View More Comments
dgdft@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
I was hoping to play around with the dataset over the weekend to toy with some text-embedding techniques, but they’ve pulled the cord on the download links.

Anyone have a copy of the full archive they’re willing to share, or a magnet link?

source
pewgar_seemsimandroid@lemmy.blahaj.zone ⁨3⁩ ⁨weeks⁩ ago
404? another source please? I don’t trust them on this exact thing.

source