Comment on Reddit will block the Internet Archive

captainastronaut@seattlelunarsociety.org ⁨1⁩ ⁨month⁩ ago

• hamFoilHat@lemmy.world ⁨1⁩ ⁨month⁩ ago

  It is my understanding that if you block the wayback machine from indexing your site it will also delist the history as well.

  source

  • Jason2357@lemmy.ca ⁨1⁩ ⁨month⁩ ago

    They do archive sites against the owners wishes when they consider it an important site for public archiving, like some news sites. They are in no obligation to delete the archives and hope they don’t.

    source

    • tal@lemmy.today ⁨1⁩ ⁨month⁩ ago

      Parties have archived the data from pushshift, which cover a lot of Reddit history.

      kagis

      academictorrents.com/…/56aa49f9653ba545f48df2e336…

      Subreddit comments/submissions 2005-06 to 2023-12

      This is the top 40,000 subreddits from reddit’s history in separate files. You can use your torrent client to only download the subreddit’s you’re interested in.

      I mean, that won’t have the past 18 months or some low-traffic subreddits, but…

      source
  • Natanael@infosec.pub ⁨1⁩ ⁨month⁩ ago

    The ability to block crawling is separate from the ability to delist old pages. The latter usually happens after domains change owners

    source
• Sxan@piefed.zip ⁨1⁩ ⁨month⁩ ago

  LOL I should have scrolled down first You said what I said, with fewer words, first.

  source