Open Menu
AllLocalCommunitiesAbout
lotide
AllLocalCommunitiesAbout
Login

Chad scraper

⁨1017⁩ ⁨likes⁩

Submitted ⁨⁨1⁩ ⁨year⁩ ago⁩ by ⁨dramaticcat@sh.itjust.works⁩ to ⁨[deleted]⁩

https://sh.itjust.works/pictrs/image/dcbf36a9-c675-411c-94c1-cbf74ab477cb.webp

source

Comments

Sort:hotnewtop
  • bill_1992@lemmy.world ⁨1⁩ ⁨year⁩ ago

    Everyone loves the idea of scraping, no one likes maintaining scrapers that break once a week because the CSS or HTML changed.

    source
    • DigitalPaperTrail@kbin.social ⁨1⁩ ⁨year⁩ ago

      spite can be a great motivator, though

      source
      • Anonymousllama@lemmy.world ⁨1⁩ ⁨year⁩ ago

        This one. One of the best motivators. Sense of satisfaction when you get it working and you feel unstoppable (until the next subtle changes happens anyway)

        source
      • archomrade@midwest.social ⁨1⁩ ⁨year⁩ ago

        I feel this

        source
    • camr_on@lemmy.world ⁨1⁩ ⁨year⁩ ago

      I loved scraping until my ip was blocked for botting lol. I know there’s ways around it it’s just work though

      source
      • pennomi@lemmy.world ⁨1⁩ ⁨year⁩ ago

        I successfully scraped millions of Amazon product listings simply by routing through TOR and cycling the exit node every 10 seconds.

        source
        • -> View More Comments
      • Touching_Grass@lemmy.world ⁨1⁩ ⁨year⁩ ago

        You guys use IP’S?

        source
        • -> View More Comments
    • dangblingus@lemmy.world ⁨1⁩ ⁨year⁩ ago

      Or in the case of wikipedia, every table on successive pages for sequential data is formatted differently.

      source
    • Matriks404@lemmy.world ⁨1⁩ ⁨year⁩ ago

      Just use AI to make changes ¯_(ツ)_/¯

      source
      • anarchy79@lemmy.world ⁨1⁩ ⁨year⁩ ago

        Here take these: \\

        source
        • -> View More Comments
  • indepndnt@lemmy.world ⁨1⁩ ⁨year⁩ ago

    I’m down with scraping, but “parses HTML with regex” has got me fucked up.

    source
    • 257m@sh.itjust.works ⁨1⁩ ⁨year⁩ ago

      Relevant SO post. stackoverflow.com/…/regex-match-open-tags-except-…

      source
      • ChickenLadyLovesLife@lemmy.world ⁨1⁩ ⁨year⁩ ago

        13 years ago my god. I wonder what Jon Skeet is doing these days.

        I remember when he passed me in the reputation ranking back in the early days and thinking that I needed to be a little bit more active on the site to catch him lol.

        source
      • DAMunzy@lemmy.dbzer0.com ⁨1⁩ ⁨year⁩ ago

        That was a great read. Thanks!

        source
    • demodawg@lemmy.world ⁨1⁩ ⁨year⁩ ago

      This is the way

      source
    • tetelestia@lemmy.world ⁨1⁩ ⁨year⁩ ago

      What’s wrong with parsing HTML with regex?

      source
      • GuybrushThreepwo0d@programming.dev ⁨1⁩ ⁨year⁩ ago

        Go and look it up on stack overflow

        source
      • indepndnt@lemmy.world ⁨1⁩ ⁨year⁩ ago

        In short, it’s the wrong tool for the job.

        In practice, if your target is very limited and consistent, it’s probably fine. But as a general statement about someone’s behavior, it really sounds like someone is wasting a lot of time and regularly getting sub-par results.

        source
  • ViscloReader@lemmy.world ⁨1⁩ ⁨year⁩ ago

    Just a heads up for companies thinking it’s wrong to scrap: if you don’t want info to be scraped, don’t put it on the internet.

    source
    • DAMunzy@lemmy.dbzer0.com ⁨1⁩ ⁨year⁩ ago

      But, but, norobots.txt!

      Chad doesn’t care!

      source
      • PoolloverNathan@programming.dev ⁨1⁩ ⁨year⁩ ago

        uses curl manually instead

        source
  • SeeMinusMinus@lemmy.world ⁨1⁩ ⁨year⁩ ago

    The sad part is that scrapping is often easier then using the api.

    source
    • Anonymousllama@lemmy.world ⁨1⁩ ⁨year⁩ ago

      Much less beholden to arbitrary rules also. Way too many times companies will just up and lift their API access or push through restrictions. No ty, I’ll just access it myself then

      source
      • AeroLemming@lemm.ee ⁨1⁩ ⁨year⁩ ago

        Cough Reddit cough

        source
      • SeeMinusMinus@lemmy.world ⁨1⁩ ⁨year⁩ ago

        API starter kit

        • Outdated and unsupported and hasn’t been replaced yet but is the standard way to use the service.
        • Lots of authorization tokens.
        • The example in the docs doesn’t work (if there is one).
        • You have no idea where the online tutorial got the information because it doesn’t have links to resources and the docs have barely anything even though its giant.
        • Uses asynchronous programming to make it faster but its still much much slower then scrapping without asynchronous programming.
        source
    • clbustos@lemmy.world ⁨1⁩ ⁨year⁩ ago

      So true that it hurta

      source
  • lnee@lemm.ee ⁨1⁩ ⁨year⁩ ago

    I scrape with bash lord help me.

    source
    • OmnislashIsACloudApp@lemmy.world ⁨1⁩ ⁨year⁩ ago

      there’s literally dozens of us!

      or maybe just 2 idk

      source
    • MaxVoltage@lemmy.world ⁨1⁩ ⁨year⁩ ago

      as a windows user i say kindly on our behalf thank you for pushing the envelope ✉

      source
  • darcy@sh.itjust.works ⁨1⁩ ⁨year⁩ ago

    someone’s never used a good api. like mastodon

    source
  • havokdj@lemmy.world ⁨1⁩ ⁨year⁩ ago

    Hold on, I thought it was supposed to be realism on the virgin’s behalf and ridiculous nonsense on the chad behalf:

    All I see is realism on both sides lol

    source
  • sebinspace@lemmy.world ⁨1⁩ ⁨year⁩ ago

    I wanted to build a Discord bot that would check NIST for new CVEs every 24 hours. But their API leaves quiiiiiiite a bit to be desired.

    Their pages, however…

    source
    • khaffner@lemmy.world ⁨1⁩ ⁨year⁩ ago

      Just use this github.com/CVEProject/cvelistV5/tree/main/cves

      source
      • sebinspace@lemmy.world ⁨1⁩ ⁨year⁩ ago

        Oh yeah, that’s much more robust

        source
  • lemmywizard@lemm.ee ⁨1⁩ ⁨year⁩ ago

    It’s all fun and games until you have to support all this shit and it breaks weekly!

    That being said, I do miss the simplicity of maintaining selenium projects for work

    source
  • snek@lemmy.world ⁨1⁩ ⁨year⁩ ago

    I used Twitter Scraper to get twitter data for my thesis. Shortly after, it became obsolete. github.com/taspinar/twitterscraper/issues/368

    rip twitter scraper

    source
  • idiosynk@lemmy.world ⁨1⁩ ⁨year⁩ ago

    I’ve just discovered selenium and my life has changed.

    source
    • XEAL@lemm.ee ⁨1⁩ ⁨year⁩ ago

      I created a shitty script (with ChatGPT’s help) that uses Selenium and can dump a Confluence page from work, all its subpages and all linked Google Drive documents.

      source
    • Wakmrow@lemmy.world ⁨1⁩ ⁨year⁩ ago

      How so?

      source
      • idiosynk@lemmy.world ⁨1⁩ ⁨year⁩ ago

        When a customer needs a part replaced, they send in shipping data. This data has to be entered into 3-4 different web forms and an email. This allows me to automate it all from a single form that has built in error checking so human mistakes are limited.

        Company could probably automate this all in the backend but they won’t :shrug:

        source
        • -> View More Comments
  • McBain@feddit.ch ⁨1⁩ ⁨year⁩ ago

    I use scrapy. It has a steeper learning curve than other libraries, but it’s totally worth it.

    source
    • rishado@lemmy.world ⁨1⁩ ⁨year⁩ ago

      Splash ftw

      source
  • Crashumbc@lemmy.world ⁨1⁩ ⁨year⁩ ago

    ROFL, Chad only thinks that shit works

    source
  • Irkam@jlai.lu ⁨1⁩ ⁨year⁩ ago

    Let me introduce you to WooB (formerly WEBooB).

    source
    • Rodeo@lemmy.ca ⁨1⁩ ⁨year⁩ ago

      Why on earth would they have changed that. WEBooB is a way better name.

      source
      • planish@sh.itjust.works ⁨1⁩ ⁨year⁩ ago

        But it’s got boob in it.

        source
  • chemicalwonka@discuss.tchncs.de ⁨1⁩ ⁨year⁩ ago

    Let’s see what WEI (if implemented ) will do with the scrapers. The future doesn’t look promising.

    source
    • kadotux@sopuli.xyz ⁨1⁩ ⁨year⁩ ago

      What’s that?

      source
      • Username@feddit.de ⁨1⁩ ⁨year⁩ ago

        A google/chrome proposal for browser verification, i.e. killing addons and custom browsers.

        source
        • -> View More Comments
  • ArchTemperedKoala@lemmy.world ⁨1⁩ ⁨year⁩ ago

    I have totally no idea what these are about…

    source
    • DarkSpectrum@lemmy.world ⁨1⁩ ⁨year⁩ ago

      ‘Scraping’ is the process of anonymously and programmatically collecting data from a webpage(s), often without the website’s permission and only limited to the content made publicly available. This is I contrast to using an API provided by the database owner which is limited by tokens, access volume, available end points etc.

      source
    • SternburgExport@feddit.de ⁨1⁩ ⁨year⁩ ago

      Everytime I think I’m good with tech, something like this shows up in my feed and makes me realize I know jackshit.

      source
  • NigelFrobisher@aussie.zone ⁨1⁩ ⁨year⁩ ago

    My undergrad project was a scraper - there just wasn’t a name for it yet,

    source
    • newIdentity@sh.itjust.works ⁨1⁩ ⁨year⁩ ago

      Scrapers have been a thing since the web exists.

      One of the first one’s is even called WebCrawler

      source
  • Touching_Grass@lemmy.world ⁨1⁩ ⁨year⁩ ago

    Fuck, I think I’ve been doing it wrong and this meme gave me more things to learn than any YouTube video has

    source
    • Rodeo@lemmy.ca ⁨1⁩ ⁨year⁩ ago

      Memes have always been superior to YouTube videos

      source
  • UraniumBlazer@lemm.ee ⁨1⁩ ⁨year⁩ ago

    Sorry, I’m ignorant in this matter. Why exactly would you want to scrape websites aside from collecting data for ML? What kind of irreplaceable API are you using? Someone please educate me here.

    source
    • coltorl@programming.dev ⁨1⁩ ⁨year⁩ ago

      API might cost a lot of money for the amount of requests you want to send. API may not include some fields in the data you want. API is rate limited, scraping might not be. API requires agreement to usage terms, scraping does not (though the recent LinkedIn scraping case might weaken that argument.)

      source
      • Kuma@lemmy.world ⁨1⁩ ⁨year⁩ ago

        This kinda reminds me of pirating vs paying. Using api = you know it will always be the same structure and you will get the data you asked for. Otherwise you will be notified unless they version their api. There is usual good documentation. You can always ask for help.

        Scraping = you need to scout the whole website yourself. you need to keep up to date with the the websites structure and to make sure they haven’t added ways to block bots (scraping). Error handling is a lot more intense on your end, like missing content, hidden content, query for data. the website may not follow the same standards/structuree throughout the website so you need to have checks for when to use x to get y. The data may need multiple request because they do not show for example all the user settings on one page but in an api call they would or it is a ajax page and you need to run Javascript scripts and click on buttons that may change id, class or text info and they may load data when you do x with Javascript so you need to emulate the webpage.

        So my guess is that scraping is used most often when you only need to fetch simple data structures and you are fine with cleaning up the data afterwards. Like all the text/images on a page, checking if a page has been updated or just save the whole page like wayback machine.

        source
        • -> View More Comments
      • olympicyes@lemmy.world ⁨1⁩ ⁨year⁩ ago

        My understanding is that the result of the LinkedIn case is that you can scrape data that you have permission to view but not to access data that you were not intended to. The end result that ClickWrap agreements are unenforceable.

        source
  • InternetTubes@lemmy.world ⁨1⁩ ⁨year⁩ ago

    So, where can I find the Chad scrapper for reddit?

    source
    • PoolloverNathan@programming.dev ⁨1⁩ ⁨year⁩ ago

      If you wanted a chad scraper, look at Pushshift. Reveddit relied on it before Reddit got it taken down.

      source
  • madcaesar@lemmy.world ⁨1⁩ ⁨year⁩ ago

    How exactly do you make money scraping?

    source
    • Sotuanduso@lemm.ee ⁨1⁩ ⁨year⁩ ago

      By getting someone to hire you to do it.

      source
      • anteaters@feddit.de ⁨1⁩ ⁨year⁩ ago

        Mind blowing stuff

        source
      • madcaesar@lemmy.world ⁨1⁩ ⁨year⁩ ago

        No I mean more what is the use case where it would be worth scrapping on a massive scale?

        source
        • -> View More Comments
    • Hawk@lemmynsfw.com ⁨1⁩ ⁨year⁩ ago

      Imagine an investment firm looking at a property market. They need data like price trends in the surrounding area.

      Real estate API is expensive, scraping is free. By hiring an employee the can save money.

      source
    • Theharpyeagle@lemmy.world ⁨1⁩ ⁨year⁩ ago

      There’s a ton of money to be made from scraping, consolidating, and organizing publicly accessible data. A company I worked for did it with health insurance policy data because every insurance company has a different website with a different data format and data that updates every day. People will pay da big bux for someone to wrap all that messiness into a neat, consistent package.

      Right now, gathering machine learning data is hot, cause you need a lot of it to train a model. Companies may specialize in getting, say, social media posts from all kinds of sites and putting them together in a consistent format.

      source
  • lnee@lemm.ee ⁨1⁩ ⁨year⁩ ago

    That’s why I use geddit

    source
  • redw04@lemmy.ca ⁨1⁩ ⁨year⁩ ago

    So uh…as someone who’s currently trying to scrape the web for email addresses to add to my potential client list … where do I start researching this?

    source
    • lutillian@sh.itjust.works ⁨1⁩ ⁨year⁩ ago

      Start looking into selenium, probably in Python. It’s one of the easier to understand forms of scraping. It’s mainly used to web testing, though you can definitely use it for less… nice purposes.

      source
    • PieMePlenty@lemmy.world ⁨1⁩ ⁨year⁩ ago

      Step one will be learning to code in any language. Step two is using a library to help with it and don’t use regex like the meme says haha. HtmlAgilityPack has always been there for me.

      source
      • Rodeo@lemmy.ca ⁨1⁩ ⁨year⁩ ago

        Virgin library user vs. Chad regex dev

        source