Looking for some selfhosted webscanner/scraper

Submitted ⁨⁨1⁩ ⁨year⁩ ago⁩ by ⁨yournamehere@lemm.ee⁩ to ⁨selfhosted@lemmy.world⁩

hi, i’m kinda out of ideas -maybe you know some selfhostable solution.

I want to monitor some of my pages for gdpr compliance and have reports on that.

So far i tried selenium container with k6, i tried selenium locally with some Python automation and even gave phantomJS with webXray a try. In any case sth. is missing…like selenium wont grab the thirdparty cookies, webxray code has outdated Python and so on.

You guys have any suggestion on how i could selfhost my own automated toolset to check entire webpages for all cookies and thirdpartyscripts? Ppl make changes which might have impact on the gdpr compliance and i dont want to check manually. How would you do it selfhosted?

source

Comments

Sort:hotnew top

morras@links.hackliberty.org ⁨1⁩ ⁨year⁩ ago
The EDPS open-sourced a cookie inspection tool:

edps.europa.eu/edps-inspection-software_en

source
- yournamehere@lemm.ee ⁨1⁩ ⁨year⁩ ago
  oh wow…how did Google not suggest to me? thanks mate.
  
  source
  - morras@links.hackliberty.org ⁨1⁩ ⁨year⁩ ago
    You are welcome :)
    
    source
steersman2484@sh.itjust.works ⁨1⁩ ⁨year⁩ ago
You could try Puppeteer, but I don’t know if you can check third party cookies.

Also I can recommend using Apache Airflow for automation. It is like a fancy crontab with a webui and email notifications, and lots of other stuff.

source
- yournamehere@lemm.ee ⁨1⁩ ⁨year⁩ ago
  thanks. was thinking about airflow or n8n for automation but i still cant find a way to check the thirdpartycookies. someone wrote firefoxwebdriver could do it but you’d need to extract from the profile an stuff…nothing really worked.
  
  source
0xD@infosec.pub ⁨1⁩ ⁨year⁩ ago
Check out nuclei or maybe greenbone.

source
- yournamehere@lemm.ee ⁨1⁩ ⁨year⁩ ago
  those look exciting! thank you very much.
  
  source