Comments

• easeKItMAn@lemmy.world ⁨1⁩ ⁨year⁩ ago

  If I’m understanding you correctly, you could make use of a shell script for this. Use WGET to download lists, then combine them into a single large file, and finally create a new file with no duplicates by using “awk ‘!visited[$0]++’”

  wget
  cat *.txt > all.txt (This overwrites all.txt) awk ‘!visited[$0]++’ all.txt > no_duplicates.txt

  source

  • BinaryUnit@lemmy.world ⁨1⁩ ⁨year⁩ ago

    When no tool is available bash to the rescue, thank you for this it seems actually simpler then I thought :)

    source
• Appoxo@lemmy.dbzer0.com ⁨1⁩ ⁨year⁩ ago

  Afaik pihole does parse and then merge the lists into a single block list.

  source
• CarbonatedPastaSauce@lemmy.world ⁨1⁩ ⁨year⁩ ago

  I doubt you’ll find something off the shelf for this. I wrote a powershell script that deduplicates lists and also does a pass over the results to convert any blocks to CIDR notation. If you’re interested I’ll share it.

  But honestly you could probably have ChatGPT whip this up for you in your language of choice. It’s pretty straightforward.

  source

  • nyar@lemmy.world ⁨1⁩ ⁨year⁩ ago

    I’d like to see your script.

    source

    • CarbonatedPastaSauce@lemmy.world ⁨1⁩ ⁨year⁩ ago

      Sorry it took a while, I’m currently on vacation! But I had some time to reread it and sanitize it for public sharing. Here you go:

      `# If you need a web proxy to download blocklists, uncomment the next line and modify the proxy URL

      $WebProxy = “example.notarealproxyserveraddress.com:8080“

      Change this path to the folder you want to store files in during processing, usually the script’s directory

      All downloaded blocklists and final merged files will be stored here before copying to final destination

      $ScriptDir = “C:\Scripts\Merge-Blocklists"

      Path to file containing list of IP Blocklist URLs

      Create this text file with one URL per line for the blocklists you want to download and merge

      $URLfile = $ScriptDir + “blocklist-URLs.txt"

      Path / filenames for the final output files

      $IPOutputFile = $ScriptDir + “iplist.txt” $NetOutputFile = $ScriptDir + “netlist.txt"

      Path to the script log file

      $LogFile = $ScriptDir + “log.txt"

      Create blank log file

      $null | Out-File $LogFile

      Path to merged file the script creates

      $MergedFile = $ScriptDir + “BL_merged-list.txt"

      Create blank merged file

      $null | Out-File $MergedFile

      Regex to validate IPv4 addresses, CIDR ranges, and blocklist URLs

      $IPregex = “^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]).){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$” $CIDRregex = “^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]).){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(/(3[0-2]|[1-2][0-9]|[0-9]))$“

      Function to validate blocklist URLs

      function ValidateURL{ param( [string]$URL )

      $URLcheck = $URL -as [System.URI]
      if($null -ne $URLcheck -and $URL -match "^http(s?)://") {
          return $True
      }
      else{
          return $False
      }
      

      }

      Function to see if an IP is inside a CIDR

      Function CIDRcontainsIP { param( $IPAddress, $CIDR )

      [System.Net.IPAddress]$IP = $IPAddress
      [System.Net.IPAddress]$Subnet = $CIDR.Split("/")[0]
      $C = $CIDR.Split("/")[1]
      [System.Net.IPAddress]$Mask = ''+[ipaddress](4.GB-(4GB-shr$C))
      
      Return ($Subnet.Address -eq ($IP.Address -band $Mask.Address))
      

      }

      Function to write to log and show in console

      Function LogThis { $TimeStamp = Get-Date -format HH:mm:ss $LogData = $TimeStamp + " - " + $log Out-File -FilePath $LogFile -InputObject $LogData -Append Write-Host $log }

      Get the list of IP blocklist URLs to pull from (ignoring lines that don’t start with http)

      try{ $BlockListURLs = Get-Content $URLfile -ErrorAction Stop | Where-Object {$_ -match “^http”} } catch{ $log = “Failed to load blocklist URLs from file $BlockListURLs”; LogThis exit }

      Validate BlockList URLs

      foreach($bURL in $BlockListURLs) { if(!(ValidateURL -URL $bURL)) { $log = “BlockList URL $bURL is invalid”; LogThis $InvalidURLDetected = $true } }

      Exit if any invalid URLs were detected

      if($InvalidURLDetected) { $log = “Invalid URLs were detected in file $URLfile - remove or correct invalid entries and rerun script”; LogThis exit }

      Clean up any pre-existing blocklist files

      $PreviousFiles = Get-ChildItem -Path $ScriptDir -Filter “BL_*.txt” if($PreviousFiles) { foreach($file in $PreviousFiles){ Remove-Item $file } }

      Download the Blocklist files

      foreach($URL in $BlockListURLs) { # Generate a filename from the domain name and target filename - strip the extension and add .txt $BlockListFile = $ScriptDir + “BL_” + $URL.Split(”/”)[2] + “-” + ($URL.Split(”/”)[-1]).Split(”.”)[0] + “.txt”

      # Download the file
      try {
          if($WebProxy) {
              Invoke-WebRequest $URL -Proxy $WebProxy -OutFile $BlockListFile -ErrorAction Stop
          }
          else{
              Invoke-WebRequest $URL -OutFile $BlockListFile -ErrorAction Stop
          }
      }
      catch {
          $log = "Failed to download blocklist file from $URL"; LogThis
      }
      

      }

      Import all the downloaded files and merge into a single file

      $BlockListFiles = Get-ChildItem -Path $ScriptDir -Filter “BL_*.txt”

      foreach($File in $BlockListFiles) { # Special handling for SpamHaus since they comment each line if($File.Name -match ‘spamhaus’) { $FileAppend = Get-Content $File | % {$.split(" ")[0]} | Where-Object {$ -match $IPregex -or $_ -match $CIDRregex} } else{ $FileAppend = Get-Content $File | Where-Object {$_ -match $IPregex -or $_ -match $CIDRregex} } $log = “Adding $($FileAppend | Measure-Object | Select-Object -ExpandProperty count) lines from $($File.Name) to merge file”; LogThis $FileAppend | Out-File $MergedFile -Append }

      Read in the merged file contents so it can be deduplicated

      $MergedList = Get-Content $MergedFile $PreDedupeCount = $MergedList | Measure-Object | Select-Object -ExpandProperty count $MergedList = $MergedList | Select-Object -Unique $PostDedupeCount = $MergedList | Measure-Object | Select-Object -ExpandProperty count $log = “Removed $($PreDedupeCount - $PostDedupeCount) entries via deduplication”; LogThis

      Separate the results into hashtables for IP addresses and CIDR ranges

      $IPList = @{} $CIDRList = @{}

      foreach($val in $MergedList) { if($val -match $CIDRregex){ $CIDRList.Add(”$val”,1) } elseif($val -match $IPregex) { $IPList.Add(”$val",1) } else{ $log = “Merged list value $val does not match IP or CIDR regex”; LogThis } }

      $IPcount = $IPList.GetEnumerator() | Measure-Object | Select-Object -ExpandProperty count $CIDRcount = $CIDRList.GetEnumerator() | Measure-Object | Select-Object -ExpandProperty count

      $log = “Found $IPcount unique IP addresses and $CIDRcount unique CIDR ranges to evaluate”; LogThis

      Build an array from $IPList hashtable so we can modify the hashtable without ending the foreach loop

      $IPListCopy = $IPList.GetEnumerator() | Select-Object -ExpandProperty Name

      Evaluate all the individual IPs to see if they are contained in an existing CIDR

      If they are, set them for removal by making the hashtable value 0

      $ProcessedIPs = 0 foreach($val in $IPListCopy) { foreach($CIDR in $CIDRList.Keys) { if(CIDRcontainsIP -IPAddress $val -CIDR $CIDR){ write-host “IP $val is in CIDR $CIDR” -fore Yellow $IPList.$val = 0 } } $ProcessedIPs++ if(($ProcessedIPs % 100) -eq 0) { Write-Host “Evaluated $ProcessedIPs of $IPcount IP addresses” } }

      $RemovedIPcount = $IPList.GetEnumerator() | Where-Object {$_.Value -eq 0} | Measure-Object | Select-Object -ExpandProperty count $log = “IP Address analysis found $RemovedIPCount IP addresses that were already contained in existing CIDR ranges”; LogThis

      Write the remaining IPs and CIDRs to the final output files

      $FileHeader = “# Last Updated $(Get-Date)” $FileHeader | Out-File $IPOutputFile -Encoding ASCII $FileHeader | Out-File $NetOutputFile -Encoding ASCII $IPList.GetEnumerator() | Where-Object {$.Value -eq 1} | Select-Object -ExpandProperty Name | Sort-Object | Out-File -FilePath $IPOutputFile -Encoding ASCII -Append $CIDRList.GetEnumerator() | Where-Object {$.Value -eq 1} | Select-Object -ExpandProperty Name | Sort-Object | Out-File -FilePath $NetOutputFile -Encoding ASCII -Append

      Change these paths to wherever you want the final files to go

      Copy-Item $IPOutputFile C:\web\blocklist\iplist.ipset -Force Copy-Item $NetOutputFile C:\web\blocklist\netlist.netset -Force `

      source
• KairuByte@lemmy.dbzer0.com ⁨1⁩ ⁨year⁩ ago

  Isnt there a tool developed by the AdGuard team to handle exactly this?

  Just looked through my files, look into this tool, it does exactly what you want: github.com/AdguardTeam/HostlistCompiler

  source

  • BinaryUnit@lemmy.world ⁨1⁩ ⁨year⁩ ago

    Thank you this looks promising

    source
• ShellMonkey@lemmy.socdojo.com ⁨1⁩ ⁨year⁩ ago

  sortmylist.com

  I’ve used this for similar tasks before. Might need a few steps if the formats vary to get them all together but it should be possible.

  source

  • BinaryUnit@lemmy.world ⁨1⁩ ⁨year⁩ ago

    This is very helpful thank you :)

    source
• tordenflesk@lemmy.world ⁨1⁩ ⁨year⁩ ago

  Try an already combined list: oisd.nl

  source

  • AtariDump@lemmy.world ⁨1⁩ ⁨year⁩ ago

    If you’re looking for blocklists, I use /u/Wally3k’s lists as well as the /u/LightSwitch05 “Developer Dan” lists.

    I no longer personally use the OISD lists,- as the maintainer tells you not to use any other lists other than theirs making it difficult to impossible to use the groups feature. Instead, I’ll use a mix of lists and regex blocks. Nor do I recommend the “Quantum Blocklist that’s been going around - here’s why

    I also suggest these regex blocks

    Make sure you read what the different symbols mean with Wally’s blocklists before applying every blocklist. If you stick with the check-marked lists you should find that it blocks ads without too many false positives.

    More blacklisted items doesn’t mean more items blocked; often time adding too many lists will break legitimate websites.

    If you want to, you can reevaluate the added lists after 14-30 days using this tool (not supported by PiHole devs) to audit which lists are actually used. I’ve run this tool and discovered that several lists I added weren’t doing anything at all (If you need help with this tool please use the GitHub page to discuss).

    With the release of v5 memory usage has been reduced when using additional block lists. Also note that with v5 lists are no longer “deduped”.

    source