Comment on AdGuard/PiHole Blocklists merge duplicates

CarbonatedPastaSauce@lemmy.world ⁨7⁩ ⁨months⁩ ago

• nyar@lemmy.world ⁨7⁩ ⁨months⁩ ago

  I’d like to see your script.

  source

  • CarbonatedPastaSauce@lemmy.world ⁨7⁩ ⁨months⁩ ago

    Sorry it took a while, I’m currently on vacation! But I had some time to reread it and sanitize it for public sharing. Here you go:

    `# If you need a web proxy to download blocklists, uncomment the next line and modify the proxy URL

    $WebProxy = “example.notarealproxyserveraddress.com:8080“

    Change this path to the folder you want to store files in during processing, usually the script’s directory

    All downloaded blocklists and final merged files will be stored here before copying to final destination

    $ScriptDir = “C:\Scripts\Merge-Blocklists"

    Path to file containing list of IP Blocklist URLs

    Create this text file with one URL per line for the blocklists you want to download and merge

    $URLfile = $ScriptDir + “blocklist-URLs.txt"

    Path / filenames for the final output files

    $IPOutputFile = $ScriptDir + “iplist.txt” $NetOutputFile = $ScriptDir + “netlist.txt"

    Path to the script log file

    $LogFile = $ScriptDir + “log.txt"

    Create blank log file

    $null | Out-File $LogFile

    Path to merged file the script creates

    $MergedFile = $ScriptDir + “BL_merged-list.txt"

    Create blank merged file

    $null | Out-File $MergedFile

    Regex to validate IPv4 addresses, CIDR ranges, and blocklist URLs

    $IPregex = “^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]).){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$” $CIDRregex = “^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]).){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(/(3[0-2]|[1-2][0-9]|[0-9]))$“

    Function to validate blocklist URLs

    function ValidateURL{ param( [string]$URL )

    $URLcheck = $URL -as [System.URI]
    if($null -ne $URLcheck -and $URL -match "^http(s?)://") {
        return $True
    }
    else{
        return $False
    }
    

    }

    Function to see if an IP is inside a CIDR

    Function CIDRcontainsIP { param( $IPAddress, $CIDR )

    [System.Net.IPAddress]$IP = $IPAddress
    [System.Net.IPAddress]$Subnet = $CIDR.Split("/")[0]
    $C = $CIDR.Split("/")[1]
    [System.Net.IPAddress]$Mask = ''+[ipaddress](4.GB-(4GB-shr$C))
    
    Return ($Subnet.Address -eq ($IP.Address -band $Mask.Address))
    

    }

    Function to write to log and show in console

    Function LogThis { $TimeStamp = Get-Date -format HH:mm:ss $LogData = $TimeStamp + " - " + $log Out-File -FilePath $LogFile -InputObject $LogData -Append Write-Host $log }

    Get the list of IP blocklist URLs to pull from (ignoring lines that don’t start with http)

    try{ $BlockListURLs = Get-Content $URLfile -ErrorAction Stop | Where-Object {$_ -match “^http”} } catch{ $log = “Failed to load blocklist URLs from file $BlockListURLs”; LogThis exit }

    Validate BlockList URLs

    foreach($bURL in $BlockListURLs) { if(!(ValidateURL -URL $bURL)) { $log = “BlockList URL $bURL is invalid”; LogThis $InvalidURLDetected = $true } }

    Exit if any invalid URLs were detected

    if($InvalidURLDetected) { $log = “Invalid URLs were detected in file $URLfile - remove or correct invalid entries and rerun script”; LogThis exit }

    Clean up any pre-existing blocklist files

    $PreviousFiles = Get-ChildItem -Path $ScriptDir -Filter “BL_*.txt” if($PreviousFiles) { foreach($file in $PreviousFiles){ Remove-Item $file } }

    Download the Blocklist files

    foreach($URL in $BlockListURLs) { # Generate a filename from the domain name and target filename - strip the extension and add .txt $BlockListFile = $ScriptDir + “BL_” + $URL.Split(”/”)[2] + “-” + ($URL.Split(”/”)[-1]).Split(”.”)[0] + “.txt”

    # Download the file
    try {
        if($WebProxy) {
            Invoke-WebRequest $URL -Proxy $WebProxy -OutFile $BlockListFile -ErrorAction Stop
        }
        else{
            Invoke-WebRequest $URL -OutFile $BlockListFile -ErrorAction Stop
        }
    }
    catch {
        $log = "Failed to download blocklist file from $URL"; LogThis
    }
    

    }

    Import all the downloaded files and merge into a single file

    $BlockListFiles = Get-ChildItem -Path $ScriptDir -Filter “BL_*.txt”

    foreach($File in $BlockListFiles) { # Special handling for SpamHaus since they comment each line if($File.Name -match ‘spamhaus’) { $FileAppend = Get-Content $File | % {$.split(" ")[0]} | Where-Object {$ -match $IPregex -or $_ -match $CIDRregex} } else{ $FileAppend = Get-Content $File | Where-Object {$_ -match $IPregex -or $_ -match $CIDRregex} } $log = “Adding $($FileAppend | Measure-Object | Select-Object -ExpandProperty count) lines from $($File.Name) to merge file”; LogThis $FileAppend | Out-File $MergedFile -Append }

    Read in the merged file contents so it can be deduplicated

    $MergedList = Get-Content $MergedFile $PreDedupeCount = $MergedList | Measure-Object | Select-Object -ExpandProperty count $MergedList = $MergedList | Select-Object -Unique $PostDedupeCount = $MergedList | Measure-Object | Select-Object -ExpandProperty count $log = “Removed $($PreDedupeCount - $PostDedupeCount) entries via deduplication”; LogThis

    Separate the results into hashtables for IP addresses and CIDR ranges

    $IPList = @{} $CIDRList = @{}

    foreach($val in $MergedList) { if($val -match $CIDRregex){ $CIDRList.Add(”$val”,1) } elseif($val -match $IPregex) { $IPList.Add(”$val",1) } else{ $log = “Merged list value $val does not match IP or CIDR regex”; LogThis } }

    $IPcount = $IPList.GetEnumerator() | Measure-Object | Select-Object -ExpandProperty count $CIDRcount = $CIDRList.GetEnumerator() | Measure-Object | Select-Object -ExpandProperty count

    $log = “Found $IPcount unique IP addresses and $CIDRcount unique CIDR ranges to evaluate”; LogThis

    Build an array from $IPList hashtable so we can modify the hashtable without ending the foreach loop

    $IPListCopy = $IPList.GetEnumerator() | Select-Object -ExpandProperty Name

    Evaluate all the individual IPs to see if they are contained in an existing CIDR

    If they are, set them for removal by making the hashtable value 0

    $ProcessedIPs = 0 foreach($val in $IPListCopy) { foreach($CIDR in $CIDRList.Keys) { if(CIDRcontainsIP -IPAddress $val -CIDR $CIDR){ write-host “IP $val is in CIDR $CIDR” -fore Yellow $IPList.$val = 0 } } $ProcessedIPs++ if(($ProcessedIPs % 100) -eq 0) { Write-Host “Evaluated $ProcessedIPs of $IPcount IP addresses” } }

    $RemovedIPcount = $IPList.GetEnumerator() | Where-Object {$_.Value -eq 0} | Measure-Object | Select-Object -ExpandProperty count $log = “IP Address analysis found $RemovedIPCount IP addresses that were already contained in existing CIDR ranges”; LogThis

    Write the remaining IPs and CIDRs to the final output files

    $FileHeader = “# Last Updated $(Get-Date)” $FileHeader | Out-File $IPOutputFile -Encoding ASCII $FileHeader | Out-File $NetOutputFile -Encoding ASCII $IPList.GetEnumerator() | Where-Object {$.Value -eq 1} | Select-Object -ExpandProperty Name | Sort-Object | Out-File -FilePath $IPOutputFile -Encoding ASCII -Append $CIDRList.GetEnumerator() | Where-Object {$.Value -eq 1} | Select-Object -ExpandProperty Name | Sort-Object | Out-File -FilePath $NetOutputFile -Encoding ASCII -Append

    Change these paths to wherever you want the final files to go

    Copy-Item $IPOutputFile C:\web\blocklist\iplist.ipset -Force Copy-Item $NetOutputFile C:\web\blocklist\netlist.netset -Force `

    source