If I’m understanding you correctly, you could make use of a shell script for this. Use WGET to download lists, then combine them into a single large file, and finally create a new file with no duplicates by using “awk ‘!visited[$0]++’”

wget
cat *.txt > all.txt (This overwrites all.txt) awk ‘!visited[$0]++’ all.txt > no_duplicates.txt