IPv6 site snapshot

[Martin Millnert]

Jared,

On Tue, 2011-04-26 at 17:28 -0400, Jared Mauch wrote:
> > On 26/04/2011 20:28, Jared Mauch wrote:
> >> So, with the help of Dan Wing, I ran a list of 1 million domains names through dns queries (primarily) and am chasing down a few people that caused trouble with bind.  There's a number of domain names that return SERVFAIL but don't when the google/comcast nameservers are queried.  Not sure if anyone has seen that behavior and tracked it down yet.
> >> 
> >> I've posted the data here as well:
> >> 
> >> http://puck.nether.net/~jared/aaaa/

dns_names.txt contains URLs and thus some overlap of DNS names.
The following purifies:

anticimex at shakira:/dev/shm$ wget -q
http://puck.nether.net/~jared/aaaa/dns_names.txt.gz
anticimex at shakira:/dev/shm$ gunzip dns_names.txt.gz
anticimex at shakira:/dev/shm$ wc -l dns_names.txt
1000000 dns_names.txt
anticimex at shakira:/dev/shm$ sed -i 's/\/.*//' dns_names.txt | sort -u >
dns_names_clean_sorted.txt
anticimex at shakira:/dev/shm$ wc -l dns_names_clean_sorted.txt
991944 dns_names_clean_sorted.txt

> I'm actually surprised how many people have pulled these down.  I did compress them as well.  I'm going to re-run it later and try to extract some more details regarding the dns failures.  I wish I could more easily do a massive 'dig' dump of these in parallel.
> 
> Don't exactly want it to take a full day ...
> 

I've attached a *very* stupid 10 min parallel bash hack that may help
you out a little.
The job control leaves much to be desired, but you *should* be able to
launch a few 10k digs in parallel on a decent machine this way which
should help you out through the job a bit quicker.
(And before you complain about the way it behaves, I did say it was
stupid... :p )

HTH,
Martin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: parallel_records.sh
Type: application/x-shellscript
Size: 1310 bytes
Desc: not available
Url : http://lists.cluenet.de/pipermail/ipv6-ops/attachments/20110426/ee4c630a/attachment.bin