IPv6 site snapshot

[Jared Mauch]

[last update, i really need to attend to family items, see below]

On Apr 26, 2011, at 7:26 PM, Martin Millnert wrote:

> Jared,
> 
> On Tue, 2011-04-26 at 17:28 -0400, Jared Mauch wrote:
>>> On 26/04/2011 20:28, Jared Mauch wrote:
>>>> So, with the help of Dan Wing, I ran a list of 1 million domains names through dns queries (primarily) and am chasing down a few people that caused trouble with bind.  There's a number of domain names that return SERVFAIL but don't when the google/comcast nameservers are queried.  Not sure if anyone has seen that behavior and tracked it down yet.
>>>> 
>>>> I've posted the data here as well:
>>>> 
>>>> http://puck.nether.net/~jared/aaaa/
> 
> dns_names.txt contains URLs and thus some overlap of DNS names.
> The following purifies:
> 
> anticimex at shakira:/dev/shm$ wget -q
> http://puck.nether.net/~jared/aaaa/dns_names.txt.gz
> anticimex at shakira:/dev/shm$ gunzip dns_names.txt.gz
> anticimex at shakira:/dev/shm$ wc -l dns_names.txt
> 1000000 dns_names.txt
> anticimex at shakira:/dev/shm$ sed -i 's/\/.*//' dns_names.txt | sort -u >
> dns_names_clean_sorted.txt
> anticimex at shakira:/dev/shm$ wc -l dns_names_clean_sorted.txt
> 991944 dns_names_clean_sorted.txt

Yeah, I noticed this as well and cleaned up some of the trailing data.  I've not re-run it yet, but hope to soon.


>> I'm actually surprised how many people have pulled these down.  I did compress them as well.  I'm going to re-run it later and try to extract some more details regarding the dns failures.  I wish I could more easily do a massive 'dig' dump of these in parallel.
>> 
>> Don't exactly want it to take a full day ...
>> 
> 
> I've attached a *very* stupid 10 min parallel bash hack that may help
> you out a little.
> The job control leaves much to be desired, but you *should* be able to
> launch a few 10k digs in parallel on a decent machine this way which
> should help you out through the job a bit quicker.
> (And before you complain about the way it behaves, I did say it was
> stupid... :p )

Oh, I get it.  I almost did some wacky xargs stuff.  Dan was surprised how I abused his application as well to max out around 700 parallel threads banging on our resolvers, and our dns team didn't scream but did notice the increased load.  I am going to see if I can do something better tomorrow.

ISC has also opened a tracking bug at minimum, and what may turn out to be a real defect depending on how this research goes.  I've also communicated with one CDN today about how their dns system works and it seems likely that there is a bug in BIND, but don't quote me on that "yet".

I do have a dig running against the full list and will leave that running overnight and see how far that gets.  I'm glad there is some interest in this, and hope to report more results sometime tomorrow.  It's processed 29k hosts of the ~1000k hosts in 2 hours.  Not that promising for a quick result, but maybe the dns servers will become less loaded overnight..

- Jared