ISOC England IPv6 Matrix Crawler

[Olivier MJ Crepin-Leblond]

 Hello folks,

whilst at the INET London conference, I have received feedback on a
recent project, whereas some European iXPs have noted slightly higher
than average UDP traffic on their networks. Whilst not 100% sure about
this, I suspect that this might be caused by our IPv6 Matrix Crawler
which we are running pro-bono to track the spread of IPv6 content.

Having officially announced some of its results at the conference which
was hosted in London by the Internet Society on 29th September 2010, I
wanted to announce this here too:

ISOC England was awarded a Community Grants Programme award in November
2009, for the design and implementation of an “IPv6 crawler,” a computer
and its software that would crawl through the DNS at pre-set intervals
in order to detect, for example, IPv6 DNS servers and IPv6 compliant Web
servers, SMTP mailers, and NTP servers.
The search for project partners led to a link-up with Nile University
(Egypt) where professors and their research assistants got to grips with
IPv6, and started the task of writing the software required in the
project, as part of their experience.
In the meantime, the London team of partners built and installed two
servers and a router at Telehouse East, one of the UK's most connected
facilities. One server was set-up as a crawler, and one as a Web
server and information storage, both connected to the Internet backbone
via Dual Stack IPv4/IPv6.

The list of domains tested consisted initially of the world's 1 million
most popular Web sites, as referenced by alexa.com. After a cleaning up
process, about 980 000 domains supported by nearly 5.6 million hosts
were tested periodically.

Results were surprising, and showed how little dual stack IPv4/IPv6 was
supported by major Web sites. Without content, it is no wonder that the
current volume of IPv6 traffic on the Internet is so low! The results
were saved in text files and SQL databases were created for easy
interrogation.
The Web site found on: http://www.ipv6matrix.org/ is an example of a
front end which interrogates those databases, and displays results using
the Google Charts APIs. Ultimately, we plan to document the SQL database
structure, so as to allow for other projects to retrieve data directly
from the databases.

Currently, a full scan of all TLDs in the database takes just over a
month, and results are stored for future analysis. It is hoped that a
growth trend of IPv6 will be seen in a few months. In the long term, the
data collected will enable researchers to find out exactly how major new
technologies, such as the migration to IPv6, spread on the Internet, and
will answer the questions relating to finding the early and late
adopters of new technology.

In the short term, the maps show a snapshot of real world IPv6
penetration data at a glance, and detailed results are also made
available, helped by a system of user-configured filters.

A presentation of the September 2010 data for Europe and Asia was made
at the INET London conference, its deck of slides downloadable from:
http://www.slideshare.net/ocl999/ipv6-matrix-exec-summary

One final note, before you ask: we have received 9 enquiries (out of 
980 000 domains) about Firewalls erroneously detecting a port attack 
of some sort during the scanning process.
The only ports which the Crawler tests connectivity to are:
- Port 25 (SMTP)
- Port 53 (DNS)
- Port 80 (HTTP)
- Port 443 (HTTPS)
- Port 123 (NTP) 

This generates an insignificant amount of traffic in the order of
a few bytes only. Let me make this clear: we are not performing port scans.

At this early alpha (or beta) stage, we would look forward to any
feedback regarding the data we collect, and for any bugs to be pointed
out for the Team to resolve. On the question of accuracy of results, we
have found that results are less accurate in TLD sample sizes under 900
domains. Larger TLD sample sizes increase accuracy. We therefore appeal
to any Registry whose zone data sample size is less than 1000 domains to
contact us, should they wish to provide a wider sample of input domains
for the zone they are responsible for.

Best regards,


Dr. Olivier MJ Crépin-Leblond
Project Leader
www.IPv6matrix.org