IPv6 brokenness in Norway, March 2010

[Tore Anderson]

Hi list,

I'm happy to announce that the client loss is at an all-time low:
0.074% over the whole month, and it's been steadily decreasing.  Buggy
versions of Opera as well as Mac OS X appears to be responsible for
about 95% of all measured client loss.

Some noteworthy dates and events:

20100302:  Opera 10.50 released, using getaddrinfo() on Windows.  Gets
           an initial boost thanks to the Microsoft browser ballot.

20100322:  Opera pushes 10.51 to their auto-update function.

20100324:  A-Pressen Interaktiv re-joins the experiment.

20100327:  First day of (extended) Easter holiday for many Norwegians.

20100331:  The Gathering, a large data-party, kicks off.  Provides
           native IPv6 connectivity to all participants.

The numbers show that the release of Opera has helped quite a lot.
Before the release of 10.50, the overall client loss was above 0.090%,
but already right before Easter holiday it's below 0.065%.

Easter holiday shows an interesting effect - both the measured amount of
native IPv6 _and_ client loss drop sharply.  The reason for this is that
there's one particular network here in Norway that is the by far largest
deployment of native IPv6 (about 80-85% of all requests over native IPv6
comes from it), but it is at the same time one of the largest causes of
client loss, because it aggravates the Mac OS X/Opera problems
significantly by filtering 6to4 traffic on ingress for a large number of
end users (that have no native IPv6 service).  The typical user on this
network is also much more likely to take an extended Easter holiday than
the the rest of Norway's population, I believe.

I've been measuring the uptake of Opera 10.50+ on Windows Vista/7 in the
whole period, and it has been steadily increasing even during the
extended Easter holiday.  About 72.5% of the Win Vista/7 Opera users
were using 10.50+ at the end of the month.  So while I do expect the
client loss number to rebound somewhat when Easter holiday is over (on
the 6th of April), I will be very surprised if it stabilises higher than
0.06-0.07%.

The Mac OS X problem appears to occur when a client station has internet
service using NAT/RFC 1918 numbering for IPv4, and 6to4 for IPv6.  The
Apple Airport Extreme SOHO router can operate in this way, and did so by
default up until recently.  In this case, OS X, or more precisely its
getaddinfo() implementation, will prioritise 6to4 over IPv4.  In
addition, GNU libc (therefore all major Linux distributions) behave in
the same way, but I'm not able to measure any impact, probably because
there's so few Linux users to begin with.  The issue is not actually a
bug, but a shortcoming of RFC 3484.  I've elaborated on it in reports to
Apple and the glibc developers here:

http://lists.apple.com/archives/Ipv6-dev/2010/Mar/msg00003.html
http://sourceware.org/bugzilla/show_bug.cgi?id=11438

As an aside it's a bit fun to see how a relatively small deployment of
IPv6 eyeballs like the data party (around 5000 participants I think) can
dramatically change the amount of hits coming in from hosts with native
IPv6.  It more than doubled overnight.

There's two reports attached, one generated from the readers of VG
Multimedia - the online edition of Norways largest newspaper, and also
Norway's largest web site, and the other from the readers of A-Pressen
Interaktiv - an umbrella organisation that publishes the online editions
of about 70 regional newspapers.  When put together, they constitute
Norway's fourth largest web site.

In addition to the overall calculation, I've done five other
calculations so you can see the individual impact of the different
problems I've identified:

- One that excludes hits from buggy versions of Opera on Windows
  Vista/7 (which auto-configures 6to4 and Teredo by default) excluded,
- one that excludes hits from buggy versions of Opera on other versions
  of Windows (where a user must explicitly enable 6to4/Teredo),
- one that excludes all hits from Mac OS X,
- one that excludes all of the above, and
- one that excludes all hits from the two significant end-user networks
  in Norway that filter 6to4 ingress traffic (one of them is the one I
  discussed above).  This calculation does _not_ exclude Opera/OS X;
  when Opera and OS X are excluded, the two networks have about the same
  levels of client loss as rest of the internet.

The seventh column shows approximately how much of the total client loss
would have been avoided (in percentage points and percent), if the
software in question didn't prefer 6to4 over IPv4, or, in the last case,
if the two end-user networks mentioned didn't filter 6to4 on ingress.

Best regards,
-- 
Tore Anderson
Redpill Linpro AS - http://www.redpill-linpro.com/
Tel: +47 21 54 41 27
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: 201003-vg.txt
URL: <https://lists.cluenet.de/pipermail/ipv6-ops/attachments/20100401/0c997afe/attachment.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: 201003-api.txt
URL: <https://lists.cluenet.de/pipermail/ipv6-ops/attachments/20100401/0c997afe/attachment-0001.txt>