Some very nice broken IPv6 networks at Google and Akamai (Was: Some very nice IPv6 growth as measured by Google)

[Tore Anderson]

* Jeroen Massar

> Also note that the Akamai problem (which still persists) is a random
> one. Hence fetching one URL is just a pure luck thing if it works or
> not. As a generic page has multiple objects though, you'll hit it much
> quicker.

Hm. As I've said before - WFM. Any more information you could provide
to help me try to reproduce it?

> I am not 'insisting' that there is no problem with PMTUD.

«No, PMTUD is fine in both IPv4 and IPv6», you said...

> I am stating that the problem has to be fixed at the source, not
> hidden in the network.

In an ideal world, perhaps. It's like with 6to4; if all relay operators
did a wonderful job, and no-one filtered proto-41, and nobody did
NAT44, then 6to4 would just be hunky-dory. But it's just too much
brokenness out there.

Same with PMTUD. It's beyond repair, IMHO. The pragmatic thing is to
accept that and move on.

> > Looking at it from the content side, users using IPv6 tunnels are
> > in a tiny, tiny minority, while still managing to be responsible
> > for a majority of trouble reports.
> 
> Maybe as those users are more technically experienced and are able to
> get their message out, while non-techie users just disable IPv6 as is
> advised in a LOT of places? :)

Could be. While I can't know for certain, my gut feeling tells me the
otherwise. When we've investigated, the most common reason for breakage
has been that the tunnel user is just technically experienced enough to
shoot himself in the foot. Or that the tunnel ingress routers rate-limit
ICMPv6 error generation.

> You are forgetting the little fact that "native" is a really strange
> word. Quite a few DSL deployments use PPPoE etc.
> 
> There are also a lot of "native" deployments out there that use 6rd.

In my experience, these ISPs deploy workarounds to avoid PMTUD. TCP MSS
clamping, and LAN RA MTUs (for IPv6). That helps.

> Instead of just coming with "TUNNELS SUCK!!!!@&$!@#&$%^!*@%!" actually
> Contact the networks that are broken and try to get them to fix the
> problem. You might not want to fix those as it is not your problem,
> but it is a problem for access networks.

I think PMTUD on the internet is broken beyond salvation, honestly, and
with it tunnels (that do not also use workarounds). Trying to fix it by
identifying and contacting all the networks that break it is a fool's
errand. It's just like with 6to4.

YMMV, of course, and I wish you the best of luck if you'd want to try
it. But forgive me for not holding my breath.

> Note btw that Google is not stating anything about the problem they
> had. And Akamai, well, they are still digging.
> 
> Thus PMTUD might be an issue, might also be something else completely.
> 
> Without insight into those systems, one just has to guess.

The available data strongly suggests that PMTUD is at fault IMHO,
because numerous users (on the forums you linked to and on nanog/outages)
report that clamping MSS/MTU values helps. Also there's the fact that
the users reporting problems seems to be mostly behind tunnels.

Note that Google normally clamps the IPv6 MSS in their end. So maybe
that broke, which would have revealed previously hidden PMTUD breakage.
If suddenly all Google traffic needs PMTUD to get into the tunnel, then
it would likely trip up any ICMP ratelimiter at the tunnel ingress.
Well, it's a theory at least.

But of course, there could very well be additional issues in addition to
PMTUD here.

> > LAN RA MTU, yes. TCP MSS, no - it can be done in the ISP's tunnel
> > router.
> 
> Do you really suggest making the Internet have an MTU of 1280? :)

Uh, no? What makes you think that?

> > For what it's worth, the vast majority of tunneled IPv6 traffic we
> > see comes from ISPs with 6RD, which generally works fine due to
> > these workarounds. Thankfully.
> 
> Till people start using non-TCP protocols, and everything breaks.

Which is precisely why I was wondering about QUIC...

> Hence, don't hide the fact, instead fix it.

> > With MTU=1500 links, PMTUD isn't necessary, so it must be some other
> > root cause.
> 
> Of course PMTUD is needed. Not all links on the Internet are 1280.

PMTUD is not needed if all links in the path is 1500 or larger. (Unless
one of the endpoints are configured with jumbo support, but that is
extremely rare for regular clients or internet-facing servers.)

1280 is a meaningless number in this context. <=1499 is the triggering
factor, because practically all nodes on the internet will assume an
initial MTU of 1500*. If all the links in the path can support that,
then great, there is zero need for PMTUD. Only when you encounter a link
with MTU <=1499 does the need for PMTUD arise.

* Except e.g. clients beind 6RD with reduced LAN RA MTU.

> Heck, there are even some "transit" providers that use tunnels and
> thus have <1500 MTU

I agree with your use of double quotes. :-)

Tore