IPv6 big packet blackhole in Teleglobe/TATA backbone?

Tue Aug 19 20:22:29 CEST 2008

I and Remi have been trying to figure out an IPv6 big packet blackhole 
that seems to be occurring in Teleglobe network, specifically around 
IP addresses 2001:5a0:200::5 and 2001:5a0:200::10.  I suspect MTU is 
not properly configured.

This can be seen by a big packet ping or "http://www.remlab.net" if 
the traffic seems to hit the link in Teleglobe network.

I've tried <IPNOC at tatacommunications.com>, but they keep asking for 
circuit ID and whether I'm their customer.

Is anyone with Teleglobe/TATA on the list, or know a better contact?

[Btw: interesting thing is that Telia seems to be exposing 1480B MTU 
in at least some parts of their backbone, but at least they're 
generating ICMPv6 packet too bigs and PMTUD has a chance of working..]

---------- Forwarded message ----------
Date: Mon, 18 Aug 2008 18:55:21 +0300
From: Rémi Denis-Courmont <rdenis at simphalempin.com>
To: Pekka Savola <pekkas at netcore.fi>
Subject: Re: remlab.net v6 PMTU problem?

Sorry for the delay, I overlooked this email.

Le mardi 22 juillet 2008 12:45:00 Pekka Savola, vous avez écrit :
>   Maybe I could raise this problem on the ipv6-ops list.
>
>   Could you send the full traceroute/tracepath output from your end, and
>   confirm up to which router ping6 with 1500 sized packets works, and
>   where it fails?

With not too big packets:

% rltraceroute6 -I csc0-ws.ipv6.funet.fi 1000
traceroute to 2001:708:0:f000::7100:2 (2001:708:0:f000::7100:2) from 
2001:41d0:1:a0d6::401:1983, 30 hops max, 1000
byte packets
    1  * 2001:41d0:1:a0ff:ff:ff:ff:ff (2001:41d0:1:a0ff:ff:ff:ff:ff)  1.717 ms
    *
    2  2001:41d0::592 (2001:41d0::592)  6.330 ms  * 5.331 ms
    3  2001:5a0:1900::9 (2001:5a0:1900::9)  4.869 ms  4.754 ms  4.686 ms
    4  2001:5a0:1900::2 (2001:5a0:1900::2)  4.669 ms  4.576 ms  4.675 ms
    5  2001:5a0:0:100::a (2001:5a0:0:100::a)  10.508 ms  10.328 ms  10.316 ms
    6  if-6-0.core1.b1d-brussels.ipv6.teleglobe.net (2001:5a0:1400::1)  10.553
    ms
    10.451 ms  10.400 ms
    7  if-6-0.core2.ad1-amsterdam.ipv6.teleglobe.net (2001:5a0:200:200::1)
    216.493 ms  94.318 ms  49.794 ms
    8  if-0-0.core1.ad1-amsterdam.ipv6.teleglobe.net (2001:5a0:200:100::1)
    18.189 ms  18.332 ms  18.095 ms
    9  2001:5a0:200::5 (2001:5a0:200::5)  18.232 ms  18.247 ms  18.279 ms
10  2001:5a0:200::10 (2001:5a0:200::10)  19.794 ms  19.355 ms  19.623 ms
11  ldn-ipv6-b1-link.ipv6.telia.net (2001:2000:3010:1::1)  29.447 ms  29.534 ms 
29.316 ms
12  ldn-ipv6-b1-link.ipv6.telia.net (2001:2000:3010::1)  60.893 ms  61.063 ms 
60.760 ms
13  2001:2000:3080:d::2 (2001:2000:3080:d::2)  45.984 ms  45.987 ms  45.714 ms
14  helsinki0-rtr.funet.fi (2001:948:0:f035::2)  56.706 ms  56.023 ms  56.185 
ms
15  csc0-x0000-helsinki0.ipv6.funet.fi (2001:708:0:f000::1:2)  52.700 ms 52.646 
ms  52.587 ms
16  csc3-g0000-csc0.ipv6.funet.fi (2001:708:0:f000::30:2)  53.087 ms  53.001 ms 
52.810 ms
17  2001:708:0:f000::7100:2 (2001:708:0:f000::7100:2)  52.578 ms  56.028 ms 
56.005 ms

With MTU packets:
% rltraceroute6 -I csc0-ws.ipv6.funet.fi 1460
traceroute to 2001:708:0:f000::7100:2 (2001:708:0:f000::7100:2) from 
2001:41d0:1:a0d6::401:1983, 30 hops max, 1460
byte packets
    1  2001:41d0:1:a0ff:ff:ff:ff:ff (2001:41d0:1:a0ff:ff:ff:ff:ff)  2.004 ms  *
    2.275 ms
    2  2001:41d0::592 (2001:41d0::592)  5.445 ms  * 11.539 ms
    3  2001:5a0:1900::9 (2001:5a0:1900::9)  5.044 ms  4.888 ms  5.176 ms
    4  2001:5a0:1900::2 (2001:5a0:1900::2)  4.764 ms  4.710 ms  4.745 ms
    5  2001:5a0:0:100::a (2001:5a0:0:100::a)  10.437 ms  * 10.568 ms
    6  if-6-0.core1.b1d-brussels.ipv6.teleglobe.net (2001:5a0:1400::1)  10.563
    ms
    * 10.720 ms
    7  if-6-0.core2.ad1-amsterdam.ipv6.teleglobe.net (2001:5a0:200:200::1)
    18.265 ms  * 18.143 ms
    8  if-0-0.core1.ad1-amsterdam.ipv6.teleglobe.net (2001:5a0:200:100::1)
    18.783 ms  * 18.122 ms
    9  2001:5a0:200::5 (2001:5a0:200::5)  186.237 ms  215.084 ms  212.344 ms
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *

As for PMTUD:

% tracepath6 csc0-ws.ipv6.funet.fi  -n
    1?: [LOCALHOST]                      pmtu 1500
    1:  2001:41d0:1:a0ff:ff:ff:ff:ff       4.198ms
    2:  2001:41d0::592                     5.  8ms
    3:  2001:5a0:1900::9                   4.995ms
    4:  2001:5a0:1900::2                   4.733ms
    5:  2001:5a0:0:100::a                 10.408ms
    6:  2001:5a0:1400::1                  10.532ms
    7:  2001:5a0:200:200::1               18.147ms
    8:  2001:5a0:200:100::1               18.193ms
    9:  2001:5a0:200::5                   18.281ms
10:  no reply
11:  no reply
12:  no reply
13:  no reply
14:  no reply
15:  no reply
16:  no reply
17:  no reply
18:  no reply

I got an interesting response here:

% ping6 -n csc0-ws.ipv6.funet.fi -s 1436
PING csc0-ws.ipv6.funet.fi(2001:708:0:f000::7100:2) 1436 data bytes
>From 2001:2000:3010::1 icmp_seq=1 Packet too big: mtu=1480
1444 bytes from 2001:708:0:f000::7100:2: icmp_seq=2 ttl=54 time=58.3 ms
1444 bytes from 2001:708:0:f000::7100:2: icmp_seq=3 ttl=54 time=59.7 ms

--- csc0-ws.ipv6.funet.fi ping statistics ---
3 packets transmitted, 2 received, +1 errors, 33% packet loss, time 2001ms
rtt min/avg/max/mdev = 58.321/59.019/59.717/0.698 ms

After lots of pings (and resets of PMTU cache), it seems that:
- packets over 40+8+1438 bytes get lost,
- packets over 40+8+1432 bytes get a Too Big error from Telia,
- other packets go through

-- 
Rémi Denis-Courmont
http://www.remlab.net/