Strange source port filtering by Tinet

Tue Jul 20 10:51:24 CEST 2010

I've noticed a very peculiar problem with IPv6 traffic transitting
through AS3257.  They did respond to a request to find the problem,
but a week has passed and I suspect that they are not taking me very
seriously.  I believe that this problem affects all traffic, so I'd be
interested if some of you could verify this.  It's a bit complicated,
but please bear with me.  Here's the story.

Netnod has recently started to do IPv6 on their anycast DNS
infrastructure.  So far, only the sites in Stockholm and London are
announcing their prefixes.  They announce less-specifics from
Stockholm, so most of the traffic is currently going to LINX.  The
I-root server is covered by 2001:7fe::/33 (yes, that's /33).  They
also announce 2001:67c:1010::/48, which they use to number the name
servers for their TLD anycast service, like the one for CH called
ch1.dnsnode.net (2001:67c:1010:2::53).

The I-root prefix is announced from AS29216, while the other prefix is
announced by the Netnod AS 8674 itself.  At this time, the site in
London is behind the LINX AS 5459.  As such, the only IPv6 transit is
provided by Tinet, AS3257.  Here's a trace from here (source
2001:620:0:113:21b:78ff:fe30:297e)

traceroute to ch1.dnsnode.net (2001:67c:1010:2::53), 30 hops max, 40 byte packets
 1  swiCE3-V300.switch.ch (2001:620:0:113::1)  0.373 ms
 2  swiZH2-10GE-1-1.switch.ch (2001:620:0:c027::2)  4.125 ms
 3  swiIX1-10GE-1-3.switch.ch (2001:620:0:c015::1)  4.640 ms
 4  swiIX2-10GE-4-4.switch.ch (2001:620:0:c008::2)  4.146 ms
 5  swissix-glb.init7.net (2001:7f8:24::7)  4.152 ms
 6  r1fra1.core.init7.net (2001:1620:2::6)  10.792 ms
 7  ge-4-0-4-295.fra21.ip6.tinet.net (2001:668:0:3::2000:111)  11.406 ms
 8  xe-7-3-0.lon20.ip6.tinet.net (2001:668:0:2::1:1242)  24.745 ms
 9  xe-0-3-0.lon21.ip6.tinet.net (2001:668:0:2::1:1ae2)  24.805 ms
10  xe-5-3-0.lon10.ip6.tinet.net (2001:668:0:2::1:1642)  21.746 ms
11  g0-0-123.tr2.tfm7.thn.linx.net (2001:668:0:3::4000:82)  23.284 ms
12  2a01:40:1003:2::3 (2a01:40:1003:2::3)  23.246 ms
13  ch1.dnsnode.net (2001:67c:1010:2::53)  23.289 ms

I've set up SmokePing DNS and ping probes for I root as well as
ch1.dnsnode.net here:

http://lg.net.switch.ch/cgi-bin/smokeping.cgi?target=Services.CHLIDNS.AutonomicaIRoot
http://lg.net.switch.ch/cgi-bin/smokeping.cgi?target=Services.CHLIDNS.AutonomicaIRootPingv6
http://lg.net.switch.ch/cgi-bin/smokeping.cgi?target=Services.CHLIDNS.Autonomicav6
http://lg.net.switch.ch/cgi-bin/smokeping.cgi?target=Services.CHLIDNS.AutonomicaPingv6

Two things are immediately obvious

- ping RTTs are very stable and lossless

- DNS probes have high jitter (3ms) and loss

The jitter is most likely due to equal-cost multipathing within AS3257
(which affects TCP and UDP flows but all ICMP traffic is on a single
path).  This is not pretty but doesn't worry me too much.

When I tried to find the reason for the packet loss, I discovered that
all UDP and TCP packets with source ports in the range 35072 through
35327 are dropped!  This can be easily tested with dig, e.g.

: gall at atitlan[gall]; dig -b2001:620:0:114:21b:78ff:fe30:2974#35071 @ch1.dnsnode.net ch. soa +short +timeout=1
a.nic.ch. helpdesk.nic.ch. 2010072010 3600 900 2592000 3600
: gall at atitlan[gall]; dig -b2001:620:0:114:21b:78ff:fe30:2974#35072 @ch1.dnsnode.net ch. soa +short +timeout=1

; <<>> DiG 9.7.0-P1 <<>> -b2001:620:0:114:21b:78ff:fe30:2974#35072 @ch1.dnsnode.net ch. soa +short +timeout=1
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

With a traceroute that allows to fix the source port, the last hop I
see is the one preceeding the first Tinet router at DE-CIX

: gall at atitlan[gall]; traceroute6 -s 2001:620:0:113:21b:78ff:fe30:297e --sport=35072 -q1 -N1 -w1 ch1.dnsnode.net
traceroute to ch1.dnsnode.net (2001:67c:1010:2::53), 30 hops max, 40 byte packets
 1  swiCE3-V300.switch.ch (2001:620:0:113::1)  0.301 ms
 2  swiZH2-10GE-1-1.switch.ch (2001:620:0:c027::2)  4.144 ms
 3  swiIX1-10GE-1-3.switch.ch (2001:620:0:c015::1)  4.312 ms
 4  swiIX2-10GE-4-4.switch.ch (2001:620:0:c008::2)  4.178 ms
 5  swissix-glb.init7.net (2001:7f8:24::7)  4.151 ms
 6  r1fra1.core.init7.net (2001:1620:2::6)  18.958 ms
 7  *
 8  *
 9  *^C

I checked with the people at Netnod that the packets are lost on the
forward path.

I actually scanned all ports from 1024 through 65535 with dig and it's
exactly those 256 ports that share this fate.  Note that the port
range in hex is 0x8900 through 0x89ff and 0x89 is 137 in decimal.  A
wild speculation is that a filter for NetBIOS at the Tinet border
could be buggy.

I can reproduce this from our own AS559 as well as from a host in
AS2914.  To support my theory, I'd be grateful if some of you could
run these dig and traceroute tests to see if you can reproduce this.

Regards,
Alex