Routing problems after multiple peering

[Olivier MJ Crepin-Leblond]

Hello,

A few months ago, our upstream provider used to peer only with one
company but ever since they have increased this to two, we're faced with
sporadic transient routing problems. Since we're supporting the IPv6
Matrix project which is used to track the spread of IPv6 connectivity,
we need stability, but this, for some reason isn't the case at the moment.
I hope that someone here can make sense of the problem.

We're connected to a router on AS31493. Details of our feed's peering
arrangements on: http://bgp.he.net/AS31493

When tracerouting (from outside) to two machines on our same subnet, one
works whilst the other one doesn't.

psg.com:/usr/home/ocl> traceroute6 www.ipv6matrix.org
traceroute6 to elephant.ipv6matrix.org (2a00:19e8:20:1::aa) from
2001:418:1::62, 64 hops max, 12 byte packets
 1  psg1.psg.com  0.353 ms  0.241 ms  0.212 ms
 2  v6-10gigabitethernet1-3.core1.sea1.he.net  0.342 ms  0.365 ms  0.715 ms
 3  10gigabitethernet3-1.core1.den1.he.net  31.981 ms  32.174 ms  32.061 ms
 4  10gigabitethernet1-1.core1.chi1.he.net  64.403 ms  60.982 ms  63.565 ms
 5  10gigabitethernet2-4.core1.nyc4.he.net  78.278 ms  85.740 ms  78.280 ms
 6  10gigabitethernet3-3.core1.lon1.he.net  151.253 ms  156.925 ms 
146.604 ms
 7  2001:7f8:4::23e7:1  146.366 ms  146.848 ms  146.412 ms
 8  2a01:178:9191:3::17  145.689 ms  146.510 ms  145.748 ms
 9  2a01:178:500::6  146.903 ms  147.354 ms  146.364 ms
10  2a00:19e8::1:d  147.731 ms  146.940 ms  147.751 ms
11  2a00:19e8::1:5  146.428 ms  147.072 ms  148.503 ms
12  elephant.ipv6matrix.org  146.880 ms  147.313 ms  146.752 ms

psg.com:/usr/home/ocl> traceroute6 turtle.ipv6matrix.org
traceroute6 to crawler.ipv6matrix.org (2a00:19e8:20:1::a2) from
2001:418:1::62, 64 hops max, 12 byte packets
 1  psg1.psg.com  0.345 ms  0.281 ms  0.221 ms
 2  v6-10gigabitethernet1-3.core1.sea1.he.net  0.430 ms  0.279 ms  0.338 ms
 3  10gigabitethernet3-1.core1.den1.he.net  32.112 ms  32.016 ms  32.079 ms
 4  10gigabitethernet1-1.core1.chi1.he.net  55.192 ms  55.770 ms  55.174 ms
 5  10gigabitethernet2-4.core1.nyc4.he.net  78.302 ms  77.632 ms  88.778 ms
 6  10gigabitethernet3-3.core1.lon1.he.net  154.363 ms  149.384 ms 
150.024 ms
 7  2001:7f8:4::23e7:1  146.460 ms  146.942 ms  146.488 ms
 8  2a01:178:9191:3::200  145.632 ms  146.211 ms  145.509 ms
 9  2a01:178:9191:3::1  146.608 ms  146.405 ms  146.006 ms
10  2a01:178:9191:3::200  146.363 ms  145.682 ms  146.127 ms
11  2a01:178:9191:3::1  145.992 ms  146.555 ms  145.989 ms
12  2a01:178:9191:3::200  145.770 ms  146.529 ms  145.866 ms
13  2a01:178:9191:3::1  146.618 ms  146.188 ms  146.504 ms
14  2a01:178:9191:3::200  145.985 ms  146.581 ms  146.493 ms

and a few hours later:

psg.com:/usr/home/ocl> traceroute6 turtle.ipv6matrix.org
traceroute6 to crawler.ipv6matrix.org (2a00:19e8:20:1::a2) from
2001:418:1::62, 64 hops max, 12 byte packets
 1  psg1.psg.com  0.240 ms  0.322 ms  0.334 ms
 2  v6-10gigabitethernet1-3.core1.sea1.he.net  0.343 ms  0.369 ms  0.321 ms
 3  10gigabitethernet3-1.core1.den1.he.net  32.104 ms  32.108 ms  32.187 ms
 4  10gigabitethernet1-1.core1.chi1.he.net  55.812 ms  62.153 ms  55.693 ms
 5  10gigabitethernet2-4.core1.nyc4.he.net  77.759 ms  78.167 ms  77.676 ms
 6  10gigabitethernet3-3.core1.lon1.he.net  146.241 ms  145.742 ms 
148.105 ms
 7  2001:7f8:4::23e7:1  146.753 ms  146.084 ms  146.749 ms
 8  2001:7f8:4::23e7:1  146.788 ms !A  146.271 ms !A  146.753 ms !A
psg.com:/usr/home/ocl>

(and this is the case for both turtle & www.ipv6matrix.org)
And a few hours later, things will work sporadically again.

As seen from the the IPv6Matrix side (where we're connected):

[ocl at crawler ~]$ traceroute6 www.kame.net
traceroute to www.kame.net (2001:200:dff:fff1:216:3eff:feb1:44d7), 30
hops max, 40 byte packets
 1  isoc-eng-v6gw1.ipv6matrix.org (2a00:19e8:20:1::1)  1.112 ms  1.611
ms  1.999 ms
 2  * * 2001:a88:0:2::39 (2001:a88:0:2::39)  1.258 ms
 3  * * *
 4  * * *
 5  ldn-ipv6-b1-link.ipv6.telia.net (2001:2000:3080::1)  29.856 ms 
30.007 ms  30.306 ms
 6  ffm-b12-v6.telia.net (2001:2000:3018:4a::1)  54.363 ms  55.640 ms 
55.672 ms
 7  * * *
 8  as-1.r23.amstnl02.nl.bb.gin.ntt.net (2001:728:0:2000::13d)  37.223
ms * as-1.r23.amstnl02.nl.bb.gin.ntt.net (2001:728:0:2000::13d)  36.956 ms
 9  as-1.r23.amstnl02.nl.bb.gin.ntt.net (2001:728:0:2000::13d)  42.785
ms * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *


Please be so kind to advise how to fix this. Looks like route flapping,
or something else?

Kind regards,

Olivier

-- 
Olivier MJ Crépin-Leblond, PhD
http://www.gih.com/ocl.html