Dear Akamai, you got a /32 there not a bunch of /48s - how to break Facebook and annoy lots of users

[Jeroen Massar]

On 2012-08-20 21:44, Patrick W. Gilmore wrote:
> Hi everyone.  How's it hangin'?  Seems like we have once again
> stirred up a lot of dust unintentionally.

Well nope, I did. I should have spammed you and/or the noc directly
instead ;)

> I'm going to try to clear
> up a few things here, so please pardon the length of this response.
> Feel free to let me know if anything was not clear.

Thanks for this!

> First, some housekeeping: Akamai should have route6 objects for all
> our announcements.  I'll have someone get on that.  Mea Culpa, please
> forgive any inconvenience.  (I thought we already cleaned that up
> after the last thread, so this is especially embarrassing.)

Shit happens. I am sure it will be resolved soon.

> Either way, since Akamai has no backbone, each node with unique,
> Akamai-owned IP space must (obviously) announce its block
> independently.  (If anyone suggests something like GRE tunnels, I
> will ridicule them in public. :)
> 
> 
> Now on to the issue at hand.  As per an earlier thread here, we have
> three /32s.  We deaggregate and hand out /48s to our individual
> nodes.  We also announce the aggregate, as shown earlier in this
> thread.  If you have trouble reaching a node, please email
> noc at akamai.com (24/7), or NetSupport-tix at akamai.com (M-F biz-hours++,
> but likely a better place to get your problem solved).

And I will do that the next time, will owe whiskey if I forget that,
before anyone thinks I am specifically targeting Akamai, which I am not,
especially as they are doing a good job in getting IPv6 out there.

> The interesting thing below is that things "sometimes" work.  If a
> prefix / path is unavailable, it should not work, full stop.

I think what is happening is that due to the missing route6 object some
filtering is over-aggressive and failing some kind of RPF check or
something else. But only way to find this out is doing checks from both
sides.

> And
> assuming you are using a topologically close recursive name server,
> our system should see the disconnectivity and not return that AAAA
> when you resolve v6 hostnames.  I'm not sure what is wrong with this
> particular situation, but we'll be looking into it.  Probably some
> weirdness around SIXXS which I cannot grok in my massively sleep
> deprived condition, but that's another matter.

I'll check again tomorrow how broken it is and follow up on the
noc at akamai address on this.

Note that SixXS is effectively networked in the same way as Akamai
nodes: each PoP uses address space and hardware from the ISP hosting it.
For this you will find no more specifics though (at least you should
not) as they are fully integrated inside the AS.

> As for whether we should deaggregate PA space, I'm afraid that
> decision is already made.  We are not asking for 1000+ /32s from the
> RIRs, and there really isn't another good solution to this problem
> AFAIK.  We are not trying to cause problems, but we have constraints
> in which we must work as well.

Fully aware. I think the policy should be updated for these kind of
network situatuations.

Note that I am not suggesting 1000+ /32's, I am suggesting that address
space used in this way comes out of a block that is properly filterable,
thus marked as 'will be de-aggregated'.

See also my note in the other message about the covering prefix along
with the route6, which is what Akamai does.

I am also heavily of the opinion that we should rename the "Aggregated"
portion of the PA naming to solve part of this issue.

[..]
> It seems more than obvious to me the
> real danger is creating obstacles to distributing traffic more
> widely, not whether we have a few thousand more or even a couple
> orders of magnitude more prefixes in the v6 DFZ.

Please note that the problem is not the proper-use of these routing
slots, you have IMHO valid reasoning to do so.

The problem is that some sites filter as they expect no longer prefixes
than /32 from the PA address space used by the RIRs and that this breaks
things that they expected from the last 10+ years of doing IPv6 already.

Greets,
 Jeroen