ASN's and Prefixes missing from IPv6 table

[Gert Doering]

Hi,

coming back to this after working through the data for my RIPE IPv6
talk... (http://www.space.net/~gert/RIPE/R52-v6-table/).

On Tue, Mar 07, 2006 at 04:33:10PM +1100, Stephan Millet wrote:
> Firstly my apologies for the X posting.
> 
> We have noticed a sudden drop in the prefix size of the IPv6 RIB and the 
> number of origin ASN's in the IPv6 network
> 
> Prefixes suddenly down from ~800+ to ~720
> ASN's down from ~603 to 569

The prefixes dropped in 4 steps, over a period of about a month,
see http://www.space.net/~gert/RIPE/R52-v6-table/page09.html

The first and last of the drops seem to be related to cleanup activities
in 2001:3c8::/32 and in Abilene - it's only more specifics that disappear
(and haven't come back since).

The 3rd drop was "something weird in Korea" - only korean prefixes
disappear, but there is no other common element, like "one common
AS path" or "one common IPv6 uplink".  I have no proper explanation
what happened - except maybe a big outage at some IPv6 exchange point,
or so.  After 2 days, most of the prefixes came back.

The 2nd drop (2006/02/14->15) is a really really weird one, see
see http://www.space.net/~gert/RIPE/R52-v6-table/page10.html

My interpretation of the AS path tails seen for these prefixes is that
the /32s had been withdrawn by the originating ASes at "some unknown
time before", and the session 680->3320 has caused ghosts - the prefix
not visible at 680, but 3320 still believing "we have learned it from
680".  On 2006/02/14, 680 did shutdown the BGP session to 3320 due
to other prefix leakage, and *boom*, all the "ghosts" are really dead
now.

Evidence suggests that the "ghost bug" is really a Cisco IOS bug, and
still present in pretty recent IOS versions.

The fact that nobody hasn't been able to reproduce it in the lab hints
at "some specific condition needs to be met".  Not being able to look
at IOS code, my suspicion is that it's related to filter updates, 
possibly something like that:

  - there's a filter on the session that permits sending of a given
    prefix to a peer / update group
  - an update is queued (but not yet sent)
  - the filter is changed to somethig that will *not* permit this
    given prefix
  - the prefix being in the update queue already, it's still sent out
  - but as the rule is "this prefix cannot have been sent anyway!" it
    is not considered for sending withdraws to the peer in question
    -> boom, ghost

All of this is pure guesswork, but would match the observed effect that
680 says "we send 5 prefixes to 3320" while 3320 said "we receive 500
prefixes from 680" - so there have been proper filters in place at 680,
but at some point, the prefixes "sneaked out".

If someone has too much time on their hands, they could try to reproduce
this - generate lots of BGP updates, change/add/remove/... filters on
a session, and validate whether the prefix set seen on the receiver
matches what the sender claims to have sent.  This is major work, though.

Gert Doering
        -- NetMaster
-- 
Total number of prefixes smaller than registry allocations:  92315

SpaceNet AG                    Mail: netmaster at Space.Net
Joseph-Dollinger-Bogen 14      Tel : +49-89-32356-0
D- 80807 Muenchen              Fax : +49-89-32356-234