Over-utilisation of v6 neighbour slots

[Phil Mayers]

All,

We ran into an "interesting" issue on our wireless network today, caused 
mainly by the known behaviour of Apple clients in generating fresh 
privacy addresses every time there's a power sleep/wake state change 
(i.e. a lot) combined with a non-default NS/ND config on our side.

Specifically, our Cisco 6500/sup720 ran out of IPv6 FIB slots, as 
num_routes + num_neighs exceeded 32k (the default IPv4/IPv6 TCAM split 
on this platform being 192k/32k).

This was probably exacerbated by longer-than-default NS cache timeouts 
on our part, recommended in response to CPU issues we faced as discussed 
here:

http://www.gossamer-threads.com/lists/cisco/nsp/161344

To combat the issue I cranked the "nd reachable-time" down from 900 to 
300 seconds and re-carved the TCAM to give 64k IPv6 and reloaded, but it 
was all most troubling. To give an idea why, some stats:

10576 distinct MAC addresses with 1+ global IPv6 addresses
29610 IPv6 addresses (not including LL)
25.3% of hosts with >= 4 v6 address
17299 IPv6 addresses consumed by that 25.3%

That is, ~25% of hosts consuming ~59% of the addresses in-use. OUIs for 
the MAC addresses in question were almost entirely Apple. Distribution was:

         frequency    rel.     cum.

  1        4651     43.98%   43.98% ***************
  2        2084     19.70%   63.68% *******
  3        1164     11.01%   74.69% ***
  4         729      6.89%   81.58% **
  5         560      5.30%   86.88% *
  6         387      3.66%   90.54% *
  7         276      2.61%   93.14%
  8         206      1.95%   95.09%
  9         177      1.67%   96.77%
10         120      1.13%   97.90%
11          70      0.66%   98.56%
12          50      0.47%   99.04%
13          26      0.25%   99.28%
14          29      0.27%   99.56%
15          16      0.15%   99.71%
16          13      0.12%   99.83%
17           9      0.09%   99.91%
18           5      0.05%   99.96%
19           1      0.01%   99.97%
20           2      0.02%   99.99%
24           1      0.01%  100.00%


Some questions:

  1. Has anyone else run into this sort of thing (neighbour table 
exhaustion) and what kind of approach did you take to solving or 
ameliorating it? DHCPv6?

  2. Does anyone know if Apple (and other vendors) understand the 
negative consequences of their aggressive rotation of IPv6 privacy 
addresses, and are going to address it?

  3. Does anyone know if any equipment vendors have more intelligent 
strategies for handling this kind of situation - LRU expiry of v6 
neighbours at >90% util rather than self-destructing FIB overflow, for 
example ;o)

[We're aware the sup720 is old, but it seems like this could be an issue 
even for more recent devices at sufficient scale]

Cheers,
Phil