Greenfield IPv4 + IPv6 broadband deployment

[Mark Smith]

Hi Martin,

On Sat, 26 Feb 2011 18:30:16 -0500
Martin Millnert <martin at millnert.se> wrote:

> Hi Mark,
> 
> I realize I might have given the impression that what I described was
> rolling today.  It is not. The design only exists on paper atm, and
> equipment is only being delivered as we speak.  Your feedback is
> appreciated.
> 
> On Sun, 2011-02-27 at 09:31 +1030, Mark Smith wrote:
> > Hi Martin,
> > What benefits are there of taking a /64 from the delegated prefix for
> > this purpose? I generally like the idea of saying to the customer (via
> > DHCPv6-PD), "here's your delegated prefix, use it how you want, I'll
> > use this different separate /64 that I choose and manage for the link
> > between us." 
> 
> Well, yeah, keeping routes down would be the motivation.  But you are
> correct in that you could just as well use a /64 from a separate range
> for the RA prefixes.  (Aggregatable per PE box, as well)
> 
> > If I understand you, you're using an IGP to push these per customer
> > routes around. I think BGP would make this scale a lot further if
> > necessary. Depending on the sorts of possible outages you have, and how
> > many customer connections are impacted by them, BGP might be worth
> > using anyway, as because it uses TCP, if a BGP peer is struggling
> > with temporary processing load, it can use TCP windows to tell it's
> > peers to back off for a while.
> 
> Possibly.  It is entirely a topic of its own though. :)  Keep in mind,
> the "PE" switches in question are 24 or 48p switches: there are a lot of
> them.  How do you set it up? (Personal experience with larger scale
> shops is limited.)
> 

I'd probably stick to BGP for everything but loopbacks model. Once you
have your route-reflectors configured, and liberally are using templated
configurations (e.g. BGP peer-groups corresponding to device roles
(e.g. core, peer, edge etc.), route maps, route filters via prefix-lists
etc.), configuring and operating BGP is mainly a cut-and-paste job. For
edge devices, sending them just a default route and applying basic
inbound filtering (which may just be a "customer route" community, which
_shouldn't_ be applied by default by the edge device, use a aggregate
prefix-list to apply it - uncontrolled redistibution is a hair trigger
in my opinion) is enough.

Alternatively you might run an IGP instance within clusters of edge
devices and then have a couple of them (or more likely upstream
distribution routers) inject those routes into BGP. Following the "less
is more" principle, I think I'd still use BGP for this purpose though
if all my edge devices can talk it.

BGP scales much better than IGPs. For example, a IGP having to deal
with 5K+ routes fluctuating is a potential nightmare I'd never want to
experience, where as with BGP it's pretty much a walk in the park (for
a reasonably good implementation). With a goal of providing stable IPv6
addresses to customers, I think there is value in pushing around
individual customer routes within your routing domain within a limited
scope (e.g. geographic region, PoP or chosen cluster of customer
aggregation routers), rather than having a single edge device being a
customer route aggregation boundary. BGP is much more suited to that
task.

> A full mesh iBGP with so many devices requires very clever configuration
> management, and has inherent scaling problems.
> Things you could do to avoid the scaling problems, I guess includes
> "hacks" such as confederation (each cross-connect room could in theory
> be its own private ASN then, peering with other cross-connect rooms
> and/or core - interesting idea actually), or use route-reflectors (Not a
> very attractive idea IMO).
> 

Why do you say that about route-reflectors? My experience using them
has been they just work. Their location tends to follow the hierarchy
of your traffic layer 3 aggregation within your network, so your
route-reflector topology matches 1 to 1 with your layer 3 aggregation
hierarchy.

If you've got access to a copy of "BGP Design and Implementation", the
case studies on ISP and large Enterprise networks is worth having a
look at.

Regards,
Mark.