failing over better with multiple prefixes?

[Dave Taht]

On 02/24/2010 06:35 PM, Benedikt Stockebrand wrote:
> Hi Dave and list,
>
> Dave Taht<d at teklibre.org>  writes:
>
>    
>> Q1) Is there a "best (and cheap) way" to get a dedicated, independent
>> IPv6 address space and BGP AS number that people are willing to route?
>>      
> I don't know, but...
>
>    
>> Note that I am in the third world
>>      
> ... you'd best ask around locally.
>    
Have. The one ISP that had a bit of a clue (cablenet.com.ni, who ran out 
cable internet to great success about 1 1/2 years ago due to hugely 
frustrated demand to most of the larger cities) got themselves bought by 
the local cell phone company (claro) and seem to have run off all the 
tech folk.

It is not encouraging to come up empty for "ipv6" for any of the local 
providers.

http://www.google.com.ni/search?hl=en&safe=off&q=ipv6+site%3Awww.claro.com.ni&btnG=Search&meta=&aq=f&oq=

It doesn't help that my technical Spanish isn't the best, either. I'm 
sure there is someone out there that groks ipv6 and has fiber in Managua...

> Out of curiosity: Where are you?
>    

Nicaragua. The internet may suck but the surf is good and my office view 
is great ( 
http://www.teklibre.com/~d/casayanqui/masterbedoomviewbetter.jpg ).

I can see Costa Rica (about 18 km) but word has it that the Net sucks 
even harder over there. Am still thinking that running a wireless link 
over to there would make sense...


>    
>> So joe has a dedicated IPv4 address and uses the resultant 2002
>> network to add IPv6 to their net.
>> [snip]
>>      
> Don't use 6to4 in production environments: If the public relay you use
> causes trouble you may not even know who to contact -- or how.  And if
> the tunnel routers in the reverse direction don't work the situation
> is even worse.
>    

You know, I get this bit a lot. I've been using 6to4 off and on for 
about 9 years (in the US, mostly), and it has generally been very 
reliable (in addition to easy), and for systems located "next door" to 
each other topologically the latency is low and bandwidth high.

Your caveat

"If the public relay you use causes trouble you may not even know who to contact -- or how"

applies more often to normal routers in between me and my destinations 
over ipv4, than to a 6to4 router out in the cloud. Down here, 
connectivity is frequently lost to the adjacent city (Rivas), the 
capital (Managua), or outside the country, so getting to local 
destinations is important.

So (after about a year of using it down here) 6to4 tends to be more 
reliable, faster, and lower latency on the whole than using tunnels, due 
to the accumulated MTBF of the other 20+ routers between me and hurricane.

I recently read of a deployment using 6RD (and just started building 
that into my kernels) which looked like an encouraging adaptation of 
6to4 tech...

Aside: What does it take to own an operate a 6to4 router?

> If I understand your setup correctly, then you are quite likely best
> off to use tunnels (HE or whatever else you like and can get) until
> your ISPs start to offer you native IPv6.
>
> You can then route these adresses through your dedicated links as
> needed.
>
> As for the failover issue: Yes, you can advertise addresses both from
> your direkt upstream and your main office in all sites, but keep the
> fallback addresses deprecated until something nasty happens.  As long
> as you are using tunnels to HE, consider moving your tunnel end point
> to wherever there is working connectivity.
I am trying to get to that point, however, things like whatsmyip tend 
not to work reliably once you go behind an ISP's NAT to the home....

>   Once you have native IPv6
> connectivity and a cooperative provider
I may well be old and gray before that happens. Most providers down here 
stick people behind at least one layer of NAT, sometimes two or three. 
I'm pretty sure this is the case throughout the third world and asia.

> , use a tunnel from them
> through another connection as a fallback -- yes, this requires
> cooperation from your ISP.
>
>    
It really sounds like I have to bite the bullet and become my own ISP.

>    
>> There are also dedicated links between the offices, in more or less of
>> a wireless mountainous mesh, many of which have their own real IPv4 IP
>> gateway from various providers as well, as well as a few that are
>> stuck behind NAT from less wealthy providers, for which they use a
>> hurricane electric tunnel currently. So it's a huge waste of very
>> limited bandwidth to backhaul everything to the main office...
>>      
> Will an AS and possibly PI addresses actually help in that case?
>    

Well the nice thing about technology is that I can take advantage of the 
latest stuff.  My next-gen private links are being built around openwrt 
based routers (ubiquity), which means the private backbone mesh will 
probably run at close to 54Mbit, compared to the 32Kbit (or less) most 
of the offices have to the internet.

I'm seriously considering adding the next generation sheevaplugs 
(guruplugs) to add even more smarts to the individual offices (512MB 
ram, 512MB flash, 5 watts), like squid, and Xorp or quagga.  Important 
is power consumption because solar/battery systems are expensive...

Anyway, getting a real, independent IPv6 address space and a AS number 
would hopefully let me egress/ingres from the multiple points where I 
have control of the IPv4 address, but no, it doesn't solve all problems.

>    
>> Every machine ends up with one IPv6 address per (IPv4 or tunnel)
>> gateway. This is kind of unwieldy, particularly if you want to wedge
>> all this into DNS, too...
>>      
> First of all, unless the situation in your setup is way beyond what
> I'd consider reasonably reliable, you can normally limit yourself to
> two or three addresses.
>    
It's way beyond what I'd consider reasonably reliable. Power outages 
last 8 hrs to 3 days, power flickers on and off 5-10 times a day, we 
have persistently low voltage (<95v) which wrecks UPSes, and a horde of 
other problems.

I'm writing and sending this email now, without power, in the dark, with 
only my monitor and backlit keyboard for illumination, on a battery 
backed up wireless link hopping over a couple mountains...

The amazing thing is that it works at all (it pays big to be running 
mail servers locally. postfix will try REALLY hard to deliver and get 
mail the way I've set it up over IPv6).

It is a dramatic contrast to how everything else is getting done "in the 
cloud" these days. It's so dark I can't even see the cloud...

> As far as the DNS issue goes: That's what scripts are there for.
>
>    
Well, it would be nice if I knew how to detect deprecated addresses on 
the client side on either windows or linux without polling.
>    
>> Still, this works pretty good when everything is working.
>>      
> It's always good when everything is working -- the headaches come from
> not thinking in time about how to keep it that way:-)
>
>    
Staying in the US - or better, moving to Europe - might have been a good 
idea.

>> Q2a) It's unclear, that once you start assigning multiple 2002: or
>> 2001: tunnel prefixes how to make the local 2002 or 2001 gateway
>> prefix be the primary source address prefix. The routers are all
>> running a reasonably recent version of linux (2.6.26 at minimum), and
>> have ip rules tables installed for source routing (ipv4 only at the
>> moment) where needed...
>>      
> You can't really do that on the routers.  Check RFC 3484 for the
> details on how address selection works.  Unfortunately, at least last
> time I checked Linux didn't have a configurable address selection
> table -- but then, you'd have to do this on the hosts as well, so it
> is pretty much out of an option.
>    
I'm not really sure what you mean here.  libc has a rule database for 
address selection built into it, am not sure about uclibc....

My thought was I would (in case of failure of a given prefix) block 
outgoing source addresses from that prefix at the local router and 
return the correct ICMPv6 unreachable message (not sure what that is).

The clients would then cycle through their other source addresses 
rapidly until it finds one that works.

(I more or less do this currently with ipv4 rules in the RPDB. Is there 
a linux/ipv6 mailing list that is active these days?)

> The pragmatic way to deal with this is through the preferred lifetime
> mechanism.
>
>    
>> Q2b) It's really unclear how IPv6 interacts with the linux RPDB these
>> days...
>>      
> I'd actually rather avoid using any Linux- or whatever-specific stuff
> for this.
>
>    
Got an alternate suggestion? I think highly of BSD but it doesn't run on 
either my wireless router or "smart server" of choice...
>> Q3) When an external link goes down they'd like the offices to
>> fail-over to routing internally to the next best gateway. But if the
>> source address is 2002:the-local-office, sending stuff out
>> 2002:some-other-office's-gateway isn't going to work very well.
>>      
> That's another reason why you don't want to use 6to4.
>
>    
That's another reason why you don't want to use multiple tunnels either, 
or anything short of a BGP mediated internet connection. Multihoming is 
actually something that NAT is useful for....
>    
>> Now, I don't really want to talk about NAT. What I would like to have
>> happen, is when an external tunnel goes down, that the IPv6 addresses
>> derived from it become deprecated and aren't used for future
>> connections, and when it goes back up they get undeprecated....
>>
>> It's not clear to me how to do this aside from rewriting the
>> radvd.conf file and restarting radvd, and even then, there doesn't
>> appear to be syntax to "deprecate" an address only (thus eliminating
>> the need to remove it from DNS)
>>      
> Yes, just set the preferred lifetime to 0.
>
>   
I was under the impression that lifetimes of less than 7200 secs were 
problematic. I'll try it!
>   
>> Is there something that I could use other than radvd?
>>      
> The problem is not so much with radvd but with the router
> advertisement protocol as such.
>
>    
No, well, I'd like it very much if radvd went away and what it did got 
integrated into Xorp or some more complete routing solution. I am 
looking over ahcp + babel ( 
http://www.pps.jussieu.fr/~jch/software/ahcp/ ) and liking what I see....
>> ip addr change 2001:470:b9d7:e::1/64 dev eth1 deprecated
>> Error: either "local" is duplicate, or "deprecated" is a garbage.
>>
>> (whenever this happens I can also remove the IP address from DNS via
>> nsupdate)
>>
>> I've been fiddling with the syntax of this ip command for a while to
>> no avail. (linux 2.6.32) It doesn't accept any of the other documented
>> flags (primary, secondary, etc) either.
>>      
> Yes, I love it, too.  At least they managed to get some sort of
> documentation done by now.
>    
ok, I'll dig into the code a bit.

>
> However, if the problem is not with HE, your script could
> alternatively start a new tunnel for that prefix from elsewhere (like
> your main office) and you could then re-route your traffic through
> there.
>
>    
as I wrote earlier whatsmyip is unreliable, so tunneling that way tends 
to be error prone.

aiccu? some other tunneling mechanism? One thought I had is that I have 
control of several machines in the states that have native ipv6 
connectivity and can also run 6to4. I don't have control of the native 
ipv6 stuff (/64s) so am limited to delegating my own 2002 from there 
here....

So I could tunnel a bunch of those 2002 prefixes down here using 
whatever mechanism I felt like, through multiple gateways... hmm... it 
would probably be as fast as hurricane and less hassle...

> Cheers,
>
>      Ben
>
>