BGP4 for failover -- is this the way? 5

[TopRung]

I am trying to determine the best way to develop a failover system. We will have two T1's from different providers.

First T1 used solely for customer base (ecommerce).
Second T1, for internal employee use.

However, I want the T1s to be in essence backups for each other in case of failure.

I was led to believe that using a single Cisco 1700 router with two WIC's, I can use BGP4 for this purpose. Basically when one T1 fails the other will be used seamlessly.


Please provide your real-world advice.


Thank you!!!

 

[baddos]

Yes... That would work if you only accept a default route from your ISP's and you have at least a /24 or larger public subnet.

 

[computerhighguy]

You are in quite a ride if you want to run BGP4. First, your 1700 will probably not be able to run BGP4 with the entire Internet's routing tables. It is tremendously large. I run a 3640 maxed out and it would spike to 100% occasionaly (you should pick one up if you are going to use BGP). You will need to join ARIN (i think I spelled that right). They are the authority that gived out IP addresses. I beleive it is 500 dollars a year. Then you need to get your own AS number from them (the process is very easy). You will need to get your IP addresses from ARIN that are assigned to your AS (BTW AS=Autonomous System). You would then contact each of your service providers and setup BGP between you and them. If you have never setup BGP, YOU WILL NEED HELP!! Depending on how your ISPs advertise thier routes, all your traffic could end up going out 1 T-1 leaving the other one not being used. there is plenty you can do to shape the way other AS's see you and how your packets are routed to and fromm your network.

Keep this in mind. You have to justify your need for Ip addresses every year (not hard).

The minimum amount of IP addresses you can advertise using BGP is a /24. Many ISP wil not accept advertisements for anything less than a /21. However, I have never had a problem and we have been runnin it for a few years.

I woudl seriously suggest that you don't use BGP unless you have a true business need for it Get a couple of T-1s to an ISP that go to different POPS. Let the ISP take care of the BGP. They have better expertise and monitoring than most small commpanies (I am assuming your small becuase you are running a 1700).


Keep this in mind. You could easily disable all Internet access by putting in the wrong command and troubleshooting advertisement problems can take a long, LONG, LONG time. There are people who make thier entire living off og BGP4 only. It is a complex routing protocol.

 

[TopRung]

wow, good information. I am what you would consider a newb to this, and this happens to be the suggestions of a company dedicated to this type of thing. They say all I need is a single 1700 and I am good to go - no probs! HMMMMMMM!!!

 

[gconnect]

computerhighguy is on target, please don't overcomplicate your situation if you don't have to. Believe me, the best practice is to keep it simple (but be secure).

all it seems you should be able to get away with floating static routes

ip route 0.0.0.0 0.0.0.0 THEFIRST-T1 (or next hop ip)
ip route 0.0.0.0 0.0.0.0 THEBACKUP-T1 254 (or next hop ip)

the first one has a weight of 1 (default static Administrative Distance)
the second has a weight of 254. so basically if the first one craps out the second one will be used. Use this with route-maps and NAT and you are good to go

i.e.,

int e0
ip nat inside

int PRIMARY-T1
ip nat outside

int BACKUP-T1
ip nat outside

access-list 101 permit ip INTERNALSUBNETS any 

route-map OUTPRIMARY permit 10
 match ip address 101

route-map OUTBACKUP permit 10
 match ip address 101

ip nat inside source route-map OUTPRIMARY interface PRIMARY-T1 overload

ip nat inside source route-map OUTBACKUP interface BACKUP-T1 overload

...or something like that

-gC-

 

[baddos]

How are your webservers accessed?

If you have say a /24 subnet from your primary ISP, and that T1 goes down... Incoming request for that /24 won't be routed to you.

You might be able to get ISP2 to announce that subnet for you, and in that case a floating static route would work fine.

If you just needed outbound redundancy, a normal floating static route will be more than enough. You just have to remember "How are clients accessing our website?".

;-)

 

[bell1996]

If you're only receiving a DEFAULT Route (0.0.0.0/0) from your ISPs (both of them), then you don't really need a large router. I typically use a 2600 series if I'm just receiving DEFAULT routes. If you want to receive PARTIAL or the ENTIRE BGP routing table then I agree with the upgrade to a fairly large router.

Secondly, your T1's can both be used to transmit and receive traffic simultaneously if you design it correctly. This way you can "load share" (not load balance) you data in an out of your network. This way one of the T1's is not just sitting idle.

I appears that you are going to terminate both T1's into the same router. This will not provide you with a failure to the router (ie. bad WIC, power issue, etc). You will still have a single point of failure. I would suggest two routers (especially if use two 1700's).

I've done this many times and it's all in the PLANNING!!

 

[TopRung]

okay a followup: Everyones suggestions have been crucial.. It disturbed me that the people I am out-sourcing didn't mention that the 1700 would be so deficient, and it was up to my research (your accurate inputs) that I need a better router. Now, they are suggesting the 3600 series, but I am finding out that the 3700 is the way go because of the ability to increase the memory to 256 (future upgradeability for the likelihood that the tables will grow).


WIth some more research, I came across this:

http://www.fatpipeinc.com

--- do any of you have any advice on such devices that are dedicated to serving dual failover setups - they apparantely don't require the hassle with BGP??

Thank you!!!!

 

[TopRung]

SORRY, the url i posted has spaces -- here it is again:

http://www.fatpipeinc.com

 

[baddos]

I beleive radware makes one of those too. I have never used one though.

 

[bell1996]

I'm all for new technology, but you've got to be careful if your going to implementing it into your network.

But, you have to wonder, if it's this good why aren't all the ISP's using it??

As Baddos indicated, there are several other vendors who make similar products.

As far as the 3600 vs. the 3700. I'd go with the 3700 series. Cisco is no longer selling the 3640 or 3620 (3660 is still available, but it's planned to be EOF soon). Also, as with any new Cisco product, it costs more, but you get more bells and whistles with it.

 

[baddos]

Yeah... The 3700 series is the same as a 3600 on steroids. There are some definate advantages with the 3700 series, but if you get a good deal on a 3600?

If your vendor is selling you a 3640, then they are probably selling you a used unit. If you buy a 3725 or 3745, then you will probably have to buy it new. That will be your big price difference.

If you aren't taking full Internet routes, then your original 1700 series route will be fine. Use BGP to advertise your subnet, and announce a better route down one of your ISP's neighbor sessions. Then setup a route-map so that internal users go down one isp, and the web & email servers go down the other.

You won't be limited on memory, because you won't take any routes from the two ISPs. You only really need full Internet routes if you want true lowest cost routing (less hops the better). If you want users down one T1 and servers down the other, then a full view or two full view is a gigantic waiste of memory.

 

[MAPNE]

I would go back to the first response from computerhighguy, BGP can be quite tricky to get right. I'd start by asking what the reasons were behind your decision to go with two upstreams? If you had two redundant T1s from the same upstream things would be a lot simpler, you probably wouldn't need BGP at all, could almost certainly get away with a lower cost router, and with a single upstream they are likely to be willing to provide you with advice, while fewer ISPs will help you with configuring a connection to another upstream (although a few still might). Two T1s to separate POPs of the same ISP can provide as good redundancy and load balancing (sometimes better) as T1s from different upstream ISPs; and, because it's easier to configure, it will also be less prone to configuration problems.

The best reference I've seen on enumerating these considerations was a paper written by Howard Berkowitz (you can see a copy at

http://quimby.gnus.org/internet-drafts/draft-berkowitz-multireq-02.txt,

although that one's fairly old and there may be updated copies).

However, if you're sure you want to go with two separate upstreams you still have several choices. Using BGP and giving every machine behind the router a globally routable address is one. Some solution with dual-homed NAT (as in gconnect's message) is another. There are others as well. Without knowing more about your situation, it's hard for us to guess which is right for you.

One piece of information about your situation that we do have is the comment in your original message that you wanted the T1s to be used for specific purposes when both were up. This is a completely different problem than the multi-homing itself. None of the other posts said anything about this, but that may be because it is nearly intractable and almost never what you really want. If you do need to do this, it is possible, but requires multiple address blocks and more than doubles the complications of configuring either BGP or the other solutions. So, is this really a required part of the solution as your original message mentions, or are better solutions that don't do this acceptable.

Anyway, continuing with the assumptions, let's assume you have decided to go for the full BGP solution (and without the traffic separation). If you are severely financially constrained you might be able to squeeze the minimum of what you need out of a 1700, but the result may not be as useful as you might want, while something in the 3600/3700 series will almost certainly be able to do anything you require.

Next, you need to configure BGP. There are a lot of subtleties in BGP. And the big problem is that you won't know how good the fallback will handle an outage until you have one (which is why I often recommend artificial outages for testing), unless you have lots of experience with BGP and outages. I have seen attempted reliability improvements that actually reduced the reliability by adding the second line, because the BGP was faulty. For this reason I usually suggest that newbies in this area consider outsourcing the BGP config, there are a number of consultants and other providers that will do this for you (disclaimer, I'm a consultant that does this sort of work but I am not specifically suggesting myself, like any sort of outsourcing you should do research and find someone you feel comfortable with, consider their level of experience and how well they communicate as well as the cost).

If you want to do the BGP yourself, you will need an AS number from ARIN (as computerhighguy mentioned) and either your own PI (provider independant) addresses, or PA (provider aggregated) addresses. PI addresses would be obtained from ARIN (in North America, other registries in other parts of the world), but require some work to get, and smaller PI blocks may not be routable by all providers. PA addresses are assigned by one of your providers and you need to get buy-in from the other provider to route these addresses (which is usually easy enough, you're paying them money and not using up any of their address space). If the two providers do not have a direct connection to one another, this buy-in will require transitive buy-in by the upstreams of the second provider, until a point of connection exists (this is to avoid the routing problem that baddos referred to in two of his posts).

In summary, as computerhighguy said near the start of this thread, "If you have never setup BGP, YOU WILL NEED HELP!!" I agree with that, you can get the help by asking in free forums and etget no guarantees just hope you get it right, or you can pay someone knowledgeable to do it for you.get no guarantees just hope you get it right, or you can g

Michael A. Patton, MAP Network Engineering
<

http://MAP-NE.com/>

 

[TopRung]

MAPNE, thank you for the time you have taken to delve into this.

This is absolutely what is needed -> Dual T1's: One for use by employees, and one for use by customer base. The idea is that employee bandwidth-use is not in anyway going to affect the customer line.

However, with that, I need a failover so that if one T1 fails, the other can compensate.

The setup will be 2 T1's, ??routers/BGP??, ISA server, File server, and Exchange server. (Server 2003)

I am at this infancy stage, and trying to determine the best route to accomplish this. I do have an "expert" on call (in ways of a local company that did our current setup), and they are suggesting the BGP4 implementation. However, I have come to find that they haven't setup BGP4 between two routers before. They only use a single router.

 

[baddos]

Two router setups are easy. I would drop your consulting company if they say they can do BGP, but only on one router. BGP is designed for multi-router setups.

Two 1700 routers, an AS number from ARIN, and two public /24 subnets delegated to you from your isp's (one from each) will do the trick.

If you can get a /24 from each ISP, then you'll easily be able to route server traffic and user traffic easily.

From what you have just posted, I would recommend a BGP setup where your ISP's only send you a default route (to save on memory). You would then use a route-map on your routers to forward the internal traffic down one pipe, and server traffic down the other.

If you are buying from big ISPs (UUNET, AT&T, Sprint, etc), most of the time they will offer consulting fees to custom setup your routers to do this. For free, they will only help you setup their link.

 

[ScubaJoe]

I'd like to add a question of my own to this thread. We have a similar failover need, but don't have a need to route specific traffic over a specific connection.

We've had BGP4 recommended to us and in fact have gone through the process of getting our AS from ARIN and had two different techs in to set everything up. We are using one Cisco 3600 router with two WICs. We have two T1 lines from different ISPs.

We've been given multiple stories on how this should work but so far no one has been able to make it do so. After reading elsewhere that BGP4 should be avoided by mid-sized companies without a specific need for it (which has been reinforced in this thread) we are looking for alternatives.

Our biggest consideration is that we need to route the same IP addresses over both ISPs and hopefully have the T1 lines share the load (giving us double bandwidth.) If we can't route the same IP addresses we won't be able to maintain our VPN tunnels (we have about 30.)

Any suggestions on non-BGP solutions would be greatly appreciated.

 

[baddos]

You can route the same IP addresses using BGP as long as your subnet is a /24 or larger (i.e. /23, /22, /21, etc.)

However, you say that you want specific traffic to go down different t1s, but you say you want your bandwidth to double. This isn't possible, but you can split your traffic down different providers to increase your potential bandwidth but never come close to doubling it.

Since you have already begun getting an AS number, then I would go forward with your BGP. Your router is enough to handle it, although you might need to upgrade your memory.

You can only use BGP to route different traffic down different peers if you are using two different subnets (or subnetting an existing one). Example would be that 1/2 your VPN's are using subnet A and the other half are using subnet B. Then you could split the traffic down your two t1s. You could also take a large subnet, and subnet it in your BGP configuration to look two smaller subnets when in fact it is really one large one.

For your setup though, it might be easier just to get another t1 from one of your ISP's and bond it to your existing one. Then use BGP for failover only. Although your budget might restrict this.