Occasional Lengthy Spikes

[paradoxwd]

Hey,

Okay, I've been working on the router config for weeks and I am finally getting somewhere. One of the things I mentioned a while ago, is that I am running a game server on my network; specifically a Quake 3 game server. It is because of this, that perfection of the router's configuration is so very important. Keeping low and steady pings is a must, otherwise, it is just impossible to play.

After making the latest changes to the router I have finally gotten the pings low and steady most of the time. However, every now and than, the pings spike up about 50ms and stay that way for a good couple of seconds. They then go back to normal. The other thing I've noticed is that certain people who had low pings of 60 are now steady at 100.

I'm thinking that after all these changes, I might have some settings in the configuration that I should not or maybe, am mising some settings that are required. So whenever you guys get a chance, maybe you could look over my latest configuration and let me know if anything hits you as out of place or missing. Again, thank you all for your help, your patience, and your time.

My latest and greatest:
Current configuration : 2244 bytes
!
version 12.2
service timestamps debug uptime
service timestamps log uptime
service password-encryption
!
hostname Router
!
ip subnet-zero
ip cef
!
!
ip domain-name alter.net
ip name-server 198.6.1.122
ip name-server 198.6.1.142
ip name-server 66.115.51.2
ip name-server 65.212.126.2
!
ip audit notify log
ip audit po max-events 100
!
interface Ethernet0/0
no ip address
shutdown
half-duplex
!
interface Serial0/0
description to UUNet u88932
bandwidth 1536
no ip address
encapsulation frame-relay
no ip mroute-cache
fair-queue
!
interface Serial0/0.1 point-to-point
bandwidth 1536
ip address 63.122.229.10 255.255.255.252
no ip mroute-cache
frame-relay interface-dlci 500 IETF
!
interface Ethernet0/1
no ip address
shutdown
half-duplex
!
interface Serial0/1
description ciberlynx line
bandwidth 1536
ip address 216.242.227.198 255.255.255.252
encapsulation ppp
no ip mroute-cache
fair-queue
!
interface FastEthernet1/0
ip address 66.115.51.1 255.255.255.0 secondary
ip address 65.212.126.1 255.255.255.0 secondary
ip address 216.242.60.161 255.255.255.224
ip policy route-map LINES
duplex auto
speed auto
!
router bgp 23325
no synchronization
bgp log-neighbor-changes
network 65.212.126.0 mask 255.255.255.0
network 66.115.51.0 mask 255.255.255.0
neighbor 63.122.229.9 remote-as 701
neighbor 63.122.229.9 version 4
neighbor 63.122.229.9 soft-reconfiguration inbound
neighbor 216.242.227.197 remote-as 13488
neighbor 216.242.227.197 version 4
neighbor 216.242.227.197 soft-reconfiguration inbou
!
ip classless
ip route 65.212.126.0 255.255.255.0 FastEthernet1/0
ip route 66.115.51.0 255.255.255.0 FastEthernet1/0
ip route 216.242.60.160 255.255.255.224 FastEthernet
no ip http server
ip pim bidir-enable
!
access-list 1 permit 66.115.51.0 0.0.0.255
access-list 1 permit 216.242.60.0 0.0.0.255
access-list 2 permit 65.212.126.0 0.0.0.255
route-map LINES permit 10
match ip address 2
set interface Serial0/0.1
!
route-map LINES permit 20
match ip address 1
set interface Serial0/1
!
!
line con 0
password
login
line aux 0
line vty 0 4
password
login
!
end

 

[svermill]

I don't know anything about game servers, but if your users are out on the Internet, you'll never get a consistent delay to your server. As you have noted, it can vary over the course of seconds, minutes, hours, days, etc. IP is a hop-by-hop technology. No predetermined path exists between any two points. Furthermore, you have no control over all of the other routers in the path. They might employ policy extensively and introduce additional delay.

 

[paradoxwd]

Hey,

This is true, but I am sure there are things that could be in my router's config that shouldn't be, that might also be the cause of increased ping and delay. I'm just asking for anyone willing to just look over my latest configuration and tell me if there is anything that jumps out at them.

I noticed that there was a lot of output drops on the one line, s0/1. I did a search on the internet and found an article on the cisco website discussing how to troubleshoot input and output drops on a cisco router. After going through some sho commands, especially: sho int s0/1 switching, I was able to come to the conclusion that there is a lot of traffic that is being forwarded through the router back to the internet. From what I've read, it is my belief, that all these forwarded packets are most likely the cause of the output drops. The only thing this article doesn't specify, is how to fix this.

So my latest question is this: How would I go about stopping these packets from entering the network and causing all this traffic? If I were to add deny anys to the bottom of both my access-lists, would that stop the packets from entering?

One thing I need to note is that it is imperative that packets are still able to get out even if the line specified in the route-map is down. I am pretty sure that with the current configuration this is not a problem, but if I were to add the deny anys, would that stop this from happening?

 

[wybnormal]

No.. the policy routing ACLS are for outbound only. They wont filter anything inbound which is what you are talking about right now. In other words, we want to stop it when the rogue packet comes into the serial port, not after it's been processed by the router. Do you know *what* type of traffic is inbound and then back out? routing updates? TCP traffic? To build the ACL, we first need to identify either what HAS to get through and block everything else OR just block the few IPs that are causing grief.

MikeS
Find me at

http://www.packetattack.com

"The trouble with giving up civil rights is that you never get them back"

 

[svermill]

OK. The one thing that jumps out at me is your route maps. They enforce policy - which takes CPU cycles and time to execute them. Is it enough to make a difference? You can usually only find that out by comparing results with policy off and then on.

As for transit Internet traffic through your router...I think you need to look at this example config:

http://www.cisco.com/warp/public/459/27.html

As for your access lists, they are "access lists" per se. They are merely used as part of your route-maps - they don't allow or disallow anything except to allow certain traffic to enter into the route maps. Which brings us to your last question (I think).

If I remember my route maps correctly, they will not set to a down interface. The default then would be to route normally as if no policy existed. The exception here might be if you route-cache your policy for performance enhancement - something you may want to research.

Finally, you don't need:

ip route 216.242.60.160 255.255.255.224 FastEthernet

but it isn't hurting you. You never need a static route to a directly attached interface. I don't think you need those routes to the secondary interfaces either for that matter.

 

[svermill]

That was supposed to read that "they *aren't* access lists per se."

 

[paradoxwd]

What stats commands would I use to figure out the answers to your questions?

Because of the outbound policies, both lines are now outputting at high levels, but it almost seeems as though the connection is slower. I believe this is because of all the forwarded packets. I really hope I can figure this one out. I believe this is also causing the pings to the game server to be higher than they should be; the spikes are probably also caused by all this unnecessary traffic.

I have a question for you. BGP, is it really necessary? The reason we have it is because we host many websites that require that they are up all the time. If one of our two lines goes down, we want the servers to use the other lines. As of right now we do not have portable ips; but if we were to get portable ips, could we have this same fail-safe functionality without BGP?

 

[paradoxwd]

Svermill,

What part of that link you gave me should I look at? They mention the problem of an AS becoming a transit for internet traffic, but dont mention which method of bgp configuration fixes the issue. Don't I want full internet routes or would I want only the direct routes with default routes to the internet?

 

[svermill]

You're getting into an awfully grey area for all but the most experienced BGP gurus (the when to use - when not to use debate). Unfortunately, I'm not one of them at the moment.

By the way, have both of your ISPs agreed to advertise each others address space? You loose a lot of what you think is redundancy if not. If you loose ISP A, how will folks out in the Internet reach your servers with addresses from ISP A via ISP B and vice versa?

I was pointing you to this part:

router bgp 300
network 1.0.0.0
network 2.0.0.0
neighbor 10.10.10.10 remote-as 100
neighbor 10.10.10.10 route-map localonly out
neighbor 20.20.20.20 remote-as 200
neighbor 20.20.20.20 route-map localonly out

Notice the route maps that prevent your router from advertising anything but your own routes (this isn't the entire config but shows where you need to focus). This prevents you from telling ISP A it can reach networks in ISP B via you. I would recommend that you read the entire document since it also addresses another issue - do you really need the entire Internet routing table? I can't think of any reason why but maybe you can? Here is another link on that subject:

http://www.cisco.com/warp/public/459/41.shtml

 

[wybnormal]

Hey Scott- we had this discussion earlier I mentioned that BGP was not a good idea and most ISPs would rather you connect with OSPF. I also mentioned that if BGP was required by the ISPs then *they* should be able to get the config correct for their network. Not leave our programmer here treading water..

I would not expect policy routing to killing performance but anything is possible.

You can do a:

show process

then you get something like this:
nemesis#show process
CPU utilization for five seconds: 6%/0%; one minute: 2%; five minutes: 1%
PID QTy PC Runtime (ms) Invoked uSecs Stacks TTY Process
1 Csp 31C3BBE 408020 1360575 299 724/1000 0 Load Meter
2 M* 0 848 127 6677 2112/4000 2 Virtual Exec

You can see a snapshot of CPU utilization and which process is sucking up the processing time. This is a good way to start finding out if the slowness is based on CPU loading (ACL) or something else like a routing protocol.. there is nothing like seeing IPX EIGRP sucking up 100% of CPU clock cycles (it was a HUGE routing table that collapsed every so often)

Whats does :

show interface S0 or S1 showe? specificly the load ratio number? it will be something like load 28/255 in the right hand side of the printout.

MikeS

MikeS
Find me at

http://www.packetattack.com

"The trouble with giving up civil rights is that you never get them back"

 

[svermill]

I guess I missed that discussion. Have been travelling a lot/working long days. So I've missed quite a bit in recent weeks.

I still think BGP is OK if you dumb it down to the minimum required functionality. Bells and whistles are great if you have the horsepower to back them up. Otherwise, just do what you need to get things running smoothly and leave the rest behind - especially the full table part in the absense of a darned good reason.

And I certainly agree that a few policies shouldn't be killing this router. But since we're concerned with every millisecond here, I thought I would point out the fact that policy can add a few here and there. Not normally detrimental, but this appears to be a special case where every bit counts. I don't even know what kind of router we're talking about here, but I assumed it was at least mid-grade since it was running BGP with full tables.

 

[wybnormal]

It's a 3600 something.. I forget exactly which one he has.. It's also his default gateway for the LAN which I have already mentioned is a bad idea for the gateway to also be the BGP router.. talk about all your eggs in one basket..

Speaking of...

Paradox.. did you ever get that guy out to config the firewall? sometimes the firewall can be the cause of spiking on a response time..

MikeS
Find me at

http://www.packetattack.com

"The trouble with giving up civil rights is that you never get them back"

 

[paradoxwd]

Hey,

No, the firewall is still inactive. I know, I know!

The answer to your question svermill, is yes. Both of my ISPs advertise there address space to eachother, which is how ip's on one line still work on theo ther if that one line is down. Which is confusing to me, because if they are doing the BGP advertising on there end, why the heck do I need BGP on my end? Furthermore, if BGP is unnecessary, what is the alternate solution to my situation? I have two lines from two seperate ISPS, even if one of my lines goes down, all my servers need to stay up. Would portable ips be my solution? If so, how would that work?

I tried contacting my ISPs for information, but they only asked me why I am playing with the configuration if it's "working" now. I tried to explain that I was trying to balance the lines and create less congestion, but it didn't get me anywhere.

Throughout the weeks that I have been working on this router, I have learned a lot and finally have a pretty good understanding of how most of the technologies behind cisco routers work. I even have a pretty good sense of how to configure cisco routers and how to troubleshoot some of the common problems. I'm no expert, but I'm getting the hang of it, that's for sure. The only thing that mystifies me as of yet are the reasons behind using BGP. I've heard so many conflicting comments about it, I am just not sure what to believe. Is it the greatest thing to have ever entered the router configuration world or is it the nightmare that keeps you from falling asleep at night? What are its pros, whats are its cons, and do its pros ever out weigh its cons?

 

[svermill]

Well, as I said, I'm no BGP expert myself. I know the *technical* side of it at about a mid level. But deciding when and where to implement it (outside of a carrier type of environment of course) gets a little rough. We've implemented it for large clients who have multiple campuses spread out over large areas. The pros clearly outweighed the cons in those cases. For the small guys such as yourself, I can't make a strong argument for or against. Since both ISPs are advertising each others address space for you, you already have inbound redundancy. If you elect to do away with routing protocols all together, you can just do two default routes outbound. Don't know how critical it is for you to have strict control (using route maps, etc). The default route is simple and thus not very robust in that regard. Also, I think you may have some symmetry issues to think about with your delay sensitive application.

Having said all of that, your BGP config is pretty basic. You're not really digging deep into the bells and whistles that BGP has to offer (such as exerting limited influence over inbound traffic through BGPs advanced attributes). One thing that Cisco will always tell you is that one of the first criteria for implementing BGP is a clear understanding of how it works and what it is going to accomplish for you.

The bottom line is that BGP allows a very high level of granularity with regard to policy enforcement, route selection, etc. and you don't seem to be doing a lot in that regard but you are doing some. I'm not 100% sure I would keep it in your case. But it is far too advanced a subject to be thoroughly explored in this forum. You would need to undertake a complete study (such as reading the book by Halabi) to make a fully informed decision for yourself.

I would make this final suggestion at a minimum: reduce the workload as much as possible. Why do you need the entire Internet routing table? If you can't think of a reason, I would try going with defaults or ISP routes and defaults. Also stop acting as a potential route between your ISPs. Only advertise out YOUR own routes. This may help your bandwidth issues quite a bit. Worth looking into in any case.

Best of luck!!

 

[paradoxwd]

Hey,

That just didn't answer anything. I do not even know the difference of a full route table and direct routes to my isp. So there is no way for me to even understand the pros or cons of either scenario. Why would I want full route tables?

As far as using default routes, well, I am trying to get things in and out of my router as far as I can. So wouldn't specific route-maps make more sense? Besides that I was trying to spread bandwidth evenly across both lines, which is what has happend with the route-maps. However, what has also happened is that my router became a transit for other internet traffic.

Couldn't I just advertise my own routes (which would mean not having the full internet routing table, correct?) and still keep the route-maps?

My only question again is, why would someone want full internet routes?

 

[svermill]

BGP makes pretty darned accurate decisions with regard to optimum route selection when it has a full table. Given your delay sensitivity, this may be of value. A user may be several hops closer to you via ISP A vs. ISP B. On the other hand, is this in itself a good enough reason to run BGP? I can't make that call for you. Again, this is one of those areas where documenting results with something turned on and then with it turned off is the only real way to know. Of course, your ISPs are likely to get a little testy if you keep calling to turn BGP on and off.

As for dealing with your transit issue between the two ISPs, the link that I gave you earlier has a simple half a dozen line config for restricting what you advertise to your ISPs to the networks that are behind your router. You don't want to be advertising everything ISP A has to ISP B and vice versa. Just sending the tables is cause enough for concern, let alone the transit traffic that you will be kindly handling for your ISPs (who's paying who?).

......

neighbor 10.10.10.10 remote-as 100
neighbor 10.10.10.10 route-map localonly out
neighbor 20.20.20.20 remote-as 200
neighbor 20.20.20.20 route-map localonly out

......

route-map localonly permit 10
match as-path 10

......

ip as-path access-list 10 permit ^$

......

In this example, the ISPs are 10.10.10.10 and 20.20.20.20. This is roughly the equivalent of a distribute list that references a route map which references an access list that permits only locally originated routes (those that are specified in a 'network' statement under the BGP config.

 

[paradoxwd]

so tht ^$ is responsible for all that? Damn!

 

[svermill]

Weird stuff, I know. But it should do the trick for you. By the way, I am back on the road again and my laptop did not survive my latest journey. So I have limitted internet access for the next few weeks. Good luck and I hope you get this thing working the way you want it. I'll drop in from time to time so keep us posted as to your progress please.