Router, Pix or Microsoft?

[rpast]

Hello all –

I have a mob of disenchanted users about to lynch me because of a problem that I cannot get to the bottom of. For the sake of simplification, let me summarize one aspect of the problem by saying that I recently migrated Microsoft Exchange Server 5.5 to 2K3, consolidating three sites in the process. Now I have a problem with the users at the remote sites, who require a restart of the centralized Exchange server once every few days in order to connect.

Although the problem manifests itself in one particular application (Exchange), I suspect that this might be a problem with the connecting routers and firewalls (I have submitted this in the MS Exchange forum as well). Forgetting a moment about Exchange, can someone tell me how I might determine whether a particular router (3640 or 2610) or Pix firewall (515 or 501) is overburdened at a given time?

I’ve done a “show process”, “show tcp stat” on the routers, and “show conn” and “show local-host” on the Pixes. But I’m suspecting there might be some kind of lingering connections that might be causing this. Exchange opens four RPC connections per client. Yet connections all time out per the defaults on the Pixes (rpc 10 minutes). The 501’s (at the remote sites) have a 50 local-host limit; the 515 has unlimited inside hosts. Everything appears normal. Is there a practical limit to the number of connections involving one host (the Exch server) that can be open on the routers or Pixes?

Maybe memory is a problem. The 3640 is maxed out at 128MB ram. But the (local) Pix 515 has only 32MB, and the (remote) 501s are at 16MB. I’m at a loss as to how the Cisco devices might be contributing to this, but don’t know what else to try.

This is a tough problem in that it affects only one application – MS Exchange, and yet only remote users of that application. I just don’t have enough troubleshooting tools or experience to nail this one down. Of course, one obvious thing I will try next time this happens is restarting the WAN devices one by one to see if that has any effect – assuming the users don’t kill me for the delays – not sure what that will prove in any case, without a better idea of the causes.

Any suggestions would be appreciated.

 

[fdurham]

I believe the problems lie with in the Pix's and the outlook clients possibly. Can you explain to me how you are connected from your remote sites to the main site. Give me what type of connections, what type of devices, and where they are in relation to the Exchange server. I diagram would be helpful as well.

Frank

 

[Cotton9]

Had what sounds like the exact same problem reciently. Win2k uses 1300 byte MTU packes while win3k uses 1500 Byte MTU. Windows by default does not use PMTUD (nice). Increase your MTU until things begin to work. Suggest: 1524, 1532, etc.
You might also try ip tcp adjust-mss 1433 or 1452 (not sure which is best) on the ethernet interface's. This command 'should' tell Windows to use the smaller MTU size and you would not have to adjust MTU.

Look up PMTU or PMTUD on Windows web site for details. It's tough to find on Cisco's web site.

 

[rpast]

Thanks for the replies. Some Exchange people had recommmended a hotfix (898060) for Win2K3 SP1. It's been four business days without problems, and I'm keeping my fingers crossed.

I had looked in the PMTUD and related ideas -- an Lsa/Kerberos tweak had helped server replication, but not this particular problem. As I understand it, PMTUD is on by default, at least for Win2K3 SP1. But I noticed another related setting, PMTUBHDetect might be necessary in cases where a 'black hole' router is not responding when it has to fragment packets that have the DF bit set -- instead it just drops them. And PMTUD needs to have those replies in order to do its thing, adjust an MTU for a particular session, etc.

So if Hotfix KB898060 doesn't do it, my next step might be to enable PMTUBHDetect (off by default). But I really don't expect that to turn up anything. As a last resort, I'll force a low MTU at the host and server levels. I hadn't considered the 'ip tcp adjust-mss' option, but will give it some thought also.

Thanks again. If I don't report back, then the hotfix fixed the problem.