Cisco 4006+4013+4232-L3 bandwidth / speed issue 2

[rwhsec]

Hello everyone!

Subj: Cisco 4006+4013+4232-L3 bandwidth / speed issue

I have a Cisco 4006 with a 4013 Supervisor and a 4232-L3 and 3 x 4148-RJ modules.

When downloading data from ServerA on the switch to ServerB on
the switch the download speed is limited to about 650KB/sec.

ServerA -> ServerB = 650KB/sec.

ServerB -> ServerA = 11MB/sec.[/b[

So the problem is in one direction.

Each server has its own VLAN and a /29 IP block and both servers are physically connected to the 4006's 4148-RJ modules.

This seems to affect servers at random.

This issue disappears if I change the /29 IP block on the server experiencing the slow downloads while still using the same VLANs and ports to a /29 not experiencing this issue.

When I change the /29 to one that is not affected the switch is capable of full wire speed which is approximately 11MB/sec sustained. When I change back to the old /29 the downloads are limited to about 650KB/sec max.

I have tried the following:

changing VLANs

changing ports on the 4006

replaced CAT5 patch cords

replaced NICs in affected servers

replaced motherboards in affected servers

had multiple people examine the configs

and a few other crazy ideas yet the problem persists.


This problem affects only a few servers on the switch however I have a similar problem on a larger group of servers as well.

Several servers are limited to 650KB/sec downloads from very fast servers outside the network on the public internet.

For example:

LocalServerA on the 4006 with a /29 is downloading from RemoteServerA on the internet and is getting about 650KB/sec.

RemoteServerA -> RemoteServerA = 650KB/sec

LocalServerB on the 4006 with another /29 is downloading from RemoteServerA on the internet and is getting 3-8MB/sec sustained.

RemoteServerA -> LocalServerB = 3-8MB/sec


If I change the /29 on LocalServerA to a /29 which is unaffected by this issue the speed problem disappears while still using the same hardware, VLANs and physical ports.

Again the problem is in one direction.

Has anyone encountered anything like this before?

Please let me know if you need any more information and if I can post it I will.

Any help will be greatly appreciated.

Thank you in advance,

rwhsec

 

[ADB100]

I don't think you have explained enough for me to understand how you are testing this:

This issue disappears if I change the /29 IP block on the server experiencing the slow downloads while still using the same VLANs and ports to a /29 not experiencing this issue.

This seems to imply you are not changing the VLAN for the server just changing it's IP address? Which shouldn't work anyway unless you have left proxy-arp enabled on the VLAN SVI?

Maybe I have misunderstood what you are doing, if so please try to explain it better?

Andy

 

[rwhsec]

Andy,

Thanks for the fast reply!

Let me see if I can explain this better...


[ localserverA ] # wget remoteserver.com/large_file

650KB/sec


[ localserverB ] # wget remoteserver.com/large_file

3-8MB/sec


[ localserverB ] # wget localserverA.net/large_file

650KB/sec


[ localserverA ] # wget localserverB.net/large_file

11MB/sec


both servers are on the same 4006, same network, same Class C IP block, same config on the 4006 except for VLAN # and IP address.

The lucky /29 downloads at full speed.

The unlucky /29 downloads at 650KB/sec from the internet and locally.


When I download data from another server connected to the same 4006 switch I should be getting full wire speed which in this case happens to be 11MB/sec, the vast majority of servers on the 4006 can download data from each other at wire speed, a handful cannot and are limited to 650KB/sec even though the data is supposed to be traveling across the 4006's multi-gigabit capable backplane. What is even more strange is that in the other direction I am getting wire speed.

rwhsec

 

[rwhsec]

Also when I change the /29 on the affected server in the router it speeds up.


To change IP I leave the VLAN the same and just change the IP address in the router, that works and does resolve the issue however I end up with unlucky IPs I cannot use.

Thanks,

rwhsec

 

[ADB100]

To change IP I leave the VLAN the same and just change the IP address in the router, that works and does resolve the issue however I end up with unlucky IPs I cannot use.

Do you change the IP address on the Server as well?

 

[rwhsec]

Yes of course.

 

[rwhsec]

So I suppose the question is what could be causing seemingly random /29s on the router to have performance issues when other /29s are operating normally?

The only variable is the IP block used on the servers. Everything else is the same.

Could it be something in the 4013 or the 4232-L3?

Thanks

rwhsec

 

[vipergg]

It could be anything . All switches are over subscribed within a group of ports so if you have a number servers within a given group of say 8 or 16 consecutive ports you could be running into something like that . If the server is say in 2/10 move it to say like 6/40 and change vlan and see if the problem follows , if so then something is going on in the subnet itself and you are going to have to get a network analyzer on there to see what is going on . could be anything from a device that has a virus to someone with faulty nic etcc....

 

[vipergg]

also as a last note make sure you don't have overlapping address ranges specially check the /29 you having trouble with that it doesn't overlap with an existing range this will cause problems , I have seen stranger things.

 

[rwhsec]

Thank you vipergg.


I already changed NICs, CAT5, motherboards, VLANs, tried different ports, different ports on the same blade, different ports on different blades.

The problem appears to follow the IP address.

Once the IP address on the router and server is changed the problem is resolved even though the physical port, blade and VLAN remain the same.

 

[rwhsec]

I do not believe it is possible to have overlapping IP address ranges.

The router will not allow us to configure overlapping ranges even if we tried.

Thanks,

rwhsec

 

[ADB100]

How is the trunk between the 4232 Router and the backplane configured? IIRC there are two Gigabit interfaces on the router that interface with the backplane, the supervisor sees these are <slot>/1 & 2. Is this configured as an EtherChannel & 802.1q Trunk? Is is possible that it is an EtherChannel load-balancing issue? What sort of utilisation are you seeing on the two physical ports? (from both the Routers and the switches perspective)

Andy

 

[rwhsec]

Yes that is exactly how the trunks are setup.

Also the utilization is very low as you can see below about 3 % combined on 2/1 and 2/2

Thanks,

rwhsec

Console> (enable) sh trunk
* - indicates vtp domain mismatch
Port      Mode         Encapsulation  Status        Native vlan
--------  -----------  -------------  ------------  -----------
 2/1      nonegotiate  dot1q          trunking      99
 2/2      nonegotiate  dot1q          trunking      99

Port      Vlans allowed on trunk
--------  ---------------------------------------------------------------------
 2/1      1-1005
 2/2      1-1005

Console> (enable) sh top
Start Time:     Aug 16 2007 14:22:01
End Time:       Aug 16 2007 14:22:35
PortType:       all
Metric:         util
Port  Band- Uti Bytes                Pkts       Bcst       Mcst       Error Over
      width  %  (Tx + Rx)            (Tx + Rx)  (Tx + Rx)  (Tx + Rx)  (Rx)  flow
----- ----- --- -------------------- ---------- ---------- ---------- ----- ----
 2/4  a-100  16            121480008     201602          0         16     0    0
 2/2   1000   2            157386681     328768         48          7     0    0
 2/1   1000   1            128842091     214013         18       2022     0    0
 2/3  a-100   1              8961129      35703          0         23     0    0
 3/18 a-100   1              8264554      19840          0         18     0    0
 5/6  a-100   1              8417351       8353          0         17     0    0
 3/4  a-100   0              6673772       7289          0         17     0    0
 4/31 a-100   0              5373375       7652          0         17     0    0
 2/9  a-100   0              4787114       8410          0         17     0    0
 4/30 a-100   0              4543134       5535          0         18     0    0
 4/39 a-100   0              4161644       5538          0         17     0    0
 5/30 a-100   0              4134969       5689          0         18     0    0
 2/20 a-100   0              3974159       6712          0         16     0    0
 3/32 a-100   0              3780219       4225          0         17     0    0
 2/21 a-100   0              3632051       6942          0         16     0    0
 5/40 a-100   0              3749360       5213          0         17     0    0
 2/16 a-100   0              3693646       4301          0         16     0    0
 2/18 a-100   0              3474364       8652          0         16     0    0
 5/8  a-100   0              3209951      22448          0         17     0    0
 3/11 a-100   0              3535724       4988          0         17     0    0
Console> (enable)