Exchange Server w/ 2GB RAM SLOW 1

[lifegard2]

I just migrated about 75 mailboxes from exchange 5.5 to 2003. Yesterday I got the last of the public folders and mailboxes migrated over and switched the Exchange org to Native 2003 mode. Since then Exchange 2003 is running doggedly slow, slow enough that everyone is noticing.

The server is a Dual P4 Xeon 3GHz with 2GB of physical RAM. You'd think it would be able to handle the load. The only apps on the server are Exchange, WINS, and Symantec AV for Servers (with the recommended Exchange folders excluded from real time scanning).

I found the M$ best practices for Exchange 2003 on servers w/ greater than 1GB of RAM. I implemented the /3GB switch and /USERVA=3030 switches in the boot.ini file. I notice I have another switch in the boot.ini for "/NoExecute=OptOut". Not sure if this could be my culprit.

Whenever I look at taskman and performance, it shows only around 1GB of RAM in use and CPU usage is below 15% most of the time.

When I try and add counters to the performance monitor, I get a Visual C++ Runtime Error that mmc.exe has crashed, so I can't monitor any of the recommended performance counters.

Any ideas?

 

[skialta]

Where is your Active Directory Global Catalog? Do you have several in your Org? What is the speed of your DC/GC's? That is certainly a possible culprit too...

 

[xmsre]

How is your disk configured?

 

[lifegard2]

Active Directory GC is on both Domain Controllers. 1 DC is same model as this new Exchange box (Dual P4 Xeon 3GHz w/ 2GB RAM and 600GB SATA RAID5). The other is on an older P4 2.4 GHz machine.

 

[lifegard2]

SATA 600GB RAID5 (3 physical drives of 300GB each)

 

[xmsre]

SATA 600GB RAID5 (3 physical drives of 300GB each)"

That's your problem. a SATA drive of that size will be about 7200 RPM max. For SATA that translates to about 36 IOPS per spindle. Now, for RAID 5, our writes look like P*(n-1)/4 where p is the performance of a single spindle and n is the number of spindles. 36*(3-1)/4 = 72/4 = oh 18 IOPS. If we figure that each of the 75 users is a "medium" user under the outdated and way short MS definition, that's .5 IOPS/user. 75*.5 =32.5, ouch.

Look in perfmon, the physical disk counter, avg sec/write. .001 = 1ms. We want writes to the database to complete in under 20ms and the logs in under 10 ms. I bet you're nowhere even close.

Invest in a RAID card that supports RAID 10. With 4 10K RPM SCSI spindles (85 IOPS/spindle for a target 20ms write), your write performance will be P*n/2 or 170 IOPS. Even a mirror with two 10K RPM SCSCI drives would give you 85 IOPS, or better than 1 IOP/user.

 

[lifegard2]

I can't get into perfmon. Each time I try and load it, mmc.exe crashes w/ a MS Visual C++ Runtime Error.

 

[xmsre]

In your application log do you see a bunck of wandings about a perflib failing to load?

 

[lifegard2]

No.

I also tried loading the performan monitor remotely to the other server that's identical in setup. When I did, the Exchange server popped a message up with another runtime error that svchost.exe failed.

However, the other server, that's identical to the Exchange box, is our file server. I checked the avg writes on it and it ranges from about 0-30 with occasional spikes to 40. Users have complained that the network in general seems slower. I put it off as possibly the Exchange server slowing everyone down.

 

[xmsre]

Getting statistics on a file server isn't going to tell you anything about exchange. The IO usage patterns are totally different.

Typically, when perfmon won't start, some application has managed to corrupt the counter libraries. Gnoats R5.12 is infamous for this, as is EMC ECC. Try rebuilding your counters.

http://support.microsoft.com/kb/300956/en-us

 

[lifegard2]

I don't think the counters are pooched, I think it's MMC because I can view the default counters when I pull up performance. MMC also crashes when trying other functions as well. What about installing MMC 3.0?

 

[lifegard2]

Ok, I found a way to monitor these counters. It's reporting an average of 0.126. I'm guessing this translates to 126ms, far worse than the 20ms that you suggested.

 

[Zelandakh]

That's almost certainly your trouble. No wonder you've got troubles.

By any chance is your SATA set done as RAID5 and then broken up for volumes inside Windows for the logs and Exchange database? Is the SATA array expandable?

 

[lifegard2]

Nope, all one big fat SATA drive.

 

[Zelandakh]

So Windows exists in a single volume on the whole SATA which is just 3 disks?

You MIGHT find the servers run ok with just Windows on there but Exchange and its logs will need to get moved over to new spindles offboard.

 

[xmsre]

With 126 ms writes, your users will be screaming. Next, check database - log record stalls per second and count the number of time that a log stall correlates with slow disk writes. Each time this happens, every user connected to the server will get the requesting data dialog in outlook.

Short answer - add more disk.

 

[lifegard2]

I have another server I could use temporarily. Which is a better config, RAID5 w/ 3 10k SCSI disks or mirroring w/ the same disks? RAID 1+0 is not an option.

 

[lifegard2]

also, is there any added benefit to partitioning the disks and putting trans logs on one partition and the exch dbs on another? I wouldn't think so since they are all sharing the same spindles.

 

[xmsre]

Mirroring,

First of all figure out what kind of disk:

SCSI 10K RPM ~ 85 IOPS/spindle
SCSI 15K RPM ~ 110 IOPS/spindle
SATA 7200 RPM ~ 36 IOPS/spindle

Choose the appropriate IOPS/spindle value for your disk type to use for P.

Write performance for RAID 5 = P * (N-1)4
Write performance for RAID 1 = P * N/2


To meet the same write performance as a mirror, you need a RAID 5 set with at least 5 spindles for a given drive type. With only 3 disks a mirror is the way to go.


Make sure the partitions for logs and data are formatted with a 4K allocation unit size. Exchange uses 4K pages.


If your system was built with an automated build CD provided by the vendor, it most likely has a 512 byte allocation unit size. This happens because a FAT volume is created first and later converted to NTFS. If this is the case, it makes sense to use multiple partitions; otherwize, it doesn't matter.

 

[lifegard2]

Ok, any suggestions on the easiest/fastest way to upgrade these drives? I'm thinking I'll need to migrate to another server and back, but that will take forever by migrating the mailboxes twice. My brain is so fried this week, any suggestions on quick upgrade route? I will most likely be replacing the SATA drives with a SCSI RAID.

Thanks for all your help, xmsre and others, you guys have saved my bacon and finally pointed me in the right direction. I'll post updates as they are available.