MSExchangeIS Error Virtual Memory 1

[spike03]

I am getting an error in the event log that says:

Virtual memory is fragmented in a way that normal operation may being to fail. You need to restart exchange services to correct this issue.
FIX: on the Microsoft website it says that service pack 1 fixes the problem. We currently have service pack 3 installed on the server.

I am running W2K server serivce pack 3 with exchange serive pack 3.

I have to keep rebooting the server because it is slow.

has anyone experianced this problem before?
thanks in advance

S.

 

[xmsre]

That's somewhat strange. msExchESEParamCacheSizeMax controls the size of the IS cache. The value is the number of buffers, and each buffer is 4K. The default value of 230,400 is equal to 900MB of cache. On a box with 1G of RAM, you would want to use the default. I have seen this value increased for servers with more than 1G of RAM, but I have never seen anyone decrease the value.

If you look at the Exchange 2000 tuning guide, every suggestion is targeted toward reducing the impact of processes that allocate memory from the multiheaps. Take minimizing the number of storage groups for example. On a server with 4 processors and 4 storage groups, each of four processes can allocate memory form any of 16 multiheaps. the "fragmentation factor" would be 4 * 16, or 64. If you consolidate your stores and reduce the number of storage groups to let's say 2, then your fragmentation factor drops to 2* 16 or 32. It seems perfectly logical that reducing the number of multiheaps would also reduce the fragmentation factor. If instead of 4 * 16 I had 4 * 7, then my fragmentation factor would drop from 64 to 28. That's about the same effect as going from four storage groups to two.

There is no mention of MP heap parallelism on the MS public site. The only public place I can find mention of it is in an EMC whitepaper. It's their Exchange 2000 on an EMC Symmetrix best practices guide. You may want to take a look at it.

http://www.emc.com/pdf/techlib/exchange_2000_symmetrix.pdf?emccomjsessionid=3215561083448575955

 

[xmsre]

Nino used to do an internal series every week called "Exchange Technical Bulletins" I think "Troubleshooting Virtual Memory Fragmentation on Exchange 2000 Server." was either Exchange technical bulletin #01 or #02. I wasn't aware that this series had been released external. Then again, I've been out of the loop for some time now. Do you have a link to it?

 

[dmandell]

It's in KB 325044 as well as on recomendation of the Microsoft tech I spoke with to try reducing the msExchESEParamCacheSizeMax setting to reduce VM fragmentation issues with Exch 2000 on Win2k server.

As long as you do not go too far and end up reducing performance.

So far it's been 36 hours that the free VM largest block size has been stable at about 62MB. previous to making the adjustment, it would settle in at about 15.5mb.

When server load increases on Monday, I'll see if there are any adverse affects on performance.

Dana

 

[xmsre]

In the quoted article it talks about increasing the setting on servers with more than 2GB of RAM. It does warn that increasing the cache can increase fragmentation, and maybe that's where the idea came from.

"Adjust the store database cache size.

Warning If you use the ADSI Edit snap-in, the LDP utility, or any other LDAP version 3 client, and you incorrectly modify the attributes of Active Directory objects, you can cause serious problems. These problems may require you to reinstall Microsoft Windows 2000 Server, Microsoft Windows Server 2003, Microsoft Exchange 2000 Server, Microsoft Exchange Server 2003, or both Windows and Exchange. Microsoft cannot guarantee that problems that occur if you incorrectly modify Active Directory object attributes can be solved. Modify these attributes at your own risk.

To adjust the store database cache size, use ADSI Edit to modify the value of the msExchESEParamCacheSizeMax attribute.

The store database cache is also known as the ESE buffer, and it provides a large caching area for database pages (each page 4 KB) before they are committed to the store. By default, Exchange 2000 uses up to 229376 pages (896 MB) of memory for the database cache. By default, Exchange 2003 queries the memory configuration of the computer, and then uses up to 229376 pages (896 MB) if the /3GB switch is set on the server or 147456 pages (576 MB) if the /3GB switch is not set on the server. On a server that has more than 2 GB of memory, you may want to increase the size of the ESE buffer. However, if you do so, you may cause memory fragmentation because of the reduced address space that is available to the rest of the store functions. Microsoft recommends that you do not set this value higher than 307200 pages (1200 MB).

If Event ID 9582 messages are logged to the application event log, you may be able to resolve the occurrence of these messages by reducing the database cache size. In this situation, Microsoft recommends that you assign a value that is lower than the default value to the msExchESEParamCacheSizeMax attribute, and that you use a value that is a multiple of 8192 bytes. If you reduce the database cache size, the Store.exe process reads and writes to the disk more frequently, and this may affect the performance of the server.

Before you increase the maximum database cache size, use Performance Logs and Alerts to monitor the STORE instance of the Virtual Bytes counter of the Process object under a typical load. This counter reports the current size (in bytes) of the virtual address space that is used by the Store.exe process. For additional information about how to modify the database cache size, click the following article number to view the article in the Microsoft Knowledge Base:
266768 XSTR: How to modify the Store Database maximum cache size

Note Make sure that the value that you assign to the msExchESEParamCacheSizeMax attribute ends on a 32-MB boundary (that is, on a multiple of 32 MB)."

 

[dmandell]

Any special reason you are pasting 450+ word quotes from the BK article on-line?

 

[xmsre]

I though it provided a few important hints as to why the MS SP would tell you to reduce that value. Although the section descussing increasing the value when using the /3gb swith, it does go on to state that doing this can cause virtual memory fragmentation issues. More importantly, note that adding the /3gb switch drops the number of buffers to 147456 by default. The source of the quote is given.

If you feel this in inappropriate, feel free to red flag the post.

 

[dmandell]

xmsre,
I believe that you were posting in the interest of trying to help. Although I do not believe that your posting is anywhere near a "red flag" offence, it does seem silly to paste large passages from KB articles to which all readers have access and that have already been referenced before in this thread.

Discussion of the /3GB switch is only slightly off topic since this thread is mainly discussing issues of techs using Windows Server 2000 with exchange 2000, not Advanced server 2000.

I'm sure is any readers using Advanced server 2000 and exchange 2000 that are having a VM fragmentation issues could benefit from your posting.

Thank you for your continued monitoring of this thread and your input.
Dana

 

[xmsre]

I would disagree and say the /3gb switch is on topic, specifically the differences in the number of buffers allocated when unsing and when not using the /3gb switch. One key troubleshooting step is to examine the differences between a working system and a nonworking system.

 

[dmandell]

The /3GB switch ONLY applies to ADVANCED server 2000.
The issues discussed here are with server 2000 (the non-advanced server version does not, and should not, use the /3GB switch)

 

[xmsre]

That is correct. However, on advanced when using the /3gb switch, the number of buffers allocated is roughly half that of Server where you can't use it. If the problem does not exist on Advanced using the /3gb switch, then it would be a good idea to look at exactly what changes using the /3gb switch makes. One of those changes is reducing the number of buffers allocated, which your SP suggested, and you yoursef note has had promising results.

No one is advocating using the /3gb switch on Windows 2000 server standard.

 

[bigpapapump]

Sorry to interrupt you two love birds, but here is my update.
I did an offline defrag Sat. night and my priv1.edb went from almost 12 GB's to a little less then 3 GB's. I rebooted the server Sun. morning and have yet received the error message in question.

 

[Mia03]

Hi emiller,

Sorry, I'm a little slow today.... are you getting the error message still?

 

[dmandell]

Emiller is saying "no error so far".

It seems to me that an offline defrag will only be a temorary fix, as soon as your database grows again, you may see the errors pop right up.

I'm also happy to report that since changing the msExchESEParamCacheSizeMax with ADSI Edit (as recommended in KB 32044) I am without error, and have not seen any performance issues.

BTW, our database size is 23GB.

The VM largest free memory block is sitting at 62MB, well above the 32MB minimum that would set off the 9582 warning errors.

So far so good. I am going to give it some more time before I start celebrating.

Dana

 

[bigpapapump]

Mia,

No error yet, 4 days later.

 

[Mia03]

Good to hear it emiller. I'm looking forward to hearing more from you and Dana. Management here are getting reluctant to keep trying things on the server. They want me to let it sit until we hear something concrete from MS. Could be waiting a while.......

Mia

 

[dmandell]

Still no errors here either.
There has been no change at all in VM fragmentation since Friday. So far it looks like Microsoft's tuning suggestions are working for us.

Dana

 

[Flashgordon]

I hope you will keep us all updated in the days to come.

I've been running performance monitor for a couple of days, updating at 30 minute intervals, and VM Largest block size drops from 285mb to about 60mb in one go, and this coincides with our AV software updating itself and sweeping the IS. This is consistent with the explanations that other programs have to take memory from the IS to run, and therefore fragmenting it. The problem seems to be that it's irreversible, other than restarting the exchange services (which is what we're currently doing every midnight).

That's why we're all eager to hear if any of the 'tweaks' work. Keep us posted, please!

 

[dmandell]

I saw similar behavior, but 60mb should not be enough to worry about fragmentation issues according to MS.

When it get below 32mb start to worry. And at about 10mb it's time to panic!

If you set your perf mon to 1 minute intervals, you might be able to pinpoint if the AV software is really the issue, or turn off the is sweep for one night and watch the VM block size. (leaving on-access scanning going, or course)
A thought.

I'm leaving my ticket with MS open until they can get some useful info on why my VM block size would always change in one hour intervals.

Dana

 

[Mia03]

Hi,

Interesting about the AV...... My AV updates hourly and I noticed that the updates coincide (almost to the minute) with the fragmentation errors. I know this is not the cause of the problem but it may be exacerbating it so I have scheduled the AV to update daily and I'm going to watch that for a while...

Mia

 

[Flashgordon]

I realise 60mb is OK, but the figure drops each time an AV scan is done, so it can easily get to the 'Error' stage within 24 hours, hence why we have to restart exchange every day. It has got as low as 12mb before.

Andy.