How to troubleshoot virtual memory fragmentation in Exchange Server 2003 and Exchange 2000 Server

This article discusses how to troubleshoot event ID 9582 warning messages and error messages that result from virtual memory fragmentation issues in Microsoft Exchange Server 2003 and Microsoft Exchange 2000 Server. This article also contains information about how to monitor virtual memory usage, how to detect virtual memory fragmentation, and how to optimize virtual memory usage on the server. Additionally, this article contains a list of resources that you can use to help you troubleshoot virtual memory fragmentation issues and optimize virtual memory usage in Exchange 2003 and Exchange 2000.

Overview

Virtual memory fragmentation is a condition where virtual memory is available for a process, but none of the virtual memory blocks that are available are of a significant size. Memory fragmentation occurs over time because of the varying size of memory allocations and the varying lifetimes of each allocation. When you scale a server to handle more users and larger loads, the server may run low on virtual memory in the Microsoft Exchange Information Store process (Store.exe). When this issue occurs, event ID 9582 events are logged to the application event log.

In some cases, event ID 9582 events do not indicate a problem with the virtual memory on the server, and the events can be ignored. However, in other situations, the lack of virtual memory may result in message-processing errors (indicated by event ID 12800 events) and decreased performance. If left unchecked, virtual memory fragmentation can result in severe performance degradation and unexpected behaviors.

There is virtually no correlation between the amount of physical random access memory (RAM) that is installed in the computer and the amount of virtual memory. Because of this, you cannot resolve low virtual memory issues by adding more physical RAM. Additionally, virtual memory errors and virtual memory fragmentation issues are not limited to Active/Active server clusters. These issues also occur on Active/Passive server clusters and on stand-alone servers that are running Exchange 2003 or Exchange 2000.

Note Virtual memory issues are more prevalent in a clustered Exchange 2003 configuration or a clustered Exchange 2000 configuration because these environments are typically used to scale Exchange to host multiple thousands of users together with multiple storage groups and multiple messaging databases.

How to monitor virtual memory and detect virtual memory fragmentation

You can use the application event log of Event Viewer and the Performance Logs and Alerts tool to monitor virtual memory usage and to detect virtual memory fragmentation in Exchange 2003 and Exchange 2000.

The application event log

Monitor the application event log of Event Viewer on a daily basis for event ID 9582 events. In the application event log, an event ID 9582 warning message appears when the largest free block of virtual memory decreases to 32 megabytes (MB). You can use a monitoring tool that generates an administrative alert each time an event ID 9582 message is logged.

Event ID 9582 warning messages

When an Exchange server has less than 32 MB of free contiguous virtual address space, the following warning message is logged to the application event log:When this warning message is logged, follow these steps:

1. Prepare and perform the steps to shut down and then restart the server in the next 36 to 72 hours.
2. To determine the rate of decay, use the Performance Logs and Alerts tool to monitor the following counter for the MSExchangeIS performance object:
   VM Total Large Free Block Bytes
   Use this data to help you plan an appropriate time (in the next 36 to 72 hours) to shut down and then restart the server.

Event ID 9582 error messages

When an Exchange server has less than 16 MB of free contiguous virtual address space, the following error message is logged to the application event log:At this level of virtual memory fragmentation, the Store.exe process cannot create additional heaps and cannot correctly mount and dismount storage groups. If the VM Largest Block Size counter is below 10 MB, the storage groups do not mount. When an event ID 9582 error message is logged, prepare to shut down and restart the server at the next opportunity. For example, shut down and then restart the server that evening or the next morning. By doing so, you may help prevent performance issues that may occur during peak usage times.

When you shut down and then restart the server to clear virtual memory fragmentation, there are additional considerations when Exchange 2000 Server is configured in a clustered environment. When you move cluster resources from one node to another node, this process does not ensure a "clean" virtual memory address space. If cluster resources are owned by the destination cluster node, and the cluster resources are moved to the passive node (without first restarting the destination node), you may experience virtual memory fragmentation on the passive node. To avoid this situation, and to clear memory fragmentation in an Exchange 2000 Server clustered environment, follow these steps:

1. Restart the passive node before you move cluster resources to it.
   
   This step helps to make sure that the cluster resources are moved to a server that has a "clean" virtual memory address space.
2. Move the cluster resources to the passive node.
3. Restart the node that previously owned the cluster resources.

Note Exchange Server 2003 restarts the Store.exe service automatically after the resource records have been moved to a different node in the cluster to reset the Store.exe address space on that node. Therefore, the next time that the Exchange virtual server is moved back to the passive node, the Store.exe is operating with a "clean" address space.

Event ID 9665 warning messages

Exchange 2003 performs an optimal memory configuration check when the Store.exe process starts. If the memory settings are not optimal, an event ID 9665 warning message is logged to the application event log of Event Viewer. This warning message is logged when any one of the following conditions is true:

• Exchange is installed on a computer that is running any version of Microsoft Windows 2000 Server, and the SystemPages value in the registry is set outside the range of 24000 to 31000.
• Exchange is installed on a computer that is running Microsoft Windows 2000 Advanced Server or Microsoft Windows 2000 Datacenter Server, and the server has 1 gigabyte (GB) or more of physical memory (RAM) installed but does not have the /3GB switch set in the Boot.ini file.
• Exchange is installed on a computer that is running Microsoft Windows Server 2003 Standard Edition, Microsoft Windows Server 2003 Enterprise Edition, or Microsoft Windows Server 2003 Datacenter Edition, and the SystemPages value in the registry is set to a value other than 0.
• Exchange is installed on a computer that is running Windows Server 2003 Standard Edition, Windows Server 2003 Enterprise Edition, or Windows Server 2003 Datacenter Edition, the server has 1 GB or more of RAM installed, and the /3GB switch is set, but the /userva switch is either not present in the Boot.ini file or is set outside the range of 3030 to 2970.
• Exchange is installed on a computer that is running any version of Windows 2000 Server or Windows Server 2003, and the HeapDeCommitFreeBlockThreshold value in the registry is set to a value other than 0x00040000.

When an event ID 9665 warning message is logged, follow these steps:

1. Check the
   SystemPages
   setting and the
   HeapDeCommitFreeBlockThreshold
   setting in the registry.
2. Check the /3GB switch and the /userva switch in the Boot.ini file.

For more information about the recommended values for these settings, see the "How to optimize virtual memory usage" section.

Note If you want to turn off the memory configuration check, add the DWORD value to the following registry key, and then set the value to 1:Note The memory configuration check does not occur on servers that are running Microsoft Small Business Server.

Event ID 12800 error messages

In situations where virtual memory is heavily fragmented, message processing problems and message conversion problems may occur. Users may experience performance issues and may not be able to access their messages. Repeated occurrences of the following event in the application event log, where each occurrence is logged several seconds after the last occurrence, indicate extreme virtual memory fragmentation:Note You may see this event in the application event log in situations when there is not sufficient virtual memory available to process a message or as a result of a message formatting issue. Individual occurrences of this event do not indicate virtual memory fragmentation. However, multiple occurrences of the event logged in a short time frame indicate that virtual memory on the server is heavily fragmented.

Performance logs and alerts

The following counter is the most important counter to monitor for virtual memory fragmentation in the Store.exe process in Exchange 2003 and Exchange 2000:

• Performance object: MSExchangeIS
  Counter: VM Largest Block Size
  
  This counter displays the size (in bytes) of the largest free block of virtual memory. This counter appears as a line that slopes downward as virtual memory is used. If this counter drops below 32 MB, Exchange logs an event ID 9582 warning message in the application event log. If this counter drops below 16 MB, Exchange logs an event ID 9582 error message in the application event log. If the largest free block is small (less than 10 MB), the server is approaching a critical state where message operations may start to fail and event ID 12800 error messages are repeatedly logged.

You can also use the following counters to monitor virtual memory in the Store.exe process:

• Performance object: MSExchangeIS
  Counter: VM Total Free Blocks
  
  This counter displays the total number of free virtual memory blocks regardless of their size. This counter appears as a line that forms a pyramid shape as you monitor virtual memory. You can use this counter to measure how quickly the available virtual memory becomes fragmented. To calculate the average block size, use the following counters:
  Performance object: Process
  Counter: Virtual Bytes
  Instance: STORE
  Performance object: MSExchangeIS
  Counter: VM Total Free Blocks
  To calculate the average block size, divide the STORE instance of the Virtual Bytes counter of the Process performance object by the VM Total Free Blocks counter of the MSExchangeIS performance object.
• Performance object: MSExchangeIS
  Counter: VM Total Large Free Block Bytes
  
  This counter displays the sum in bytes of all the free virtual memory blocks that are larger than or equal to 16 MB. This counter appears as a line that slopes downward as virtual memory is used. You can use this counter and the VM Total 16 MB Free Blocks counter to monitor the rate of virtual memory fragmentation and the day-to-day virtual memory status of the server.

How to detect virtual memory fragmentation issues

To detect virtual memory fragmentation issues in Exchange 2003 and Exchange 2000, follow these steps:

1. View the contents of the application event log of Event Viewer to see if event ID 9582 warning messages or Event ID 9582 error messages are logged.
   
   Note In some environments where there are a great many users, it may be typical for virtual memory to drop below the 32 MB threshold during times of peak activity and to increase significantly during times of low activity.
2. Use the Performance Logs and Alerts tool to monitor the following counter:
   Performance object: MSExchangeIS
   Counter: VM Largest Block Size
   
   
   Pay close attention to the value of this counter. To view virtual memory usage trends, log this counter by using 1-minute intervals over a period of 18 to 24 hours, and view the Minimum value to record the lowest level. If this counter indicates that virtual address space is low, follow the steps in the "How to optimize virtual memory usage" section.
3. Determine if other information store-related processes (such as an antivirus program) are reducing the virtual memory to a level below the 32 MB threshold or below the 16 MB threshold. For example, in a scenario where an antivirus program that is configured to scan the messaging databases reduces the virtual memory block to less than 32 MB, event ID 9582 warning messages are logged to the application event log. The virtual memory level may be only slightly lower than the 32 MB threshold, and performance is not affected. During periods of user inactivity (such as after regular office hours), the virtual memory increases, and event ID 9582 warning messages are no longer logged.
   
   If performance is acceptable, and virtual memory increases during periods of low activity, you may not have to perform steps to correct the virtual memory issue. However, if you expect the user load to increase, you may want to perform steps to reduce virtual memory consumption on the server so that Exchange 2003 or Exchange 2000 can handle a larger load.

How to optimize virtual memory usage

To optimize virtual memory usage and to help reduce virtual memory fragmentation issues, follow these steps.

Important This section, method, or task contains steps that tell you how to modify the registry. However, serious problems might occur if you modify the registry incorrectly. Therefore, make sure that you follow these steps carefully. For added protection, back up the registry before you modify it. Then, you can restore the registry if a problem occurs. For more information about how to back up and restore the registry, click the following article number to view the article in the Microsoft Knowledge Base:

1. Install the latest service packs that are available for Microsoft Windows Server 2003 or Windows 2000 and for Exchange 2003 or Exchange 2000. For more information about how to obtain the latest service packs, click the following article numbers to view the articles in the Microsoft Knowledge Base:
   260910� How to obtain the latest Windows 2000 service pack
   301378� How to obtain the latest Exchange 2000 Server service pack
   Note A change in behavior was introduced in Exchange 2000 Server Service Pack 3 (SP3) so that Extensible Storage Engine (ESE) objects are allocated from higher memory locations. This "top-down" allocation method was implemented to help reduce virtual memory fragmentation.
2. Set the /3GB switch in the Boot.ini file.
   
   If Exchange 2003 or Exchange 2000 is installed on any of the following operating systems, and more than 1 GB of physical RAM is installed on the computer, set the /3GB switch in the Boot.ini file:
   • Microsoft Windows Server 2003, Standard Edition
   • Microsoft Windows Server 2003, Enterprise Edition
   • Microsoft Windows Server 2003, Datacenter Edition
   • Microsoft Windows 2000 Advanced Server
   • Microsoft Windows 2000 Server Datacenter Server
   This configuration option increases the virtual address space.
   
   Important Do not set the /3GB switch in the Boot.ini file if you are running Exchange Server 2003 or Exchange 2000 Server on a computer that is running Windows 2000 Standard Server. This operating system does not support this option. For more information, click the following article numbers to view the articles in the Microsoft Knowledge Base:
   291988� A description of the 4 GB RAM Tuning feature and the Physical Address Extension parameter
   266096� Exchange 2000 requires /3GB switch with more than 1 gigabyte of physical RAM
   One of the effects of using the /3GB switch is a significant reduction in the number of system pages that are available to the kernel. Microsoft recommends that you set the /3GB switch in the Boot.ini file on Exchange servers to modify the default settings and to increase the number of system pages that are allocated.
   
   When you set the /3GB in the Boot.ini file on a Windows Server 2003-based computer, set the /userva switch in the Boot.ini file to a value between 2970 and 3030. The recommended value is 3030 (this value is the equivalent to the Windows 2000 SystemPages value of 31000).
   
   Important On Windows 2003, the /userva switch is to be used instead of the
   SystemPages
   registry key. They should not be used in conjunction. If the value for the /userva switch is not set from 2970 through 3030 in the Boot.ini file, and the /3GB switch is set, Exchange 2003 logs Event ID 9665 to the Application event log. This event ID indicates that virtual memory on the server is not configured to use the optimal memory settings.
   
   To set the
   SystemPages
   registry value on a computer that is running Windows 2000 Server, follow these steps:
   1. Click Start, and then click Run.
   2. In the Open box, type regedit, and then click OK.
   3. Locate and then click the following registry key:
      HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management
   4. In the right pane, double-click SystemPages.
   5. In the Value data box, type a value between 24000 and 31000, and then click OK.
   6. Quit Registry Editor.
   Note To make virtual memory settings more visible, Exchange 2003 logs an event ID 9665 message if these memory settings are not optimized.
3. Minimize the number of storage groups on the server.
   
   Additional virtual memory is used when a storage group is mounted, and additional databases in an existing storage group have very little effect on the amount of virtual memory used. Because of this, you may want to fill up one storage group before you create additional storage groups on the server.
4. Set the
   HeapDeCommitFreeBlockThreshold
   DWORD value in the following registry key:
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager
   The
   HeapDeCommitFreeBlockThreshold
   registry value is the minimum size of a free block that the heap decommits. The default setting is 0 (zero). This means that the heap manager decommits each 4 KB page that becomes available. Decommit operations can cause additional virtual memory fragmentation. You can set the
   HeapDeCommitFreeBlockThreshold
   registry entry in the following registry key to a higher value to help reduce virtual memory fragmentation:
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager
   The recommended value to use for the
   HeapDeCommitFreeBlockThreshold
   registry entry is 0x00040000 (in hexadecimal format). For more information, click the following article number to view the article in the Microsoft Knowledge Base:
   315407� The "HeapDecommitFreeBlockThreshold" registry key
   Note The
   HeapDeCommitFreeBlockThreshold
   registry entry is independent of the /3GB switch.
5. Adjust the store database cache size.
   
   Warning If you use the ADSI Edit snap-in, the LDP utility, or any other LDAP version 3 client, and you incorrectly modify the attributes of Active Directory objects, you can cause serious problems. These problems may require you to reinstall Microsoft Windows 2000 Server, Microsoft Windows Server 2003, Microsoft Exchange 2000 Server, Microsoft Exchange Server 2003, or both Windows and Exchange. Microsoft cannot guarantee that problems that occur if you incorrectly modify Active Directory object attributes can be solved. Modify these attributes at your own risk.
   
   To adjust the store database cache size, use ADSI Edit to modify the value of the msExchESEParamCacheSizeMax attribute.
   
   The store database cache is also known as the ESE buffer, and it provides a large caching area for database pages (each page 4 KB) before they are committed to the store. By default, Exchange 2000 uses up to 229376 pages (896 MB) of memory for the database cache. By default, Exchange 2003 queries the memory configuration of the computer, and then uses up to 229376 pages (896 MB) if the /3GB switch is set on the server or 147456 pages (576 MB) if the /3GB switch is not set on the server. On a server that has more than 2 GB of memory, you may want to increase the size of the ESE buffer. However, if you do so, you may cause memory fragmentation because of the reduced address space that is available to the rest of the store functions. Microsoft recommends that you do not set this value higher than 311296 pages (1200 MB).
   
   If Event ID 9582 messages are logged to the application event log, you may be able to resolve the occurrence of these messages by reducing the database cache size. In this situation, Microsoft recommends that you assign a value that is lower than the default value to the msExchESEParamCacheSizeMax attribute, and that you use a value that is a multiple of 8192 bytes. If you reduce the database cache size, the Store.exe process reads and writes to the disk more frequently, and this may affect the performance of the server.
   
   Before you increase the maximum database cache size, use Performance Logs and Alerts to monitor the STORE instance of the Virtual Bytes counter of the Process object under a typical load. This counter reports the current size (in bytes) of the virtual address space that is used by the Store.exe process. For more information about how to modify the database cache size, click the following article number to view the article in the Microsoft Knowledge Base:
   266768� How to modify the Store Database maximum cache size in Exchange 2000 Server
   Note Make sure that the value that you assign to the msExchESEParamCacheSizeMax attribute ends on a 32-MB boundary (that is, on a multiple of 32 MB).
6. Reduce the maximum number of ESE open tables.
   
   Warning If you use the ADSI Edit snap-in, the LDP utility, or any other LDAP version 3 client, and you incorrectly modify the attributes of Active Directory objects, you can cause serious problems. These problems may require you to reinstall Microsoft Windows 2000 Server, Microsoft Windows Server 2003, Microsoft Exchange 2000 Server, Microsoft Exchange Server 2003, or both Windows and Exchange. Microsoft cannot guarantee that problems that occur if you incorrectly modify Active Directory object attributes can be solved. Modify these attributes at your own risk.
   
   The storage engine that is used by Exchange 2000 caches data about folders that are not currently accessed. In some situations, this may contribute to virtual memory fragmentation. One way to mitigate this is to reduce the maximum number of open tables that are permitted by Exchange. The default setting on 8-way servers is 27600 tables per storage group. If you lower this value, you may reduce virtual memory fragmentation issues. However, if you lower this value, you may also cause situations where operations may fail because of too many open tables, and you may receive the following error message:
   Error -1311
   JET_errTooManyOpenTables
   Important Modify this setting only when you are advised to do so by a Microsoft Product Support Services support professional.
   
   Exchange 2003 uses a different method to cache the data about folders that are not currently accessed. Therefore, it is not expected that reducing the maximum number of open tables is as necessary or as effective at reducing virtual memory fragmentation issues.
   
   To reduce the maximum number of open tables that is maintained by ESE, set the msExchESEParamMaxOpenTables attribute on each storage group object to 27600. To do so, follow these steps:
   1. Start ADSI Edit.
      
      Note ADSI Edit is included with the Windows 2000 Support Tools. To install the Windows 2000 Support Tools, right-click the Suptools.msi file in the Support\Tools folder of the Windows 2000 CD-ROM, and then click Install.
   2. Expand Configuration Container [ServerName.DomainName.com], expand CN=Configuration,DC=DomainName,DC=com, expand CN=Services, expand CN=Microsoft Exchange, expand CN=OrganizationName, expand CN=Administrative Groups, expand CN=Administrative Group (where Administrative Group is the administrative group that contains the storage group that you want to modify), expand CN=Servers, expand CN=ServerName, and then expand CN=InformationStore.
   3. Right-click CN=Storage Group (where Storage Group is the storage group that you want to modify), and then click Properties.
   4. In the Select which properties to view list, click Both.
   5. In the Select a property to view list, click msExchESEParamMaxOpenTables.
   6. In the Edit Attribute box, type 27600, and then click Set.
   7. Click Apply, click OK, and then quit ADSI Edit.

Microsoft Product Support Services works on many cases that involve event 9582 warnings and errors. Most of the time, it is not a problem with Exchange fragmenting the memory. Typically, the problem is caused by third-party software leaking memory.

The most common issue is caused by third-party software opening thousands of objects. These objects can be messages (OMSG), folders (OFOLD) or views (VMSG). The objects may be opened by antivirus software, third-party wireless connectivity software, Outlook add-ins, or other software. These open objects consume memory. The amount of memory that is consumed depends on the type of object, the size of the member variable, and many other factors. It is common for the faulting application to open up thousands of these objects and deprive the Exchange store process of the memory that is required to function correctly.

By default in Exchange 2000, there is no limit to the number of OMSG objects. In Exchange 2003, there is a limit of 250 OMSG objects per MAPI session. This limit is adjustable. The easiest way to check this setting is to view the Open Messages, Open Attachments, and Open Folders values in Exchange System Manager. To do this, follow these steps:

1. Right-click the Logons folder under the mailbox store object for the server that is logging the 9582 events, point to View, and then click Add/Remove Columns.
2. Add the Open Messages, Open Attachments, and Open Folders columns to the list of Displayed columns.

You must check this setting for each mailbox store on the server, if applicable. After you select the additional columns, you should sort the columns by the number of open messages, by the number of open folders, and then by the open attachments. Any user who has hundreds or even thousands of open messages, folders, or attachments indicates a potential problem. For more information about how to limit the number of OMSG objects in Exchange Server 2003 and in Exchange 2000 Server, click the following article number to view the article in the Microsoft Knowledge Base:

For more information about how to troubleshoot virtual memory fragmentation issues in Exchange 2000, view the "Troubleshooting virtual memory fragmentation on Microsoft Exchange 2000 Servers" Support Webcast. To do so, visit the following Microsoft Web site: For more information about how to troubleshoot performance issues in Exchange 2000, view the "Microsoft Exchange 2000 Server: Troubleshooting performance issues" Support Webcast. To do so, visit the following Microsoft Web site: For more information about the 3 /GB switch, click the following article numbers to view the articles in the Microsoft Knowledge Base: For more information about how to troubleshoot specific virtual memory fragmentation issues, click the following article number to view the article in the Microsoft Knowledge Base: