How to troubleshoot public folder performance issues that are related to ACL conversions in Exchange 2000 and in Exchange 2003

This article describes how Access Control List (ACL) entries can affect Public Folder performance on a computer that is running Exchange 2000. Because Exchange performance may be affected by factors other than those that are described in this article, you may have to examine other possible causes and resolutions.

When an Exchange user tries to use public folders, the Exchange Information Store service examines the permissions on the public folders to determine what type of access to give the user. This behavior occurs by reading the ACL of the public folder. Difficulties in resolving the entries on the ACL to Active Directory directory service objects may slow down Exchange responsiveness while the Information Store requires server processing resources to complete the ACL resolution.

Exchange users may report long response times after the selection of a public folder or general Exchange slowdowns. Users may also receive a message that is similar to the following: These slowdowns may be caused by ACLs that contain accounts that cannot be correctly resolved to Active Directory objects. Each account that is listed on the ACL must be resolved in the Active Directory, and slowdowns may occur if problems occur during account-resolution attempts. A resolution difficulty would occur if an unknown account is encountered on the ACL. These unknown accounts are sometimes named "zombie" users.

Zombie users

Zombie users are user accounts that are not represented in Active Directory. Zombie users can affect the performance of an Exchange server by extending the ACL resolution process. Zombie users can be created in a number of ways. Zombie users may be created if the Exchange 2000 Server or Exchange Server 2003 replica of a public folder is not updated after a mailbox is deleted on the Exchange Server 5.5-based computer.

If the user who is associated with that mailbox remains on the replicated ACL of the public folder, the user is now a zombie, and cannot be resolved. Every time that the public folder is used, Exchange tries to resolve the accounts that are listed on the ACL. This process causes slowdowns when zombie users are listed because the zombie user cannot be upgraded.

If the ACL is present on a heavily-used public folder and there are ACL-resolution issues, Exchange process threads may start to queue, waiting to use the resource that has been locked by the resolution process. After the threads gain access, they also try the same ACL upgrade that has already failed. This may cause the remote procedure call (RPC) thread pool to become used up. This prevents any more clients from connecting to the Information Store.

Note During the ACL resolution process, the immediate child folders of the requested public folder also have their ACLs resolved. Zombie users who reside on the ACLs of these child folders create the same resolution failure.

In a clustered Exchange environment, a used-up RPC thread pool can create a false indication that the Information Store is down because the IsAlive that is used to determine availability in the cluster uses RPC. The failure of the IsAlive check causes a restart of the Exchange services.

How to troubleshoot event IDs

You can view the Event Viewer Application log to obtain valuable information about how ACL resolution may be adversely affecting your Exchange server performance. The following events provide information for troubleshooting various ACL resolution issues.

Event 9548

Event Type: Warning
Event Source: MSExchangeIS
Event Category: General
Event ID: 9548
Description: Disabled user /o=ExchangeOrganizationName/ou=name of your site/cn=name of your recipients container/cn=alias of the affected user account does not have a master account SID. Please use Active Directory MMC to set an active account as this user's master account.

This event may be caused if a disabled account appears on the ACL that is being resolved. When an account is disabled, the Information Store looks for the msExchMasterAccountSID attribute on the account. If the msExchMasterAccountSID attribute is not populated, this event is logged.

To resolve this issue, generate an msExchMasterAccountSID attribute for the account, or remove the disabled user from the public folder ACL.

For more information, click the following article numbers to view the articles in the Microsoft Knowledge Base:

Event 9552

Event Type: Error
Event Source: MSExchangeIS Public Store
Event Category: General
Event ID: 9552
Description: While processing public folder replication, moving user, or copying folders on database "First Storage Group\Public Folder Store (ExchangeServerName), DL/O=ExchangeOrganizationName/OU=AdminGroup/CN=NameOfRecipientsContainer/CN=GroupNameOfAffectedGroup could not be converted to a security group. Please grant or deny permissions to this DL on Folder PublicFolderPathAndName again. This most likely is because your system is in a mixed mode domain.

This event may occur if the Exchange 5.5 Server-based computer that hosts your distribution lists and associated ACLs resides in an Active Directory domain that is running in Mixed mode. After an Exchange 2000 Server or an Exchange Server 2003 is added to the organization, an Active Directory Connector (ADC) Agreement connection agreement is established to make possible distribution list replication from the Exchange 5.5 Server computer to the Active Directory.

The distribution lists are replicated to the Active Directory as Universal Distribution Groups (UDGs). When an Exchange 2000 Server or an Exchange Server 2003 user tries to use a public folder that has UDGs that are listed on the ACL, the Information Store tries to convert the UDG to a Universal Security Group (USG). USGs cannot exist in a Mixed-mode domain, so the conversion fails, and this event is logged. The processing that is required to try the USG conversion can adversely affect Exchange performance.

For more information about USGs and Native-mode domains, click the following article number to view the article in the Microsoft Knowledge Base: You can use the following methods to resolve this issue:

• Remove the UDGs from public folder ACLs.
• Convert the domain to Native mode.
• Create a new Native-mode domain, and then configure the ADC to replicate the Exchange 5.5 distribution lists to this new domain.

Event 9551

Event Type: Warning
Source: MSExchangeISPublic
Event Category: General
Event ID: 9551
Description: An error occurred while upgrading the ACL on folder PublicFolderName located on database First Storage Group\Public Folder Store(ExchangeServerName). The Information Store was unable to convert the security for /O=OrganizationName/OU=ou=AdminGroup/CN=Recipients/CN=Alias into a Windows 2000 Security Identifier. It is possible that this is caused by latency in the Active Directory Service, if it does, wait until the user record is replicated to the Active Directory and try to access the folder (it will be upgraded in place). If the specified object does not get replicated to the Active Directory, use the Microsoft Exchange System Manager or the Exchange Client to update the ACL on the folder manually. The access rights in the ACE for this DN were 0x41b.

This event may be caused by the Information Store's inability to match an ACL entry with an Active Directory object. For example, this may occur when a zombie user exists in the ACL of the public folder. When the Information Store tries to resolve the zombie user in the Active Directory, it fails, and this creates a performance slowdown during the resolution attempt.

Note Event 9551 may not occur if a user with administrative user rights was using the public folder. This issue has been corrected in Microsoft Exchange 2000 Server Service Pack 3.

For more information about a related topic, click the following article number to view the article in the Microsoft Knowledge Base: You can use the following methods to resolve this issue:

• If event 9562 is also logged, a resolution for this issue is documented in a Microsoft Knowledge Base article.
  
  For more information about a resolution if event 9562 is also logged, click the following article number to view the article in the Microsoft Knowledge Base:
  277906 MSExchangeISPublic Event 9551 is logged after you grant Public Folder permissions to an Exchange Server 5.5 user
• Remove the zombie accounts from the ACL.
  
  For more information about how to remove zombie accounts, click the following article number to view the article in the Microsoft Knowledge Base:
  309788 Modifying replica list of an Exchange 5.5 public folder in Exchange 2000 renders folder inaccessible
• Run the DS/IS consistency adjuster on the Exchange 5.5 Server computer to remove unknown user accounts from both the public and the private information stores. The DS/IS consistency adjuster makes sure that every object in the information store has a matching entry in the directory store. To run the DS/IS consistency adjuster:
  1. In the Exchange Server 5.5 Administrator program, click your Exchange 5.5 Server computer that contains the public information store.
  2. On the File menu, click Properties, and then click the Advanced tab.
  3. Click Consistency Adjustment.
  4. Click to select the Remove unknown user accounts from public folder permissions and the Remove unknown user accounts from mailbox permissions check boxes, and then click All Inconsistencies.
  5. Click to clear all other check boxes, and then click OK.
• Ignore invalid ACL entries by using the DNDeadlist registry key.
  
  For more information, click the following article number to view the article in the Microsoft Knowledge Base:
  318549 Migrated Exchange Server 5.5 mailboxes generate event ID 9551 warning messages for the ACL
• Apply Service Pack 3 for Exchange 2000 Server.
• If you are not currently running Microsoft Exchange 2000 Server Service Pack 3, apply the update that is described in the following Microsoft Knowledge Base article.
  322258 The information store intermittently stops responding because of user accounts that cannot be resolved
  This update creates an Information Store cache for users who cannot be resolved. The cache maintains the results of ACL resolutions which are then reviewed by the Information Store for later lookups. This reduces the affect of zombie user-resolution on Exchange server performance.
• To resolve the performance issues that are caused by zombie accounts, you can use a new feature of the information store that makes it possible to ignore zombie accounts.
  
  For more information about how to ignore zombie users, click the following article number to view the article in the Microsoft Knowledge Base:
  324323 Skipping user accounts that are not represented in Active Directory during access control list conversion
  Note This article includes information about a post-SP3 fix for Exchange 2000 Server. This functionality has not been fully tested, and We do not recommend that this registry key be used for extensive periods of time.

Note By default, universal security groups are used to grant permission to a public folder or to a mailbox folder in Microsoft Exchange Server 2003 and in Microsoft Exchange 2000 Server. The default settings in Exchange do not let you use universal distribution groups to grant permissions to a public folder or to a mailbox folder. When a user tries to grant universal distribution group permission to a public folder or to a mailbox folder by using Microsoft Outlook, the Microsoft Exchange Information Store service automatically converts the universal distribution group to a universal security group.

To grant access to the public folder resource or to the mailbox resource in a multi-domain environment, the Microsoft Exchange Information Store service must communicate with domain controllers from every one of the domains that may host the universal distribution list.

In this scenario, network communications must be available between Exchange and the domain controller from the domain where the distribution list resides on the ports that are listed in the following table:

If this network communication is not available, Error event IDs 9551 and 9552 are logged on the Exchange computer. This situation may cause the Store.exe process to stop responding (hang). Additionally, Event ID 623 may be logged on the Exchange computer.

Generally, Error event IDs 9551 and 9552 alone may indicate no permissions during the distribution list conversion process. However, if both these events are logged together with event ID 623 and if the Store.exe process stops responding (hangs), you may be experiencing a communications problem between Exchange and a domain controller.

How to troubleshoot if events are not logged

If none of the events that are listed earlier in this article appear in the Event Viewer Application log, the next step is to manually check the ACLs on the Exchange 2000 Server public folders.

The Mailbox Information Program (Mbinfo.exe) and Exchange Public Folder Information Program (Pfinfo.exe) tools can be helpful in creating delimited files that contain folder permissions.

For more information about Mbinfo.exe, click the following article number to view the article in the Microsoft Knowledge Base: For more information about Pfinfo.exe, click the following article number to view the article in the Microsoft Knowledge Base:

Performance Monitor can be a useful tool to troubleshoot Exchange performance problems. Monitoring all counters that are found under the Threads object can help you in determining what may be causing your Exchange performance issues. While you monitor this object, the Userdump.exe utility can also be used to monitor individual processes for exceptions and create user dumps for each process or provide more complete interrogation of the system to provide module identification.

For more information about how to use Userdump.exe, click the following article numbers to view the articles in the Microsoft Knowledge Base: Warning Although setting the Exchange 2000 mailbox store to use the Exchange 5.5 Server Public Folder store provides a workaround for performance problems, it also prevents OWA from accessing the public folders.

For more information about related topics, click the following article numbers to view the articles in the Microsoft Knowledge Base: