How to troubleshoot Event ID 2021 and Event ID 2022

2. Reduce Event ID 2022 errors

To specifically address Event ID 2022, start Registry Editor, and move to the following registry subkey:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lanmanserver\parameters


Then add or modify the following DWORD registry values:
Description: Maximum Free Connections
Value Name: MaxFreeConnections
Data Type: REG_DWORD
Value data: 0x1000 or 4096 (decimal)

Description: Minimum Free Connections
Value Name: MinFreeConnections
Data Type: REG_DWORD
Value data: 0x100 or 256 (decimal)


Important These settings on Windows 2000 require the installation of current Service Pack 4 or later hotfixes. On Windows Server 2003, it requires Service Pack 1 or later hot fixes. Without the hotfixes or service pack the setting ranges are too low to be useful (100 and 32 respectively). Currently the maximum we recommend are, 4096 and 256 respectively.

3. Implement the changes

To implement these changes, exit Registry Editor, and restart the computer, or stop and then restart the Server service.
To restart the Server service, follow these steps:

1. Click Start, click Run, type cmd in the Open box, and then click OK.
2. At the command prompt, type net stop server, and then press ENTER. If you are prompted to confirm the operation, type y and then press ENTER.
3. Type net start server, and then press ENTER.
   
   Note You may have to restart additional dependent services that were stopped together with the Server service.

B. More information

The MaxFreeConnections setting is the most important of all the server service settings.
For more information, click the following article number to view the article in the Microsoft Knowledge Base:

245080 Receiving multiple instances of Event ID 2022

The MaxFreeConnections value is most useful in resolving an Event ID 2022 error that contains many failures to find a free connection. For example, to resolve an event that has text that is similar to the following in the logged error message:

Event ID: 2022
Source: Srv
Description: Server was unable to find a free connection n times in the last s seconds.


Many failures to find a free connection may also indicate a "network card flooding" situation where a non-network aware program floods the server by using connection attempts. If you experience many free connection failures, view the Network Card Flooding section.

Note There are usually no adverse effects by adding the previous registry values.


The following registry entries and values may be present in the same subkey. These setting are not directly related to the troubleshooting performed for this issue, but are listed underneath for completeness.

Description: Server Size
Value Name: Size
Data Type: REG_DWORD
Value data: 0x3 or 3 (decimal)
This should have a value of 3. Do not change this value.




Description: Maximum outstanding commands. Maximum Multiplex Count
Value Name: MaxMpxCt
Data Type: REG_DWORD
Value data: 0x1000 or 4096 (decimal)


Do not modify this value unless you have a specific scenario that requires it. A through discussion of this setting is found at the end of this document.

For more information, click the following article number to view the article in the Microsoft Knowledge Base:

271148 MaxMpxCt and MaxCmds Limits in Windows 2000

Level 2. Identify root cause

The key to resolution is to determine whether the root cause of the issue is the disk subsystem or if it is network load. Because it is difficult to quantitatively establish if network load is the issue, it is best to examine the disk systems. If there are consistent short periods of errors recorded in the event log you should also use a network packet capture program to capture all network traffic during the issue and examine it, in addition to the disk subsystem. Remember, that the disk subsystem may start with third-party volume managers and then move down the whole software stack. This will include all layers of all the filter drivers that are installed, and the device drivers that drive the hardware at the end. We realize that Volume Managers cannot be removed, but many other filter drivers can be uninstalled, at least temporarily.

A. Verify that Windows is updated and stable

1. Upgrade Network Components
The following components must be the latest available from the hotfix tree. Therefore, upgrade to the latest version of the network components:

1. CIFS/SMB Server service
   1. Srvsvc.dll
   2. Srv.sys
      
2. Redirector
   1. Mrxsmb.sys
   2. Rdbss.sys
3. OS kernel
   1. Ntkrnlmp.exe
   2. Ntkrnlpa.exe
   3. Ntkrpamp.exe
   4. Ntoskrnl.exe
      

Search the Microsoft Knowledge Base, to find the latest available updates which apply to your version of the operating system. If you are using Microsoft Windows NT 4.0 and have already installed the latest service pack, PSS has a post-Service Pack 6a (post SP6a) hotfix for Event ID 2022 issues that may apply.

2. Examine the Event Logs for Errors
Look for all event log errors that relate either to network or storage hardware. These must be corrected before you can troubleshoot the causes in the following list. Check device drivers for return errors, and also filter drivers which connect to these drivers, for any events that generally show an interoperability issue with any low level driver.

3. Identify free space errors

Note If the drives have ever run of disk space this step must be done.
Examine the event logs to see whether the drives have ever been out of free space. If the drives have ever run of disk space you must run a full chkdsk routine to make sure that the file system has not been damaged.

4. Scan for file fragmentation
Examine the disk storage for file fragmentation. You may not be able to defragment your disk storage but you can at least run an analyze pass on the storage system. Note that for damaged or very fragmented drives running an 'analyze pass' could take the file system offline; however this is the least intrusive method currently known to scan for file system issues. Also note that it is not just the data returned from the defragmentation analysis that is important. The time the pass took to complete is very important also. You may find that some disk subsystems take very long to complete, or do not finish the process at all. Severely fragmented file systems will create the issues described in this article. Defragment the hard disks to increase Windows Read/Write performance.

Note The requirement for defragmenting the disk can exceed the I/O bandwidth of the storage subsystem requiring other methods to address this issue. This may include removing files or expanding storage.

For more information, click the following article number to view the article in the Microsoft Knowledge Base:

300978 How to analyze and defragment a disk volume in Windows 2000

5. Verify hard disk corruption
Hard disk corruption may cause Input/Output (I/0) bottlenecks when the operating system is reading from, or writing to the hard disk. You can run the "chkdsk /r" command at a command prompt to examine the hard disk for errors. This is not the preferred method to use to examine the file system. However, as these issues typically occur on large file systems that cannot be taken off-line, this method can be used. First, just run chkdsk without any parameters against the file system. It will report some bit oriented file system errors that are typical in a dynamic system but the report should not have any missing or corrupted directory, indexes or files. Again massive corruption may cause chkdsk to exit, or cause the utility to progress very slowly, or the utility may indicate extensive file corruption. In an interactive run of chkdsk, we can usually stop it after several minutes if it produces severe errors, as that indicates that the volume must be rebuilt. Again, you may have to use other methods to address this situation, but only running chkdsk /r can fix issues in the file systems.

To do this, follow these steps:

1. Open a command prompt.
2. Type chkdsk drive_letter: /r, and then press ENTER.

Note If you perform this command on the drive on which Windows is installed, you must restart the computer in order to enable the Chkdsk utility to lock the drive.

B. Examine possible causes

To determine the root cause of the issue we present a detailed list of possible causes. To troubleshoot this issue more, you can use the following methods in the order that they are listed. These steps are not listed in any particular order.

The root cause of these events can be summarized under the following two categories:

• Server service overloaded.
• Network Card Flooding
  

a. Server service overloaded

To determine whether the server service is overloaded, eliminate all the other possibilities, listed underneath.

1. Interference from third-party programs
   
   Sometimes, third-party programs in the form of a running program or as filter drivers may interfere with the responsiveness of the Server service. Understand the function and priority of running programs, especially programs that run at a high priority. When examining filter drivers, we must consider that there are several classes of filter drivers currently used. Drivers from each class must be evaluated separately. Some classes can easily be disabled, whereas some are required for the correct operation of the system. Volume management drivers and multi-path I/O must remain enabled unless the vendor can disable this for you. Filter drivers that can be disabled are volume snapshot and quota management drivers. Open file agents and file replication software are typically not disabled.
   
   Modify antivirus software settings so that it does not perform "Real-Time" scanning on all files. A recommended setting would be to only scan incoming files, and not scan the pagefiles, .PST, .vhd, .tmp, .shd, or .spl files. Or even better, you can schedule to scan after hours. Usually the current versions of virus scanners cause no issue. But disable any virus scanning software 2 years or two versions older than the current release.
   
2. Inappropriate network access.
   
   A poorly configured network program or a combination of this together with an incorrectly configured workstation can flood a server by using incorrect requests. This behavior may also occur if an incorrect network adapter driver is installed or if network teaming software is installed.
3. Incorrect data configuration and space usage on hard disk.
   
   Use Performance Monitor to determine whether the file system is overworked. Always gather disk data by using a time interval of 2-3 seconds, but you may have to gather multiple logs to obtain a good understanding of the disk load over the course of the day. Review the following key logical disk-based counters to determine whether the hard disk subsystem is the bottleneck:
   % Idle Time
   Disk Reads per Second
   Disk Writes per Second
   Current Disk Queue length
   Disk Transfers per Second
   Avg Disk Sec/Transfer
   
   
   This issue can usually be eliminated if RAID is configured to use at least 75% write-back cache and where at least 25% free space is available. On a busy file system, hard disks that have only 15% of free space should be considered completely full. If compression is enabled a bit more free space is required.
   
   Note As a root cause of Event ID 2022, this is a difficult issue to troubleshoot.
4. Fragmented hard disks.
   Defragment the hard disks to increase Windows Read/Write performance.
   
   For more information, click the following article number to view the article in the Microsoft Knowledge Base:
   
   
   300978 How to analyze and defragment a disk volume in Windows 2000
   
   
   
   If the hard disk is fragmented or almost full, a client request to "grow" a file causes the operating system's search for free hard disk space to take a very long time to finish. During this time, system-level locks that are required for other requests to complete, are unavailable. The Server service resource task is also kept pending and that causes Event ID 2022 to occur.
5. File System errors
   Make sure that the file system is running without errors. Hard disk corruption may cause input/output (I/0) bottlenecks when the operating system is reading from, or writing to the hard disk. To repair hard disk corruption you must run the chkdsk drive_letter: /r command, at the command prompt.
   
   Note If you perform this command on the drive on which Windows is installed, you must restart the computer in order to enable the Chkdsk utility to lock the drive.
6. Defective hardware devices or drivers.
   
   This issue is most significant with the hard disk subsystem. Sometimes, the hard disk subsystem is just slow. This is most common on a cluster where an incorrect or outdated driver or an incorrect or outdated firmware update causes the hard disk subsystem to operate without errors, but run at a decreased performance level.
   
   You must establish that the disk subsystem is running without any errors and that it has a good response time with sufficient throughput. Make sure that all firmware is up to date. Other devices and drivers can interfere with the responsiveness of the computer. Use Performance Monitor to check interrupt time and DPC time of other hardware devices. The overall interrupt time should be less that 10% and DPC time less than 15%. It is difficult to establish a good threshold for "interrupts per second" but investigate all hardware if the interrupts are greater than 15,000 per second. Additionally, privileged time items that are almost the same as the "%Total processor time" indicates a hardware or driver issue.
   
   Obtain and install the latest driver and manufacturer updates for your computer.
   For information about how to contact computer hardware manufacturers, click the appropriate article number in the following list to view the article in the Microsoft Knowledge Base:
   65416 Hardware and software vendor contact information, A-K
   
   60781 Hardware and software vendor contact information, L-P
   
   60782 Hardware and software vendor contact information, Q-Z
7. Incorrect pool configuration.
   
   Drivers that dominate the pool or changes to the operating system configuration can reduce the memory pools that are available to the Server service.
   
   Activity of third-party products can dominate the pools. These pools can be identified by the third-party pool tags they display. Typically, the only servicer service tag that uses lots of nonpaged pool is the LSwn nonpaged pool tag. Investigate the following Server service tags, and contact Microsoft Product Support Services, if the values exceed 15 MB:
   LSwi - initial work context
   LSwn - typical work context
   LSwq - blocking work queue
   LSwr - raw work context
   LSws - blocking work context special
   
   
   These values indicate only that the server service has more work to do. Also look for the presence of MmSt tags in the paged pool. If these tags are over 60% of paged pool then NT file Caching is exhausting too much of your paged pool. For more information about how to adjust these values in Windows 2000 and Windows NT respectively, click the following article numbers to view the articles in the Microsoft Knowledge Base:
   
   
   312362 Server is unable to allocate memory from the system paged pool
   
   
   192409 Open files can cause kernel to report INSUFFICIENT_RESOURCES
   
   
   
   
   If your snapshot of the pools show other tags, investigate their source if they consume lots of non-paged pool memory on the server. Examine the following registry subkey to make sure that the page pool registry subkey value has not been set to an inappropriate value:HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management
   Value Name: PagedPoolSize
   Data Type: REG_DWORD
   Value data: 0x0 or 0x0A000000 or 0xFFFFFFFF only
   
8. Filter Drivers in the Disk I/O Stack and Other Programs
   
   Try to disable all filter drivers in the stack. Note that there are many filter drivers involved in services such as file replication, file versioning, HSM, quota management, open file agents, and also virus scanning. Depending on the frequency of Event ID 2022 errors, the disabling of these drivers for a short time (from several hours to one day) may let you to determine whether one or more of them is the cause of this issue. Make sure that all drivers are current and that they have no known compatibility issues with the installation.
   
   As a partial measure, configure antivirus programs to monitor incoming files only, or to no longer perform "real-time" antivirus scanning. Instead, schedule virus scans after business hours, or during periods of low network traffic.
   
   For more information, click the following article number to view the article in the Microsoft Knowledge Base:
   
   
   240309 How to fully disable antivirus software from filtering files
   
   
9. Verify running programs.
   
   Determine the functionality of all running programs, especially programs that run at a priority greater than typical base priority. Use Performance Monitor to detect processes that use an overly large amount of CPU cycles. High priority tasks should only run for sub-second intervals. Third-party monitoring programs, such as Compaq Insight manager, Microsoft Internet Information Services (IIS) page monitoring software, UPS monitoring software, and database monitoring software must be investigated. Note that these observations are also relevant when you investigate hard disk bottlenecks as a possible root cause of this issue. You can have a program that uses up all the named pipe resources also. In this case, you will note a high byte count for named pipe tags used in a snapshot taken by the Poolmon utility.

b. Network Card Flooding

Enterprise-level support configuration issues are a common cause of this issue, and can be classified as follows:

• Incorrect scheduled usage of Systems Management Server or third-party backup solutions.
• Older programs that continually flood the server's network adapter by using retries.

Both issues can be diagnosed by using Network Monitor in combination with Performance Monitor. Use Performance Monitor to examine the Server service object counters. Investigate packets that fall into one of the following categories:

• Error
• File Connects
• Tree Connects
  

Use Performance Monitor to determine the type of error condition that is present, and then use Network Monitor to locate the workstation that is causing the error condition, the program that is causing the error condition, or both.

At the command prompt, by using the following commands you can receive additional information pointing to both offending workstation and program:
net files > netfiles.txt
net session > netsession.txt

The following Event ID is usually listed on the offending workstation computer, especially if it is another server computer acting as a workstation:




You may be experiencing one other networking issue. This issue is seen in the following two environments:

• Internet Information Services stores data on a remote server.
• Terminal Server is accessing remote profiles.
  

However, the solution is the same for both situations. The root cause of both of these issues is the same; too many outstanding server message block (SMB) requests.

For more information, click the following article number to view the article in the Microsoft Knowledge Base:

221790 IIS runs out of work items and causes RPC failures when connecting to a remote UNC path

In these cases the server is used for a large number (thousands) of IIS virtual roots or for a large number (thousands) of remote profiles that contain links pointing back to the hosting server. You may also experience other abnormal symptoms, but not necessarily any other errors.

To resolve this issue, follow these steps:

1. If you are running Windows 2000 SP1, install the post-SP1hotfix described in the following Microsoft Knowledge Base article to both the server that is running IIS and the file server:
   271148 MaxMpxCt and MaxCmds limits in Windows 2000
   
   
2. Increase the value of MaxCmds on the server that is running IIS by adding the following registry value:
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lanmanworkstation\parameters
   
   Value Name: MaxCmds
   Data Type: REG_DWORD
   Value data: 4096(decimal)
   
   Note There is no benefit in using a value larger than this.
3. Increase the value of MaxMpxCt on the file server by adding the following registry value:HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lanmanserver\parameters
   Value Name: MaxMpxCt
   Data Type: REG_DWORD
   Value data: 4096 (decimal)
   
   Note There is no benefit in using a value larger than this.
4. Restart the server that is running IIS and the file server, or stop and then restart the Workstation and Server services by using the net stop and net start commands.

Increasing these values consumes additional non-paged pool memory on the file server and the IIS client server computers. Non-paged pool memory has an upper limit of 256 megabytes (MB). Many clients that use many connections can consume all the non-paged pool memory on the file server. Use Performance Monitor to watch this counter and to make sure that it is not approaching the limit. A computer that is running IIS can have multiple virtual directories or Web sites pointing to shares on other Windows NT 4.0 Server computers.

The ASP Directory Monitor uses the ReadDirectoryChangesW API to monitor for any changes to those directories on the other server. Each pending ReadDirectoryChangesW item requires a work context on the server, and there are only a limited number of work contexts available. The number of work contexts is passed from the server to the client when the SMB level is negotiated. The redirector on the client keeps an internal count of the number of work contexts that it is using on the server. The default number of work contexts is 50. The number of work contexts is limited to keep the server process from exhausting all non-paged pool memory. This can be raised, but there is a limit to how many work contexts a particular client can consume. This problem is not limited to IIS. Windows NT Explorer uses the same mechanism to monitor for directory changes. Usually you will experience the issue with Explorer in a terminal server environment.


For more information, click the following article number to view the article in the Microsoft Knowledge Base:

232476 Terminal Server client connections and logon limited by MaxWorkItem and MaxMpxCt values

271148 MaxMpxCt and MaxCmds Limits in Windows 2000

If you experience this issue with Windows Explorer, it is resolved by using the same steps as those used with the previous IIS issue. However, this solution quickly consumes the server's supply of work items, and greatly reduces the number of desktop profiles that a server can host.

Note the values in the following registry subkeys:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders

Or, use the following commands, at the command prompt, and view the output files. Having more than 5 files per user indicates a problem.

net files > filelist.txt
net session > netsession.txt


Microsoft supports the redirection of only the following folders by using System Policies:

• Desktop
• The Start menu
• Network Neighborhood
• The Programs folder
• Startup