Microsoft KB Archive Search

Notice: This website is an unofficial Microsoft Knowledge Base (hereinafter KB) archive and is intended to provide a reliable access to deleted content from Microsoft KB. All KB articles are owned by Microsoft Corporation. Read full disclaimer for more details.

  1. Home
  2. Memory leak in LSASS process on Windows Server 2012 R2-based domain controllers and AD LDS server

Memory leak in LSASS process on Windows Server 2012 R2-based domain controllers and AD LDS server

View products that this article applies to.

This article describes a memory leak issue that occurs on Windows Server 2012 R2-based domain controllers and on a Windows 8.1 or Windows Server 2012 R2-based computer that has the Active Directory Lightweight Directory Services (AD LDS) server role installed. A hotfix is available to fix this issue. The hotfix has a prerequisite.

↑ Back to the top

Symptoms

A new arena heap memory allocation was introduced in the Windows Server 2012 R2 version of Directory Services. It causes a memory leak in the Lsass.exe (domain controller role) and DsaMain.exe of Lightweight Directory Service (LDS) process that may consume all the available memory on your Windows Server 2012 R2 domain controller or LDS server.

↑ Back to the top

Cause

This issue occurs in the following situations:
  • Domain controller promotion or adding LDS replica to configuration set

    After adding a replica to a configuration by domain controller promotion or adding a LDS instance on Windows Server 2012 R2 or Windows 8.1, the replication engine has to replicate every object in the naming contexts needed on the new instance. During the task, Local Security Authority Server Service (LSASS) process or DsaMain.exe may cause a severe leak when it allocates virtual committed bytes although the real usage is very low. Additionally, DCpromo.log shows the following error message:

    Date    Time[INFO] Replicating data DC=Contoso,DC=com: Received XXXXXX out of approximately XXXXXX objects and XXXXXX out of approximately XXXXXX distinguished name (DN) values...

    Date Time[WARNING] Non critical replication returned 14

    Error code 14 translates to: ERROR_OUTOFMEMORY - Not enough storage is available to complete this operation.

    The following event is logged in the event log during or shortly after dcpromo finishes:
    Event LogEvent SourceIDDescription
    Directory Services Replication 2094Performance warning: replication was delayed while applying changes to the following object. If this message occurs frequently, it indicates that the replication is occurring slowly and that the server may have difficulty keeping up with changes.

    Object DN: CN=Client Name,OU=OU,DC=Contoso,DC=com
    Object GUID: GUID
    Partition DN: DC=Contoso,DC=com
    Server: _msdcs DNS record of replication partner
    Elapsed Time (secs): XX

    User Action

    A common reason for seeing this delay is that this object is especially large, either in the size of its values, or in the number of values. You should first consider whether the application can be changed to reduce the amount of data stored on the object, or the number of values. If this is a large group or distribution list, you might consider raising the forest functional level to Windows Server 2003 or greater, since this will enable replication to work more efficiently. You should evaluate whether the server platform provides sufficient performance in terms of memory and processing power. Finally, you may want to consider tuning the Active Directory Domain Services database by moving the database and logs to separate disk partitions.

    If you wish to change the warning limit, the registry key is included below. A value of zero will disable the check.

    Additional Data

    Warning Limit (secs): XX

    Limit Registry Key: System\CurrentControlSet\Services\NTDS\Parameters\Replicator maximum wait for update object (secs)
    Directory ServicesKCC1308The Knowledge Consistency Checker (KCC) has detected that successive attempts to replicate with the following directory service has consistently failed.

    Attempts:
    2
    Directory service:
    CN=NTDS Settings,CN=DC001,CN=Servers,CN=site1,CN=Sites,CN=Configuration,DC=Contoso,DC=com
    Period of time (minutes):
    XXXXXXX

    The Connection object for this directory service will be ignored, and a new temporary connection will be established to ensure that replication continues. Once replication with this directory service resumes, the temporary connection will be removed.

    Additional Data
    Error value:
    14 Not enough storage is available to complete this operation    .

    Note The issue doesn't occur if Install from Media (IFM) promotion for domain controllers is used. However, IFM promotion has to be sourced from the same operating system version.
  • The following event is logged in the event log during or shortly after dcpromo finishes:
    Event LogEvent SourceIDDescription
    Directory Services Replication 2094Performance warning: replication was delayed while applying changes to the following object. If this message occurs frequently, it indicates that the replication is occurring slowly and that the server may have difficulty keeping up with changes.

    Object DN: CN=Client Name,OU=OU,DC=Contoso,DC=com
    Object GUID: GUID
    Partition DN: DC=Contoso,DC=com
    Server: _msdcs DNS record of replication partner
    Elapsed Time (secs): XX

    User Action

    A common reason for seeing this delay is that this object is especially large, either in the size of its values, or in the number of values. You should first consider whether the application can be changed to reduce the amount of data stored on the object, or the number of values. If this is a large group or distribution list, you might consider raising the forest functional level to Windows Server 2003 or greater, since this will enable replication to work more efficiently. You should evaluate whether the server platform provides sufficient performance in terms of memory and processing power. Finally, you may want to consider tuning the Active Directory Domain Services database by moving the database and logs to separate disk partitions.

    If you wish to change the warning limit, the registry key is included below. A value of zero will disable the check.

    Additional Data

    Warning Limit (secs): XX

    Limit Registry Key: System\CurrentControlSet\Services\NTDS\Parameters\Replicator maximum wait for update object (secs)
    Directory ServicesKCC1308The Knowledge Consistency Checker (KCC) has detected that successive attempts to replicate with the following directory service has consistently failed.

    Attempts:
    2
    Directory service:
    CN=NTDS Settings,CN=DC001,CN=Servers,CN=site1,CN=Sites,CN=Configuration,DC=Contoso,DC=com
    Period of time (minutes):
    XXXXXXX

    The Connection object for this directory service will be ignored, and a new temporary connection will be established to ensure that replication continues. Once replication with this directory service resumes, the temporary connection will be removed.

    Additional Data
    Error value:
    14 Not enough storage is available to complete this operation    .

    Note The issue doesn't occur if Install from Media (IFM) promotion for domain controllers is used. However, IFM promotion has to be sourced from the same operating system version.
  • Error code 14 translates to: ERROR_OUTOFMEMORY - Not enough storage is available to complete this operation.The following event is logged in the event log during or shortly after dcpromo finishes:
    Event LogEvent SourceIDDescription
    Directory Services Replication 2094Performance warning: replication was delayed while applying changes to the following object. If this message occurs frequently, it indicates that the replication is occurring slowly and that the server may have difficulty keeping up with changes.

    Object DN: CN=Client Name,OU=OU,DC=Contoso,DC=com
    Object GUID: GUID
    Partition DN: DC=Contoso,DC=com
    Server: _msdcs DNS record of replication partner
    Elapsed Time (secs): XX

    User Action

    A common reason for seeing this delay is that this object is especially large, either in the size of its values, or in the number of values. You should first consider whether the application can be changed to reduce the amount of data stored on the object, or the number of values. If this is a large group or distribution list, you might consider raising the forest functional level to Windows Server 2003 or greater, since this will enable replication to work more efficiently. You should evaluate whether the server platform provides sufficient performance in terms of memory and processing power. Finally, you may want to consider tuning the Active Directory Domain Services database by moving the database and logs to separate disk partitions.

    If you wish to change the warning limit, the registry key is included below. A value of zero will disable the check.

    Additional Data

    Warning Limit (secs): XX

    Limit Registry Key: System\CurrentControlSet\Services\NTDS\Parameters\Replicator maximum wait for update object (secs)
    Directory ServicesKCC1308The Knowledge Consistency Checker (KCC) has detected that successive attempts to replicate with the following directory service has consistently failed.

    Attempts:
    2
    Directory service:
    CN=NTDS Settings,CN=DC001,CN=Servers,CN=site1,CN=Sites,CN=Configuration,DC=Contoso,DC=com
    Period of time (minutes):
    XXXXXXX

    The Connection object for this directory service will be ignored, and a new temporary connection will be established to ensure that replication continues. Once replication with this directory service resumes, the temporary connection will be removed.

    Additional Data
    Error value:
    14 Not enough storage is available to complete this operation    .

    Note The issue doesn't occur if Install from Media (IFM) promotion for domain controllers is used. However, IFM promotion has to be sourced from the same operating system version.
  • Security descriptor (SD) propagation

    A memory leak issue may occur when a security change on a container object (domain root or organizational unit (OU)) is inherited on many child objects or subordinate OUs though SD propagation. Depending on the size of the access control entry (ACE) to be changed and the number of objects (for example, 500,000), the propagation may take a long time. During this time, the LSASS process or DsaMain process may constantly allocate committed bytes.
  • Long-running queries

    Unexpectedly high virtual memory consumption during long-running Lightweight Directory Access Protocol (LDAP) queries on Windows Server 2012 R2 or Windows 8.1-based LDAP servers may result in memory outages. The long-running queries consume memory by obtaining a heap for the security descriptor of each object that is touched.
In these situations, when the committed bytes reach the number of bytes that are available, the system becomes unusable and may even crash.

↑ Back to the top

Resolution

To fix this issue, apply the hotfix that is mentioned in the following section.

To use the hotfix in the context of Domain Controller Promotions (DCPROMO), follow these steps:
  1. Install the Active Directory Domain Services or AD LDS role.
  2. Install this update or newer Windows Server 2012 R2 updates and security updates that contain the same Ntdsai.dll binary as they are always cumulative.
  3. Promote the computer to a status of domain controller.
Important If you install a language pack after you install this hotfix, you must reinstall this hotfix. Therefore, we recommend that you install any language packs that you need before you install this hotfix. For more information, see Add language packs to Windows.

Hotfix information

A supported hotfix is available from Microsoft. However, this hotfix is intended to correct only the problem that's described in this article. Apply this hotfix only to systems that are experiencing this specific problem.

If the hotfix is available for download, there's a "Hotfix Download Available" section at the top of this Knowledge Base article. If this section doesn't appear, submit a request to Microsoft Customer Service and Support to get the hotfix.

Note If additional issues occur or if any troubleshooting is required, you might have to create a separate service request. The usual support costs will apply to additional support questions and issues that don't qualify for this specific hotfix. For a complete list of Microsoft Customer Service and Support telephone numbers or to create a separate service request, go to the following Microsoft website:
http://support.microsoft.com/contactus/?ws=support
Note The "Hotfix Download Available" form displays the languages for which the hotfix is available. If you don't see your language, it's because a hotfix isn't available for that language.

Prerequisites

To apply this hotfix, you must have the April 2014 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 (2919355) installed on Windows 8.1 or Windows Server 2012 R2.

Registry information

To apply this hotfix, you don't have to make any changes to the registry.

Restart requirement

You may have to restart the computer after you apply this hotfix.

Hotfix replacement information

This hotfix doesn't replace a previously released hotfix.

↑ Back to the top

Status

Microsoft has confirmed that this is a problem in the Microsoft products that are listed in the "Applies to" section.

↑ Back to the top

More Information

If you enable NTDS\Diagnostics "9 Internal Processing" level 2 as mentioned in Microsoft Knowledge Base article 314980, you may see the following ActiveDirectory_DomainService events. These events show the long-running SD propagation task.

Event 1257 - Indicating the start of the SD Propagation
Internal event: The security descriptor propagation task is processing a propagation event starting from the following container.
Container: OU=OU,DC=Contoso,DC=com

Event 2007 - Indicating the progress of the SD Propagation, logged every 5 minutes
Internal event: The security descriptor propagation task has reached the following container and will continue with the propagation.
Container: OU=OU,DC=Contoso,DC=com
Number of objects processed so far: XXXXXX

Event 1258 - Indicating the end of the SD Propagation, after some hours
Internal event: The security descriptor propagation task has finished processing a propagation event starting from the following container.
Container: OU=OU,DC=Contoso,DC=com
Number of objects processed: ZZZZZZ

After Active Directory replication to any other Windows Server 2012 R2-based domain controller or LDS instances in the domain, the same issue may occur because the SD propagation is performed locally.

You won't see the memory leak in the Active Directory Collector Set Report because it shows only used bytes. However, you can use the created performance monitor data BLG file-checking counter Object:Process, Instance:Lsass, Counter:Private Bytes, and Object:Memory, Counter:Committed Bytes.

For longer observation of the leak development, you can use a "Graph with Performance Monitor Properties, Graph Elements, Sample" every 60 seconds. Duration: 15,000 seconds (> 4 hours).

The increasing allocation can also be observed in Task Manager, tab Performance, item Memory, Committed. This is displayed under Committed/Available gigabytes (GB).

Indicators of the error condition are as follows:

repadmin /showrepl returns:

Not enough storage is available to complete this operation.

Directory Service logs Error event 1699:
This directory service failed to retrieve the changes requested for the following directory partition. As a result, it was unable to send change requests to the directory service at the following network address.
Extended request code: 0
Additional Data
Error value: 8446 The replication operation failed to allocate memory.

Application popup:
Windows - Out of Virtual Memory: Your system is low on virtual memory. To ensure that Windows runs properly, increase the size of your virtual memory paging file. For more information, see Help.

 

↑ Back to the top

References

Learn about the terminology that Microsoft uses to describe software updates.

↑ Back to the top

File Information

The English (United States) version of this software update installs files that have the attributes that are listed in the following tables. The dates and times for these files are listed in Coordinated Universal Time (UTC). Be aware that dates and times for these files on your local computer are displayed in your local time and with your current daylight saving time bias. The dates and times may also change when you perform certain operations on the files.
Windows 8.1 and Windows Server 2012 R2
Important Windows 8.1 hotfixes and Windows Server 2012 R2 hotfixes are included in the same packages. However, hotfixes on the Hotfix Request page are listed under both operating systems. To request the hotfix package that applies to one or both operating systems, select the hotfix that is listed under "Windows 8.1/Windows Server 2012 R2" on the page. Always refer to the "Applies To" section in articles to determine the actual operating system that each hotfix applies to.

Notes
  • The files that apply to a specific product, milestone (RTM, SPn), and service branch (LDR, GDR) can be identified by examining the file version numbers as shown in the following table:
    VersionProductMilestoneService branch
    6.3.960 0.18 xxxWindows 8.1 and Windows Server 2012 R2RTMGDR
  • GDR service branches contain only those fixes that are widely released to address widespread, critical issues. LDR service branches contain hotfixes in addition to widely released fixes.
  • The MANIFEST files (.manifest) and the MUM files (.mum) that are installed for each environment are listed in the "Additional file information" section. MUM, MANIFEST, and the associated security catalog (.cat) files, are very important to maintain the state of the updated components. The security catalog files, for which the attributes are not listed, are signed with a Microsoft digital signature.
x86 Windows 8.1
File nameFile versionFile sizeDateTimePlatform
Ntdsa.mofNot applicable227,76518-Jun-201312:21Not applicable
Ntdsai.dll6.3.9600.181162,583,55203-Nov-201502:41x86
x64 Windows 8.1 and Windows Server 2012 R2
File nameFile versionFile sizeDateTimePlatform
Ntdsa.mofNot applicable227,76518-Jun-201314:45Not applicable
Ntdsai.dll6.3.9600.181163,676,16003-Nov-201502:54x64
Additional file information
x86 Windows 8.1
File propertyValue
File nameUpdate.mum
File versionNot applicable
File size1,600
Date (UTC)04-Nov-2015
Time (UTC)05:53
PlatformNot applicable
File nameX86_532408894ea01e6280e568d78cbfbc21_31bf3856ad364e35_6.3.9600.18116_none_70de56a241809c7b.manifest
File versionNot applicable
File size712
Date (UTC)04-Nov-2015
Time (UTC)05:53
PlatformNot applicable
File nameX86_microsoft-windows-d..toryservices-ntdsai_31bf3856ad364e35_6.3.9600.18116_none_85b3851fd48201e8.manifest
File versionNot applicable
File size3,352
Date (UTC)03-Nov-2015
Time (UTC)05:09
PlatformNot applicable
x64 Windows 8.1 and Windows Server 2012 R2
File propertyValue
File nameAmd64_1c835b965fc6e60c22f413386606599e_31bf3856ad364e35_6.3.9600.18116_none_b800a1506ce79afa.manifest
File versionNot applicable
File size716
Date (UTC)04-Nov-2015
Time (UTC)05:53
PlatformNot applicable
File nameAmd64_microsoft-windows-d..toryservices-ntdsai_31bf3856ad364e35_6.3.9600.18116_none_e1d220a38cdf731e.manifest
File versionNot applicable
File size3,356
Date (UTC)03-Nov-2015
Time (UTC)06:04
PlatformNot applicable
File nameUpdate.mum
File versionNot applicable
File size2,061
Date (UTC)04-Nov-2015
Time (UTC)05:53
PlatformNot applicable

↑ Back to the top

Keywords: kb, kbqfe, kbsurveynew, kbfix, kbautohotfix, kbhotfixserver, kbexpertiseinter

↑ Back to the top

Article Info
Article ID : 3083038
Revision : 4
Created on : 6/22/2017
Published on : 6/22/2017
Exists online : False
Views : 808