Event ID 1988 Logged in Directory Service Log after Schema Update

When you install a schema extension that adds attributes to the set of attributes included in the global catalog, you may see multiple NTDS Replication Event ID 1988 events and possibly also Event ID 1388 on various domain controllers.

Event Type: Error
Event Source: NTDS Replication
Event Category: Replication
Event ID: 1988
Description:
The local domain controller has attempted to replicate the following object from the following source domain controller. This object is not present on the local domain controller because it may have been deleted and already garbage collected.

Source domain controller:
<guid1>._msdcs.contoso.com
Object:
CN=object\0ADEL:<guid>,CN=Deleted Objects,DC=contoso,DC=com
Object GUID:
<guid>

Replication will not continue with the source domain controller until the situation has been resolved.

 

 

Event Type: Error
Event Source: NTDS Replication
Event Category: Replication
Event ID: 1388
Description:
This destination system received an update for object which should have been present locally, but was not. The attribute set included in the packet is not sufficient to create the object. A full copy of the object will be requested...

A few hours after the events are first seen they stop being logged. The maximum duration of events is 12 hours, which is the default interval for garbage collection.

If Strict Replication is enabled, you will also see replication errors reported during this interval. When you check whether the objects referenced are still present, you may find the objects are not found in any database.

To identify whether you are affected by this problem, please follow these steps:

1. Identify the setting for the tombstoneLifetime attribute using the information towards the end of the Microsoft Knowledge Base article below. The tombstoneLifetime attribute specifies the number of days a deleted object is kept as a tombstone.
   
   910205 Information about lingering objects in a Windows 2000-based forest or in a Windows Server 2003-based forest
   
   
2. Copy the object GUID of the object cited in Event ID 1988 on the destination domain controller (text between DEL: and ,cn=Deleted), for example A32C892F-E03F-48F7-A6EB-8E35CAA52743.
   
   
3. Run the following command against the object GUID of the object on the source domain controller:
   
   repadmin /showobjectmeta <source DC> <GUID=A32C892F-E03F-48F7-A6EB-8E35CAA52743>
   
   Example output:
   
   

   Loc.USN	Originating DC	Org.USN	Org.Time/Date Ver	Ver	Attribute
   12046	f92c9c73-0e7d-480e-944a-b94c605d697d	1410	2001-05-02 16:55:32	1	objectClass
   1531604	8486def0-f74e-441e-a71a-5df1dbaad068	1531604	2008-03-31 10:10:18	2	cn
   3654557	<site>\<DCname>	7258127	2009-09-22 12:32:04	1	IsDeleted

   
   The time stamp you are interested in is on the same line as IsDeleted. The time stamp should be past tombstone expiry (current time minus days of tombstone lifetime) by a short period of time.
   
   
4. Run repadmin /showobjectmeta against the object GUID of the object on the destination domain controller. You should receive an object not found error or similar because garbage collection has fully purged the deleted object.

Schema extensions add attributes to object classes, or change the presence settings of attributes on tombstoned objects that reside in writable and read-only directory partitions. All existing objects, both live and tombstoned objects subject to the schema update are modified in each local copy of the Active Directory database. The database is updating the tombstoned objects with the additional attributes so it has the correct set of attributes in case the object is reanimated.

Live objects are not a problem here. All domain controllers should have the same set of objects and attributes. However, for tombstoned objects that are close to removal from the database, the object population may vary amongst domain controllers. Although each copy of a tombstone has the same deletion time stamp and thus could be removed from all domain controllers at the same time, garbage collection runs at a different time on each domain controller.

Some tombstones may still exist on the source of a replication agreement and have an attribute added to the partial attribute set of the object which needs to be replicated out. However, the same object was garbage collected on the target domain controller when it is replicated. When this happens, the destination domain controller logs Event ID 1988 and possibly Event ID 1388.

Garbage collection runs every 12 hours by default and is independent of other tasks such as Active Directory replication. The error condition should remove itself as the problem objects are also garbage collected on the source domain controller the next time it runs. Subsequent attempts to replicate the naming context should succeed.

You cannot completely resolve the issue. You can only work on minimizing the impact of the varying object population. The reference on how changes are implemented is in the More Information section. Some of the different approaches include:

1. Let the errors occur and reactively triggering garbage collection.
   
   From the time of the schema update until the last domain controller has removed the overdue tombstones, ignore the errors that your monitoring system may log, based on the Event ID 1988 and Event ID 1388 events. You may reduce the frequency by manually starting garbage collection on the source domain controllers of the events that are reported.
   
   This approach should work best if your replication monitoring is reporting errors quickly and you are replicating frequently, so for a certain source domain controller the problem would be cleared on the next replication cycle.
   
   
2. Minimize divergence by running garbage collection more often.
   
   Shorten the garbage collection interval to maybe three hours a day before the schema extension. This should reduce the number of objects that are a potential problem, and also the time until the set of objects converges. The default interval is 12 hours. Changing the interval to three hours should not be a problem for most deployments. You can make the change several days before the schema extension and create a performance baseline to see whether this has adverse impact on domain controllers.
   
   
3. Synchronize garbage collection execution and extend tombstone lifetime.
   
   One day before the schema extension, do not allow any domain controller reboots. Then use scripts to create scheduled tasks on each domain controller in the forest. The scheduled task should run on all domain controllers at the same time (UTC), during a time of day when historically little or no Active Directory management occurs.
   
   The schedule task should:
   
   - Trigger garbage collection
   - Increase tombstone lifetime by five days
   
   If you were to skip triggering garbage collection, domain controller reboots would result in garbage collection being unsynchronized again, and you would again have to trigger it across all domain controllers. Changing tombstone lifetime on each domain controllers prevents replication latency from causing any inconsistency.
   
   After the schema is extended, the set of objects present on all domain controllers should be the same, as all domain controllers run garbage collection at the same time. Therefore the objects should not differ much between domain controllers.
   
   If you still see Event ID 1988, you may choose to trigger garbage collection on the source domain controller manually as described in the first method above. When the schema extension and object cleanup is done, you can reset tombstone lifetime to its original value.
   

You can use any of the above approaches. They are ranked by effectiveness, but also by impact to the environment. Therefore the third method will have the least impact on Active Directory replication and yield the lowest number of events, but it also has the highest impact on forest operations. What approach you choose will also depend on how you are monitoring Active Directory replication.

For example, given a monitoring data collection interval of eight hours, and a default garbage collection interval of 12 hours, chances are the flood of events is already over until you can take steps to control it (e.g. as described in the first method). If your monitoring system is reporting problems quickly, the first method may be a good approach.

The second method works best if you have low agility in replication monitoring and you cannot easily trigger garbage collection on all domain controllers in the forest, or run other commands. Shortening the garbage collection interval is a change distributed through replication of the configuration naming context, so all domain controllers will get it.

When your goal is to have the absolute minimum of false alarms as the number of alarms seen is part of your service level agreement, the third method is likely the best option for you.

These are the references for the resolution steps:

1. For more information about changing the garbage collection interval, click the following article number to view the article in the Microsoft Knowledge Base:
   
   198793 The Active Directory database garbage collection process
   
   
2. Manually triggering garbage collector can be done using the LDFIDE tool:
   
   ldifde /s <server> /i /f dogarbage.ldif
   
   Contents of dogarbage.ldif (the "-" on its own line and the additional empty line are required):
   
   dn:
   changetype: modify
   replace: DoGarbageCollection
   dogarbagecollection: 1
   -
   
   
   The line with "-" and the empty line are part of the file.
   
   
3. For more information about finding and changing the tombstone lifetime, click the following article number to view the article in the Microsoft Knowledge Base (see method 4 within the article):
   
   910205 Information about lingering objects in a Windows 2000-based forest or in a Windows Server 2003-based forest
   
   The article discusses using the ADSIEDIT tool, but you can accomplish the same thing with LDIFDE:
   
   ldifde /s %computername% /i /f TSL-185.ldif
   
   Contents of TSL-185.ldif (the "-" on its own line and the additional empty line are required):
   
   dn: CN=Directory Service,CN=Windows NT,CN=Services,cn=configuration,dc=contoso,dc=com
   changetype: modify
   replace: TombstoneLifetime
   TombstoneLifetime: 185
   -
   
   
   The line with "-" and the empty line are part of the file.

Note This is a "FAST PUBLISH" article created directly from within the Microsoft support organization. The information contained herein is provided as-is in response to emerging issues. As a result of the speed in making it available, the materials may include typographical errors and may be revised at any time without notice. See Terms of Use for other considerations.