ADDS: Deploying the 1st W2K8 R2 or later DC in an existing forest may temporarily halt AD replication to strict mode destination DCs for up to 12 hours

Abstract

This article was written to describe a behavior where adding the 1st post Windows Server 2008 domain controller to a given domain may temporarily halt Active Directory replication to strict mode destination DCs. Windows Server 2008 R2 and later domain controllers that support the Active Directory Recycle Bin feature stamp an "isdeleted" attribute on all objects located the deleted objects container. The act of stamping such attributes creates a replication event to partner DCs. Each domain controller independently deletes objects when garbage collection executed every 12 hours after the last boot. That DCs last booted at different times can lead to a condition where "isdeleted" time stamps are being outbound replicated to strict mode destination DCs that have already purged those same objects from their copy of the Active Directory database. The condition is resolved within 12 hours or less once all domain controllers garbage collect the same objects.

A somewhat related problem is that the addition of new schema changes for new OS and application versions can add new indexes to Active Directory databases which can generate high disk utilization on the NTDS.DIT volume. To mitigate this problem, KB 2846725 describes deferred index creation on Windows Server 2008 R2 DCs that is built into Windows Server 2012 and later OS versions.

Secondly, in addition to causing strict mode replication failures for up to 12 hours due to the way that domain controllers individually purge tombstoned objects, the stamping of isrecycled on large populations of deleted objects can create a significant replication event that can result in a replication event log.

Finally, Exchange Servers and other server roles register as change notification clients for 2 DNTs in the configuration partition. The introduction of the 1st recycle bin aware DC to an AD forest would write isrecycled on objects subject to change notification.

Symptoms

1. An existing forest consists exclusively of pre-Windows Server 2008 R2 domain controllers in an Active Directory forest. Strict replication is enabled on at least one domain controller in the forest.
   
   
2. Objects, including user accounts, are deleted from Active Directory partitions. These deleted objects transition to the deleted objects container and are removed from Active Directory by the garbage collection daemon tombstone lifetime (TSL) number of days in the future.
   
   
3. Windows Server 2008 R2 or a later version of ADPREP /FORESTPREP is executed.
   
   
4. The first post-Windows Server 2008 DC is added to the forest, which has the side-effect of stamping the isRecycled attribute on live objects AND deleted objects that reside in the deleted objects container.  This includes objects that are at the cusp of TSL expiration and about to be garbage collected. This update triggers an outbound replication event to replica DCs hosting common partitions.
   
   
5. Shortly after step 4, NTDS Replication Event 1988 is logged on destination DCs where strict mode destination DCs that received a request to inbound replicate an update to an object from the source DC cited in the event that the destination DC has already seen, deleted and garbage collected. The DN path in the 1988 event for this scenario are all "delete mangled". Text from a sample 1988 event is shown below:
   
   Event Type:       Error
   Event Source:    NTDS Replication
   Event Category: Replication
   Event ID:           1988
   Date:                <date>
   Time:               <time>
   User:                NT AUTHORITY\ANONYMOUS LOGON
   Computer:         <hostname of DC that logged event - i.e. the destination DC in the context of replication>
   Description:
   Active Directory Replication encountered the existence of objects in the following partition that have been deleted from the local domain controllers (DCs) Active Directory database.  Not all direct or transitive replication partners replicated in the deletion before the tombstone lifetime number of days passed.  Objects that have been deleted and garbage collected from an Active Directory partition but still exist in the writable partitions of other DCs in the same domain, or read-only partitions of global catalog servers in other domains in the forest are known as "lingering objects". 
    
   This event is being logged because the source DC contains a lingering object which does not exist on the local DCs Active Directory database.  This replication attempt has been blocked.
    
    The best solution to this problem is to identify and remove all lingering objects in the forest.
    
   Source DC (Transport-specific network address):
   <object guid of source DCs NTDS Settings object or CNAME record in DNS>._msdcs.<forest root domain>
   Object:
   <DN path of updated object being outbound by source DC that has been seen deleted and garbage collected by destination DC>
   Object GUID:
   <32-character long object GUID for object being updated by source DC that has been garbage collected by the destination DC>
   
   
6. REPADMIN /SHOWOBJMETA output, run against the objects cited in the 1988 events, shows that the isRecycled attribute is populated on objects that were deleted on the cusp of approximate TSL number of days in the past. A closer check reveals that the object cited in the 1988 is temporarily "live" (for up to 12 hours) on the source DCs cited in the 1988 events but missing on destination DCs that logged the 1988 event.