Error -4001 during Transaction Log Replay on CCR or Database Recovery on Stand Alone server when Exchange 2007 SP3 RU3 v.08.03.159.000 is installed

An issue exists which may lead to database corruption if you are running Version 08.03.159.000 of Exchange 2007 Service Pack 3 with Update Rollup 3 (KB2492691).

This issue is of specific concern in two scenarios: 1) when transaction log replay is performed by the Replication Service as part of ensuring the passive database copy is up-to-date, and/or 2) when a database is not cleanly shut down and recovery occurs.

When this issue occurs, the following similar events are logged in the Application Event log of the Mailbox server. Regardless of whether you see these types of events, you should review the recovery instructions and begin that process. If you are uncomfortable performing any of these steps please contact Microsoft Support for assistance.

Event ID: 454
Event Type: Error
Event Source: ESE
Event Category: Logging/Recovery
Description: Microsoft.Exchange.Cluster.ReplayService (12716) Recovery E20 SG1\DB1: Database recovery/restore failed with unexpected error -4001.

Event ID: 2095
Event Type: Error
Event Source: MSExchangeRepl
Event Category: Service
Description: Log file D:\logs\SG1\E200006AFAE.log in SG1\DB1 could not be replayed. Re-seeding the passive node is now required. Use the Update-StorageGroupCopy cmdlet in the Exchange Management Shell to perform a re-seed operation

Event ID: 2097
Event Type: Error
Event Source: MSExchangeRepl
Event Category: Service
Description: The Microsoft Exchange Replication Service encountered an unexpected Extensible Storage Engine (ESE) exception in storage group 'SG1\DB1'. The ESE exception is a read was issued to a location beyond EOF (writes will expand the file) (-4001) ().

In addition, in environments utilizing Continuous Replication, comparison of the database file between the active and passive nodes will indicate that the database file has decreased in size.

This issue was introduced in Version 08.03.159.000 of Exchange 2007 SP3 RU3 by a change in how the database is grown during transaction log replay when new data is written to the database file and there are no available free pages to be consumed.

Even if you are not experiencing this issue, we strongly recommend taking the actions below, to ensure that you do not experience any data loss or outage events associated with this issue.

For example:

• If you have deployed your Mailbox servers utilizing Cluster Continuous Replication (CCR), failure of the active copies may affect your service SLA as you may have no viable passive copies to activate. Hardware failures may result in you not having a means to recover up to the point of failure and thus may experience data loss.
• If you have deployed your Mailbox servers utilizing Single Copy Clusters (SCC), switchovers or failovers may result in this issue as there is only one copy of the database and recovery is performed during switchovers and failovers.

For environments leveraging CCR and/or Standby Continuous Replication (SCR)

If you note the listed events in your environment the following steps must be taken in order to restore your high-availability configuration:

1. Install the re-release of Exchange 2007 Service Pack 3 Update Rollup 3 Version 08.03.159.002 (KB2530488) on the CCR Mailbox server hosting the passive database copies and any SCR target Mailbox servers.
2. Re-seed all database copies on the CCR Mailbox server and any SCR target Mailbox servers hosting the passive database copies.
3. Verify the database copy status is healthy for all passive copies.
4. Perform a switchover and install the re-release of Exchange 2007 Service Pack 3 Update Rollup 3 Version 08.03.159.002 (KB2530488) on the remaining CCR Mailbox server.

If you are not seeing these events in your continuous replication enabled environment, we recommend the following steps:

1. Install the re-release of Exchange 2007 Service Pack 3 Update Rollup 3 Version 08.03.159.002 (KB2530488) on the CCR Mailbox server hosting the passive database copies and any SCR target Mailbox servers.
2. Perform a switchover and install the re-release of Exchange 2007 Service Pack 3 Update Rollup 3 Version 08.03.159.002 (KB2530488) on the remaining CCR Mailbox server.

For environments leveraging Single Copy Clusters (SCC)

1. Install the re-release of Exchange 2007 Service Pack 3 Update Rollup 3 Version 08.03.159.002 (KB2530488) on passive nodes within the SCC environment.
2. Perform a switchover and Install the re-release of Exchange 2007 Service Pack 3 Update Rollup 3 Version 08.03.159.002 (KB2530488) on the remaining SCC Mailbox server nodes.
3. Restore and recover any damaged databases leveraging a last known good backup.

For environments leveraging standalone Mailbox servers

1. Install the re-release of Exchange 2007 Service Pack 3 Update Rollup 3 Version 08.03.159.002 (KB2530488) on the standalone Mailbox servers.
2. Restore and recovery any damaged databases leveraging a last known good backup.

For Hub Transport and Edge Transport servers

1. Install the re-release of Exchange 2007 Service Pack 3 Update Rollup 3 Version 08.03.159.002 (KB2530488) on the standalone transport servers.
2. Recover damaged mail.que databases by following the steps in Working with the Queue Database on Transport

Working with the Queue Database on Transport
http://technet.microsoft.com/en-us/library/bb124343(EXCHG.80).aspx

While only a small number of customers have been affected to date, we believe the risk is significant enough that we are recommending all customers to install the re-release of Exchange 2007 Service Pack 3 Update Rollup 3 Version 08.03.159.002 (KB2530488) on all Mailbox Servers and Transport servers.

Description of Update Rollup 3 for Exchange Server 2007 Service Pack 3
http://support.microsoft.com/kb/2530488

Note This is a "FAST PUBLISH" article created directly from within the Microsoft support organization. The information contained herein is provided as-is in response to emerging issues. As a result of the speed in making it available, the materials may include typographical errors and may be revised at any time without notice. See Terms of Use for other considerations.