MSDTC fails to mutually authenticate when computers do not run in the same domain

When two computers that are not running in the same domain use Microsoft Distributed Transaction Coordinator (MSDTC) for communication and for distributed transactions, the mutual authentication may fail.

When you run a distributed transaction against a linked server in Microsoft SQL Server 2000 on a computer that is running Microsoft Windows Server 2003 Enterprise Edition, you may receive the following error message:

The MSDTC proxy may not correctly authenticate MSDTC when the communicating computers are not in the same domain.

Important This section, method, or task contains steps that tell you how to modify the registry. However, serious problems might occur if you modify the registry incorrectly. Therefore, make sure that you follow these steps carefully. For added protection, back up the registry before you modify it. Then, you can restore the registry if a problem occurs. For more information about how to back up and restore the registry, click the following article number to view the article in the Microsoft Knowledge Base:

For information about how to edit the registry, view the "Change Keys and Values" Help topic in Registry Editor (Regedit.exe) or the "Add and Delete Information in the Registry" and "Edit Registry Data" Help topics in Regedt32.exe. Microsoft recommends that you back up the registry before you edit it.

To work around this problem, turn off the RPC security on your Windows Server 2003 computers. RPC security is a new DTC feature in Windows Server 2003. When you disable RPC security, the DTC authentication security level for RPC calls goes back to a level that is available in Microsoft Windows 2000 Server. To do this, follow these steps to set the DWORD value for the TurnOffRpcSecurity registry value to 1:

1. Start Registry Editor (Regedt32.exe).
2. Locate the following key in the registry:
   HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSDTC
3. On the Edit menu, click Add Value, and then add the following registry value:

   Value name	Data type	Value
   TurnOffRpcSecurity	REG_DWORD	1

4. Quit Registry Editor.

Note After you install Windows Server 2003 Service Pack 1, the registry entry TurnOffRpcSecurity is set to 0. You have to restore the value to 1 by using Registry Editor before you restart the computer.

This behavior is by design.

Steps to reproduce the behavior

1. Make sure that both computers are running Windows Server 2003.
2. Install SQL Server 2000 with Service Pack 3 (SP3) on both the computers.
3. Make sure that MSDTC is started on both the computers.
4. On the first computer, start the SQL Query Analyzer (Isqlw.exe) utility, and then connect to the local SQL Server.
5. Add the second computer as the linked server. To do so, run the following Transact-SQL statement in SQL Query Analyzer:

   EXEC sp_addlinkedserver  'remote_server',  N'SQL SERVER'
   GO

   Note Replace remote_server with the name of the second computer.
6. Run the following Transact-SQL statement in SQL Query Analyzer:

   SET xact_abort ON 
   GO
   USE  pubs
   GO
   BEGIN DISTRIBUTED TRANSACTION
   SELECT  *  FROM remote_server.pubs.dbo.authors
   COMMIT TRAN
   GO
   

   You may receive the error message that is mentioned in the "Symptoms" section of this article.

For more information, click the following article number to view the article in the Microsoft Knowledge Base: For more information about Microsoft COM+ 1.5, visit the following Microsoft Developer Network (MSDN) Web site: