How to Configure IPSec on an Exchange Server 2003 Back-End Server That Is Running on a Windows Server 2003 Server Cluster

This article discusses the use of Internet Protocol security (IPSec) on Exchange 2003 back-end servers that are running on a Windows Server 2003-based server cluster.

Microsoft supports Exchange 2003 running on Windows Server 2003-based computers and using IPSec transport mode Encapsulating Security Payload (ESP) to encrypt communication with clustered Exchange 2003 servers that use Network Load Balancing clusters or server clusters. When you use IPSec between front-end servers and back-end servers, failover times depend on Exchange 2003 recovery time plus the time it takes for IPSec to renegotiate security associations during the failover process.

Front-end Exchange 2003 servers such as Outlook Web Access (OWA) servers cannot use Secure Sockets Layer (SSL) or Transport Layer Security (TLS) to encrypt traffic to back-end Exchange 2003 servers. The communication between front-end servers and back-end servers always uses unencrypted forms of traffic when Hypertext Transfer Protocol (HTTP), Post Office Protocol version 3 (POP3), or Internet Messaging Access Protocol 4 (IMAP4) are used.

In Exchange 2003, the HTTP protocol uses secure authentication where possible. The Exchange 2003 front-end server uses Kerberos or NTLM authentication to communicate with the back-end server if the back-end server is configured to accept integrated Windows authentication. This helps protect user password information from a malicious user who might try to capture the network traffic between the front-end server and the back-end server.

The POP3 protocol and the IMAP4 protocol can only use basic authentication to communicate with a back-end server. Because of this limitation, user password information is not protected against a malicious user who might try to capture network traffic between the front-end server and the back-end server.

The following list describes some of the reasons why you may want to use IPSec to establish trust and to encrypt network traffic between an Exchange 2003 front-end server and a back-end server:

• Your network environment may require that user passwords be encrypted. This is relevant to POP3 clients and IMAP4 clients that are using a front-end server.
• You may require the encryption of all the data in the network traffic between the front-end server and the back-end server. The e-mail messages may be considered sensitive and must be encrypted between the front-end server and the back-end server.
• You may have a firewall configured between the front-end server and the back-end server that only permits IPSec connections, or you want to send all network traffic through this firewall over a single port.

To perform these tasks, you can configure IPSec to establish trust and to encrypt network traffic between the front-end server and the back-end server. This scenario is supported in Windows Server 2003 whether clustering technologies are used or not, but the scenario is only supported in Microsoft Windows 2000 Server configurations that use a non-clustered environment. For additional information about the use of IPSec in a Windows 2000 Server cluster or a Network Load Balancing cluster, click the following article numbers to view the articles in the Microsoft Knowledge Base: When you configure IPSec on a back-end server in a Windows Server 2003 server cluster, the following behavior occurs when you use the Cluster Administrator utility to move a clustered resource that uses a virtual IP address:

1. The virtual IP address is programmatically removed from the first cluster node.
2. The removal of the virtual IP address from the first cluster node causes IPSec to "cleanly" remove the IPSec security association state with the front-end server.
3. If the front-end server still tries to connect to the back-end server, the IPSec Internet Key Exchange (IKE) negotiation protocol immediately tries to renegotiate a security association with the new back-end cluster node.
4. When the new back-end cluster node configures itself with the virtual IP address, the IPSec component on that node responds to the front-end server and establishes new IPSec security associations.

This procedure may take approximately 3 to 6 seconds, but the actual time depends on the CPU load on both cluster nodes and the network conditions that exist during this process.

In a scenario where the Microsoft Cluster service back-end cluster node stops responding (crashes), the Cluster service starts the failover procedure for the clustered program and the virtual IP address . However, in this case, the front-end IPSec computer still believes it has a secure link to the virtual IP address. The IPSec component uses its idle timer to determine that the back-end node no longer exists. On a Windows 2000-based computer, the idle time minimum is 5 minutes. On a Windows Server 2003-based computer, the idle time is automatically reduced to 1 minute if the registry key is set. As soon as IPSec removes the idle IPSec security associations, IKE tries to renegotiation new IPSec security associations. After 1 minute, IKE tries to establish a new Main Mode negotiation to the virtual IP address and is then successful in creating new security associations with the new cluster node. Because of this procedure, the total time it takes for IPSec to fail over is 6 minutes: the IPSec idle time of 5 minutes plus 1 minute for IKE to renegotiate a new Main Mode negotiation with the virtual IP address.

To Modify the Renegotiation Time

To allow IPSec to renegotiate the session to a back-end cluster node in a shorter period of time after an unexpected failover, set the registry value in the following registry subkey on the front-end Exchange 2003 server:To do so:

WARNING: If you use Registry Editor incorrectly, you may cause serious problems that may require you to reinstall your operating system. Microsoft cannot guarantee that you can solve problems that result from using Registry Editor incorrectly. Use Registry Editor at your own risk.

1. On the front-end Exchange 2003 server, click Start, and then click Run.
2. In the Open box, type regedit, and then click OK.
3. Locate the following registry subkey:
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PolicyAgent\Oakley
4. In the right pane, right-click NLBSFlags, and then click Modify.
5. In the Value data box, type 1 (one), and then click OK.
   
   Note After you set the
   NLBSFlags
   registry value to 1, the total time it takes for IPSec to fail over is 2 minutes: 1minute for the idle time to expire plus 1 minute for the IKE to renegotiate security associations.
6. Quit Registry Editor.

Suggested IPSec Policy Design

While this article does not provide step-by-step instructions to configure IPSec policies, the following suggested IPSec policy implementations may be used with Exchange 2003:

• To encrypt POP3 user passwords and IMAP4 user passwords, use IPSec to encrypt traffic to the back-end Exchange 2003 servers on port 110 (POP3) and port 143 (IMAP4).
• To encrypt the actual message stream to the back-end Exchange 2003 servers, use IPSec to encrypt all traffic to the back-end servers on port 80, port 110, and port 143.
• To encrypt all communications between the front-end Exchange 2003 servers and the back-end Exchange 2003 servers and to domain controllers, encrypt all network traffic including the following traffic:
  • Lightweight Directory Access Protocol (LDAP)
  • Remote Procedure Call (RPC)
  • Kerberos
  • LDAP communication with global catalog servers

For additional information about how to configure IPSec, search for IPSec in the Windows Server 2003 Help and Support Center.