Security rules for Windows Firewall and for IPsec-based connections in Windows Vista and in Windows Server 2008

This article describes the following:

• Security rules for Windows Firewall in Windows Vista
• Security rules for IPsec-based connections in Windows Vista and in Windows Server 2008
• How to establish an encrypted connection between Windows Vista and Windows XP or between Windows Vista and Windows Server 2003

In Windows Vista, the new Windows Firewall security rules and the new IPsec-based connection security rules are fully integrated into Group Policy. Therefore, the support settings merge, and they delegate behavior in the same manner as other Group Policy settings.

Security rules for Windows Firewall in Windows Vista

The "Windows Firewall with Advanced Security" Microsoft Management Control (MMC) snap-in supports the following types of security rules.

Note The list of Windows Firewall security rules and the list of connection security rules are merged from all applicable Group Policy settings. Then, these security rules are processed in the following order. The following order is always enforced, regardless of the source of the security rules.

• Windows service hardening rule
  The Windows service hardening rule restricts services from establishing connections. Service restrictions are configured in such a way that Windows services can only communicate in a specific way. For example, you can restrict traffic through a specific port. However, until you create a firewall rule, traffic is not permitted.
• Connection security rule
  The connection security rule defines how and when computers are authenticated by using the IPsec feature. Connection security rules are used to establish server isolation and to establish domain isolation. Additionally, connection security rules are used to enforce the Network Access Protection (NAP) policy.
• Authenticated bypass rule
  The authenticated bypass rule lets certain computers be connected if the traffic is protected by using the IPsec feature regardless of other incoming rules. Specified computers can bypass incoming rules that block traffic. For example, you can enable remote Firewall administration from certain computers only by creating authenticated bypass rules for the computers. Or, you can enable support for remote assistance by calling Helpdesk.
• Block rule
  The block rule explicitly blocks a certain type of incoming traffic or a certain type of outgoing traffic.
• Allow rule
  The allow rule explicitly allows for certain types of incoming traffic or certain types of outgoing traffic.
• Default rule
  The default rule is configured in such a way that the incoming default rule blocks connections, and the outgoing default rule allows for connections.

Security rules for IPsec-based connections in Windows Vista and in Windows Server 2008

The following two types of IPsec rules can be applied to a Windows Vista-based computer or to a Windows Server 2008-based computer:

• IPsec rules
  Earlier IPsec rules are currently deployed in Windows 2000 and in Windows Server 2003. The earlier IPsec rules are managed by the Policyagent service. These IPsec rules are Internet Key Exchange (IKE) rules that support only computer-based Kerberos authentication, x.509 certificates, or preshared key authentication. These IPsec rules are configured in the "IPsec Policy Management" MMC snap-in. IKE-based Policyagent rules are applied in the same manner as in Windows 2000 and in Windows Server 2003. Although multiple policies can be applied to a given computer, only the last policy that is applied is successful. This is according to the "last writer wins" method. Additionally, IKE policy settings cannot be merged.
• Connection security rules
  Connection security rules are only supported in Windows Vista and in Windows Server 2008. Connection security rules are supported by an extension to IKE that is called Authenticated IP (AuthIP). AuthIP adds support for the following authentication mechanisms:
  • Interactive user Kerberos credentials or interactive user NTLMv2 credentials
  • User x.509 certificates
  • Computer Secure Sockets Layer (SSL) certificates
  • NAP health certificates
  • Anonymous authentication (optional authentication)
  You can configure security rules for the "Windows Firewall and Advanced Security" snap-in by using the following tools:
  • Domain-based Group Policy
  • The "Windows Firewall with Advanced Security" snap-in
    
    Note The "Windows Firewall with Advanced Security" snap-in is the default storage location for policies that can be accessed by using the wf.msc command.
  • The local Group Policy snap-in (Gpedit.msc)
  • The netsh advfirewall command
    
    Note The netsh advfirewall command points to the same store as the wf.msc command.
  Like other firewall and Group Policy rules, connection security rules are merged from all applicable Group Policy objects.
  
  Connection security policies can be configured to create IPsec-based policies that are compatible with IKE version 1-based clients, such as Microsoft Windows 2000, Windows XP, and Windows Server 2003. Connection security policies can also be configured to create policies that support communications only between Windows Vista-based computers and Windows Server 2008-based computers.
  
  An administrator can create a connection security policy by using the following methods:
  • The administrator can create a connection security policy that supports only Kerberos authentication, user x.509 certificates, or computer authentication.
    
    Notes
    • In this case, the Mpssvc service automatically creates both AuthIP and earlier IKE policies.
    • If computer-based preshared key authentication is specified, AuthIP rules are not created.
  • The administrator can create a connection security policy that requires user authentication.
    
    Note In this case, only AuthIP policy is created for Windows Vista-to-Windows Vista negotiations and for Windows Vista-to-Windows Server 2008 negotiations. This is because IKE does not support user authentication.
  • The administrator can create a connection security policy in which user authentication options are added to the policy. Additionally, the administrator can also select the Second Authentication is optional option.
    
    Note The Mpssvc service creates both AuthIP policies and earlier IKE policies. Optional user authentication is included in the AuthIP set. Optional user authentication is not included in the earlier IKE policy.
  • The administrator can create a connection security policy that requires NTLM authentication.
    
    Note In this case, only AuthIP policy is created for Windows Vista-to-Windows Vista negotiations and for Windows Vista-to-Windows Server 2008 negotiations because IKE does not support NTLM authentication.
  • The administrator selects the "Diffie-Hellman" algorithm in the global "Main Mode Key Exchange" algorithm that is incompatible with earlier clients, such as "Elliptic Curve Diffie-Hellman P-256" algorithm.
    
    Notes
    • In this case, the "Diffie-Hellman" algorithm will not be supported by earlier IKE version 1 clients. Additionally, the IKE negotiations are unsuccessful.
    • We recommend that you use the "Diffie-Hellman Group 2" setting because this setting is supported across the broadest range of clients.
    • The "Diffie-Hellman Group 14" setting is supported in Windows XP Service Pack 2 (SP2) and in Windows Server 2003.
    • In this case, the key behavior change is that the "Diffie-Hellman" algorithm is generally not used for AuthIP-based negotiations.
    • When the Mpssvc service creates AuthIP rules, the Mpssvc service specifies that Windows Vista-to-Windows Vista negotiations or Windows Vista-to-Windows Server 2008 negotiations must only use the "Diffie-Hellman" algorithm if the main mode authentication method is set to Anonymous.
    • When the Mpssvc service creates IKE rules, the Mpssvc service always uses the "Diffie-Hellman" algorithm that is specific to the global Main Mode Key Exchange setting.
    • A Security Support Provider Interface (SSPI)-shared secret is used to generate the keying material in AuthIP exchanges in which the Main Mode Key exchange does not have a Diffie-Hellman exchange.

  Certificates that are used by AuthIP

  • AuthIP uses SSL certificates that have client authentication settings configured or that have server authentication settings configured. SSL certificates can be client authentication certificates. Or, SSL certificates can be both client authentication certificates and server authentication certificates.
  • If you construct policies to use certificate authentication for Windows Vista, you must have certificates that work with AuthIP. This means that the certificates that you deploy to the clients have to be SSL certificates that use client authentication or server authentication. The authentication depends on whether you want one-way authentication or mutual authentication. The SSL certificates differ from the standard digital certificates that are used in Windows XP or in Windows Server 2003.
  • By default, SSL certificates are used by NAP implementations.

Group Policy processing for the "Windows Firewall with Advanced Security" snap-in

Connection security rules are merged from all applicable Group Policy objects. However, there is a related group of settings for IPsec and for AuthIP that manages default, non-additive IPsec behavior. This group of settings includes global authentication sets, Quick Mode settings, Key Exchange settings, and Internet Control Message Protocol (ICMP) exemptions.

The default set of connection security options or IPsec options that are applied from the highest precedence Group Policy object (GPO) will succeed on a Windows Vista-based client. For example, all connection security rules on the Windows Vista-based client that use default authentication sets or cryptographic sets will use the sets from the highest precedence GPO. If you want more flexibility, you can use the following options:

• For authentication sets, configure authentication by using the connection security rule instead of by using default authentication.
• For Quick Mode cryptographic sets, use the netsh command to configure Quick Mode cryptographic settings for each connection security rule as required.
• For Main Mode cryptographic sets, only one Main Mode cryptographic set is supported for each policy. When multiple Main Mode cryptographic sets are received, the Main Mode cryptographic set from the highest precedence GPO will be applied to all connection security rules in Group Policy. However, you cannot customize the rules to use different Main Mode cryptographic sets.
• When you configure GPOs for connection security policies and for firewall policies, you can disable use of local firewall rules and connection security rules. Therefore, only Group Policy settings that are linked to site GPOs, domain GPOs, or organizational unit (OU) GPOs can control Windows Firewall behavior.

To view the "Introduction to Windows Firewall with Advanced Security" white paper for Windows Vista, visit the following Microsoft Web site:For more information about AuthIP in Windows Vista, visit the following Microsoft Web site:For more information about the new Windows Firewall in Windows Vista and in Windows Server 2008, visit the following Microsoft Web site:

How to establish an encrypted connection between Windows Vista and Windows XP or between Windows Vista and Windows Server 2003

In Windows Server 2003 and in Windows XP, IPsec policy configuration usually consists of a set of rules to protect most of the traffic on the network and another set of rules for protected traffic exemptions. A configuration may include server isolation and domain isolation. Exemptions are required for unprotected communication with network infrastructure servers such as Dynamic Host Configuration Protocol (DHCP) servers, Domain Name System (DNS) servers, and domain controllers. When a computer starts, the computer must be able to obtain an IP address and must be able to use DNS to find a domain controller. Additionally, the computer must be able to log on to its domain before the computer can start to use Kerberos authentication to authenticate itself as an IPsec peer. Other exemptions are required to communicate with network nodes that are not IPsec-capable. In some cases, there are many exceptions that make it more difficult to deploy IPsec protection on an enterprise network and to maintain the enterprise network over time.

In Windows Server 2008 and in Windows Vista, IPsec provides an optional behavior to negotiate IPsec protection. Assume that IPsec is enabled and that an IPsec node that is running Windows Server 2008 or Windows Vista initiates communication with another network node. In this case, the IPsec node will try to communicate without encryption, or "in the clear." The node will try to negotiate protected communication in parallel. If the initiating IPsec peer does not receive a response to the initial negotiation try, the communication continues in the clear. If the initiating IPsec peer receives a response to the initial negotiation try, the communication in the clear continues until the negotiation is completed. When the negotiation is complete, successive communications receive increased protection. The initiating IPsec node can find out whether the network node that it communicates with can perform IPsec. The initiating IPsec node then behaves accordingly. This behavior of the IPsec node is because of the IPsec optional behavior. Additionally, this behavior is because of the recommended configuration which requires protection for incoming initiated communications and which requests protection for outgoing initiated communications. This new behavior greatly simplifies IPsec policy configuration. For example, the initiating IPsec node does not have to have a set of predefined IPsec filters to exempt a set of hosts that cannot protect network traffic by using IPsec. The initiating IPsec node tries both protected traffic and unprotected traffic in parallel. If protected communication is not possible, the initiating IPsec node uses unprotected communication.

The new negotiation behavior also improves the performance of unprotected connections that are made to hosts. An IPsec node that is running Windows Server 2003 or Windows XP is configured to request protected communications, but enables unprotected communications. This IPsec node behavior is known as fallback to clear. The IPsec node sends negotiation messages and waits for a response. The initiating IPsec node waits up to three seconds before falling back to clear and trying unprotected communications. In Windows Server 2008 and in Windows Vista, there is no longer a three second delay when falling back to clear because communications in the clear are already in progress when the initiating IPsec node is waiting for a response.

For more information about server isolation and about domain isolation, visit the following Microsoft Web site:

Key points in establishing an encrypted connection between Windows Vista and Windows XP or between Windows Vista and Windows Server 2003

• If you only enable an encrypted connection in Windows Vista, Windows Vista negotiates only at the IPsec level with all other clients. This includes Windows XP-based clients.
• By default, Windows XP or Windows Server 2003 does not negotiate at the IPsec level. Therefore, we have to assign an IPsec rule in Windows XP or in Windows Server 2003 to establish an encrypted connection between Windows Vista and Windows XP or between Windows Vista and Windows Server 2003.
• By default, if the Allow only secure connections option is selected, Windows Vista negotiates by using the AES-128 encryption method and the 3DES encryption method. The AES-128 encryption method is not supported in Windows XP. The 3DES encryption method is supported in Windows XP and in Windows Server 2003.
• By default, if IPsec is enabled in Windows XP or in Windows Server 2003, IPsec will negotiate by using the 3DES encryption method.

To establish an encrypted connection between a Windows Vista-based computer and a Windows XP-based computer by using the "Windows Firewall with Advanced Security" snap-in, follow these steps:

1. On the Windows Vista-based computer, click Start
   , type firewall in the Start Search box, and then click Windows Firewall with Advanced Security in the Programs list.
2. In the console tree, click Inbound Rules.
3. In the list of Inbound Rules, double-click Remote Desktop (TCP-In), and then click the General tab.
4. In the Action area, click Allow only secure connections, click to select the Require encryption check box, and then click OK.
5. In the console tree, click Connection Security Rules, and then click New Rule on the Action menu.
6. Click Isolation, and then click Next.
7. Click Require authentication for inbound and outbound connections, and then click Next.
8. Click Default, and then click Next.
9. Click to select the following check boxes, and then click Next:
   • Domain
   • Private
   • Public
10. Type the name of the rule in the Name box, type the description of the rule in the Description (optional) box if you want, and then click Finish.
11. On the File menu, click Exit.
12. On the Windows XP-based computer, click Start, click Run, type secpol.msc, and then click OK.
13. In the console tree, right-click IP Security Policies on Local Computer, and then click Create IP Security Policy.
14. Click Next, and then follow the instructions in the IP Security Policy Wizard to create the IP Security Policy. Use the default settings to create this policy.
15. Right-click New IP Security Policy, and then click Assign.
16. On the File menu, click Exit.