Troubleshooting and data collection in Windows Server 2012 R2 Hyper-V Network Virtualization (HNV) environments

Important Make sure that all recommended hotfixes are applied to the Hyper-V Network Virtualization (HNV) environments. For a complete list of recommended hotfixes, please see the following article in the Microsoft Knowledge Base:

Perform basic troubleshooting and connectivity checks

Basic connectivity must be established before communication can occur. HNV uses NVGRE encapsulation to provide this connectivity between hosts. This connectivity is provided over the network that is commonly generally known as the "provider network" or "PA network."

This "network" is a real network that must be routable across the customer's infrastructure. However, it actually exists only between the virtual switches of the hosts. It is this network that makes HNV possible. One can ping across this network by using a special variation of the ping command. However, before the ping can be performed, the provider addresses that are being used on a particular host must be determined.

You can do this by issuing the Windows PowerShell cmdlet Get-NetVirtualizationProviderAddress from an Administrative PowerShell command prompt on the host computer.

Note This command is available only on HNV-capable hosts. An example of this output is shown in the following screen shot.



A host should have one PA address (ProviderAddress in the output here) for each tenant network that also has at least one VM on that host. Each AddressState field should be listed as Preferred.

If this field is listed as Duplicate, this is an issue that has to be corrected. Duplicate addresses can be revoked, but revocation must be used very cautiously. Revoking an IP address immediately puts that address back into the appropriate pool for reassignment. If the underlying issue that first caused the address to be duplicated is not corrected, the address could be duplicated again after it is reassigned. After you have a listing of PA addresses, you can ping the addresses from another host by using the ping -p <PA_Address> command. You have to ping only one address per host.

The following screen shot shows an example of this command in use.



If this ping is successful, the physical connectivity of the NVGRE network is working. If this ping fails, you may have to check physical connectivity and host configuration.

Issues here include, but are not limited to, switch misconfiguration, physical network failure, and incorrect VLAN tagging. 

If the PA ping is successful, try to ping between VMs on the same tenant network. This includes VMs on the same host and also VMs on different hosts.

This helps determine whether there is internal connectivity for the VMs themselves. Because the tenant VMs are completely unaware of the NVGRE network, no special commands are necessary to ping between tenant VMs. This is just a standard ping. As an additional test, have the tenant VM ping the following two addresses:

If the .1 address in unreachable, the tenant VM is not reaching the gateway host. Drop back, and verify that the PA ping works to the gateway host.

If the .1 address is reachable, this behavior resembles the tenant VM pinging another tenant VM. It proves that the NVGRE network is functioning correctly. Try to ping the .2 address.

If the .1 address is reachable but the .2 address is not, the ping cannot reach the internal interface of the HNV gateway VM. Troubleshooting should focus here.

If both the .1 and .2 addresses are reachable, the back-end HNV network is functioning correctly, and the tenant can communicate with the HNV gateway VM. Have the tenant try to ping something external to the HNV environment. If this fails, the issue could be in the gateway VM.

Additional validation would include an attempt to ping from the HNV gateway VM to the same external address. If the ping is successful, the HNV gateway may not be NAT'ing addresses correctly, and troubleshooting should focus on the gateway VM. If this is unsuccessful, the issue may be external to the gateway and HNV environment. Additional troubleshooting would have to be done to determine why the external configuration is failing.

If there are issues with a site-to-site VPN, please see the following websites for more information:

Collect basic data

Run the HNV gateway script

The HNV gateway script is useful in identifying issues that are related to Hyper-V Network Virtualization (HNV) gateway. The script helps by validating the HNV gateway and finding the most common gateway problems.

The HNV gateway script does following:

• Checks whether the gateway is set up correctly
• Checks connectivity between a running VM and its gateway
• Checks connectivity between two VMs

You can download the script from the following website:
The easiest way to start the script is to start it on the computer on which VMM Console is installed and to specify UseDefaultCredential $false. This prompts for credentials for every operation. If the user who is running the script already has administrator permissions to the HNV hosts, VMM, and gateway VMs, run the script without any parameter.

The script displays help and provides an example at the start. This can be useful for running only a targeted scenario. Warnings are displayed in yellow, and errors are displayed in red. Make sure that you watch for warnings and errors. This may help in reducing the overall investigation time for HNV gateway issues. The script performs read-only operations on all the systems that are involved. (However, the script starts any gateway cluster resources that are in a failed state.)

Note This script is interactive and uses stdout. It does not generate an output file. So, please carefully read the information that it provides.

Enable debug logging for Hyper-V Network Virtualization (HNV)

For more information about how to enable debug logging for HNV, please see the following Microsoft Knowledge Base article:

Note The netsh virtualization captures can become very large very quickly and should be used sparingly.