Windows Server 2012 R2 and Server 2016 computers that experience disconnections to iSCSI attached targets may show many different symptoms. These include, but are not limited to:
- The operating system stops responding
- You receive Stop errors (Bugcheck errors) 0x80, 0x111, 0x1C8, 0xE2, 0x161, 0x00, 0xF4, 0xEF, 0xEA, 0x101, 0x133, or 0xDEADDEAD.
- User log on failures occur together with a "No Logon Servers Available" error.
- Application and service failures occur because of ephemeral port exhaustion.
- An unusually high number of ephemeral ports are being used by the System process.
- An unusually high number of threads are being used by the System process.
Cause
This issue is caused by a locking issue on Windows Server 2012 R2 and Windows Server 2016 RS1 computers, causing connectivity issues to the iSCSI targets. The issue can occur after installing any of the following updates:
Windows Server 2012 R2
|
Release date
|
KB
|
Article title
|
|
May 16, 2017
|
KB 4015553
|
April 18, 2017—KB4015553 (Preview of Monthly Rollup)
|
|
May 9, 2017
|
KB 4019215
|
May 9, 2017—KB4019215 (Monthly Rollup)
|
|
May 9, 2017
|
KB 4019213
|
May 9, 2017—KB4019213 (Security-only update)
|
|
April 18, 2017
|
KB 4015553
|
April 18, 2017—KB4015553 (Preview of Monthly Rollup)
|
|
April 11, 2017
|
KB 4015550
|
April 11, 2017—KB4015550 (Monthly Rollup)
|
|
April 11, 2017
|
KB 4015547
|
April 11, 2017—KB4015547 (Security-only update)
|
|
March 21, 2017
|
KB 4012219
|
March 2017 Preview of Monthly Quality Rollup for Windows 8.1 and Windows Server 2012 R2
|
Windows Server 2016 RTM (RS1)
|
Release date
|
KB
|
Article title
|
|
May 16, 2017
|
KB 4023680
|
May 26, 2017—KB4023680 (OS Build 14393.1230)
|
|
May 9, 2017
|
KB 4019472
|
May 9, 2017—KB4019472 (OS Build 14393.1198)
|
|
April 11, 2017
|
KB 4015217
|
April 11, 2017—KB4015217 (OS Build 14393.1066 and 14393.1083)
|
Verification
- Verify the version of the following MSISCSI driver on the system:
c:\windows\system32\drivers\msiscsi.sys
The version that will expose this behavior is 6.3.9600.18624 for Windows Server 2012 R2 and version 10.0.14393.1066 for Windows Server 2016.
- The following events are logged in the System log:
|
Event source
|
ID
|
Text
|
|
iScsiPrt
|
34
|
A connection to the target was lost, but the Initiator successfully reconnected to the target. Dump data contains the target name.
|
|
iScsiPrt
|
39
|
The Initiator sent a task management command to reset the target. The target name is given in the dump data.
|
|
iScsiPrt
|
9
|
Target did not respond in time for a SCSI request. The CDB is given in the dump data.
|
- Review the number of threads that are running under the System process, and compare this to a known working baseline.
- Review the number of handles that are currently opened by the System process, and compare this to a known working baseline.
- Review the number of ephemeral ports that are being used by the System process.
- From an administrative Powershell, run the following command:
Get-NetTCPConnection | Group-Object -Property State, OwningProcess | Sort Count
Or, from an administrative CMD prompt, run the following NETSTAT command together with the "Q" switch. This shows "bound" ports that are no longer connected:
NETSTAT –ANOQ
Focus on ports that are owned by the SYSTEM process.
For the three previous points, anything more than 12,000 should be considered suspect. If iSCSI targets are present in the computer, there is high probability that the issue will occur.
Resolution
If the event logs indicate that many reconnections are occurring, work with your iSCSI and network fabric vendor to help diagnose and correct the reason for the failure to maintain connections to iSCSI targets. Make sure that iSCSI targets can be accessed over the current network fabric. Install updated fixes when they become available. This article will be updated with the specific KB article number of the fix to install when it becomes available.
Note We do not recommend that you uninstall any of the March, April, May, or June security rollups. Doing so will expose the computers to known security exploits and other bugs that are mitigated by monthly updates. We recommend that you first work with iSCSI target and network vendors to resolve the connectivity issues that are triggering target reconnects.