Notice: This website is an unofficial Microsoft Knowledge Base (hereinafter KB) archive and is intended to provide a reliable access to deleted content from Microsoft KB. All KB articles are owned by Microsoft Corporation. Read full disclaimer for more details.

Certain IDL constructs are marshaled and unmarshaled incorrectly by RPC


View products that this article applies to.

Important This article contains information about how to modify the registry. Make sure to back up the registry before you modify it. Make sure that you know how to restore the registry if a problem occurs. For more information about how to back up, restore, and modify the registry, click the following article number to view the article in the Microsoft Knowledge Base:
256986 (http://support.microsoft.com/kb/256986/ ) Description of the Microsoft Windows registry

↑ Back to the top


Symptoms

Programs that use remote procedure calls (RPC) to communicate with other Microsoft Windows-based computers over a network may fail. For example, when you use a custom Microsoft Visual Basic program between two remote computers, and the Visual Basic program uses user-defined type marshaling, you may find that certain Interface Definition Language (IDL) constructs are incorrectly marshaled and unmarshaled. For example, you may experience one of the following symptoms:
  • The remote call fails, and an RPC_X_BAD_STUB_DATA error code is returned by the Visual Basic program.
  • Incorrect data is unmarshaled.

↑ Back to the top


Cause

This problem may occur if the RPC component incorrectly marshals or unmarshals certain IDL constructs. Versions of Windows that may incorrectly marshal and unmarshal IDL constructs include the following:
  • Microsoft Windows XP (original version) and Windows XP Service Pack 1 (SP1) installations that include the MS03-039 security update. This update is documented in the following Microsoft Knowledge Base article:
    824146� MS03-039: A buffer overrun in RPCSS could allow an attacker to run malicious programs
  • Microsoft Windows 2000
  • Microsoft Windows NT 4.0
  • Microsoft Windows 98
This problem may occur in a Visual Basic program if both of the following conditions are true:
  • The IDL file uses a structure that contains an 8-byte field that uses any one of the following types:
    DATE
    Double
    CY
    VARIANT
    DECIMAL
    INT64/UINT64
  • The 8-byte field is a non-first field, and there is a field that is 4 bytes or smaller before the 8-byte field that is contained in the same structure.
For example, this problem may occur if the IDL file uses a structure that is similar to the following:
Public Type myType

member1 As Long

member2 As Double

End Type

↑ Back to the top


Resolution

Hotfix information for Windows Server 2003

To communicate with versions of Windows that incorrectly marshal and unmarshal IDL constructs, follow these steps:
  1. Do one of the following:
    • Install the latest service pack for Windows Server 2003. For more information, click the following article number to view the article in the Microsoft Knowledge Base:
      889100� How to obtain the latest service pack for Windows Server 2003
    • Install the security update that is described in security bulletin MS05-051. For more information about security bulletin MS05-051, click the following article number to view the article in the Microsoft Knowledge Base:
      902400� MS05-051: Vulnerabilities in MS DTC and COM+ could allow remote code execution
  2. Add the UDTAlignmentPolicy registry entry. To do this, follow these steps:

    Warning Serious problems might occur if you modify the registry incorrectly by using Registry Editor or by using another method. These problems might require that you reinstall your operating system. Microsoft cannot guarantee that these problems can be solved. Modify the registry at your own risk.
    1. Click Start, click Run, in the Open box, type regedit, and then click OK.
    2. Locate the following registry subkey:
      HKEY_LOCAL_MACHINE\Software\Microsoft\Rpc
    3. On the Edit menu, point to New, and then click DWORD value.
    4. Type UDTAlignmentPolicy, and then press ENTER.
    5. Double-click UDTAlignmentPolicy, and then type a value to determine the data format that RPC will use based on the following information:

      If the UDTAlignmentPolicy value is set to 0, RPC will marshal by using the "old" incorrect data format. RPC will communicate with versions of Windows that incorrectly marshal and unmarshal IDL constructs. For example, RPC will communicate with Windows 2000 or Windows XP SP1 installations that include the MS03-039 update.

      If the UDTAlignmentPolicy value is set to 0, RPC can communicate with Windows 2000-based computers.

      If the UDTAlignmentPolicy value is set to 1, RPC will marshal by using the "new" correct wire format. RPC will communicate with versions of Windows that correctly marshal and unmarshal IDL constructs. For example, RPC will communicate with Windows Server 2003 or Windows XP installations that include the MS03-026 update.

      If the UDTAlignmentPolicy entry does not exist, the remote call will fail, and RPC will return an RPC_X_BAD_STUB_DATA error code when RPC tries to marshal or unmarshal affected types.

      Important When the UDTAlignmentPolicy value is set to 0, your Windows Server 2003-based computer can exchange data for the IDL construct with versions of Windows that incorrectly marshal and unmarshal IDL constructs. However, the IDL construct is unmarshaled to the incorrect format. Using incorrect marshaling and unmarshaling code may prevent the interoperability of this same IDL construct and other IDL constructs when programs communicate with computers that run versions of Windows that correctly marshal and unmarshal IDL constructs.

Service pack information for Windows XP

To resolve this problem, obtain the latest service pack for Windows XP. For more information, click the following article number to view the article in the Microsoft Knowledge Base:
322389� How to obtain the latest Windows XP service pack
In Windows XP Service Pack 2, you can modify the UDTAlignmentPolicy registry entry to adjust the way RPC communicates with different versions of Windows. To do this, follow step 3 in the "Hotfix information for Windows Server 2003" section.

Hotfix information for Windows XP

To communicate with versions of Windows that incorrectly marshal and unmarshal IDL constructs, such as Windows 2000 or Windows XP SP1 installations that include the MS03-039 update, install the MS03-039 update on your Windows XP SP1-based computers. For more information about the MS03-039 update, click the following article number to view the article in the Microsoft Knowledge Base:
824146� MS03-039: A buffer overrun in RPCSS could allow an attacker to run malicious programs
A supported hotfix is not currently available for Windows XP-based computers that communicate by using RPC with versions of Windows that correctly marshal and unmarshal IDL constructs. These versions are Windows Server 2003 and Windows XP installations that include the MS03-026 update.

↑ Back to the top


Status

Microsoft has confirmed that this is a problem in the Microsoft products that are listed in the "Applies to" section. This problem was first corrected in Microsoft Windows XP Service Pack 2.

↑ Back to the top


More information

Versions of Windows that correctly marshal and unmarshal the IDL construct that is mentioned in the "Cause" section include the following:
  • Microsoft Windows Server 2003 and later versions
  • Microsoft Windows XP Service Pack 1 (SP1) installations that include the MS03-026 update but do not include the MS03-039 security update. The MS03-026 update is documented in the following Microsoft Knowledge Base article:
    823980� MS03-026: Buffer overrun in RPC interface may allow code execution
  • Microsoft Windows XP Service Pack 2 (SP2)
Note A supported hotfix is not currently available for Windows 2000-based computers that communicate by using RPC with versions of Windows that correctly marshal and unmarshal IDL constructs. These versions are Windows Server 2003 and Windows XP installations that include the MS03-026 update.

↑ Back to the top


Keywords: kbhotfixserver, kbqfe, kbtshoot, kbwinxpsp2fix, kbwinxppresp2fix, kbfix, kbbug, KB824835

↑ Back to the top

Article Info
Article ID : 824835
Revision : 19
Created on : 10/26/2006
Published on : 10/26/2006
Exists online : False
Views : 330