The key pairs (public-private) reside in Crypto API, a key container object. Applications do not need to know where and how the key container is implemented. The key container object should be an opaque object that contains the keys and is accessed by published interfaces.
The base Cryptographic Service Provider (CSP) implements the key containers in the registry. The CSP can reside in the HKEY_CURRENT_USER or HKEY_LOCAL_MACHINE. The location is determined by passing a flag to CryptAcquireContext. Other CSPs may implement the key container in a different location, such as a smart card. MSMQ does not assume anything about the location of the keys and always works with Crypto API, never directly with the keys themselves.
MSMQ uses the base CSP to have data encrypted or decrypted. While the current implementation for the CSP puts the private keys in HKLM\Software\Microsoft\Cryptography\MachineKeys\MSMQ, this should not be relevant information for any application.
As for authentication, it is a common mistake to think that the private and public keys reside in the certificate. A certificate is a public piece of information. It does not matter who receives the certificate. The private key, however, is a most secret piece of information that should be kept in a location that is as secure as possible.
For every certificate there should be some key container that is associated with it. This key container contains the public and private keys that are associated with the certificate. A copy of the public key is also placed in the certificate together with many other pieces of information.
Crypto API also defines an object that is called a Certificate Store. A Certificate Store contains certificates and additional information about each certificate (for example, information about the keys that are associated with the certificate). This information is an identification of the specific CSP that is used with the certificate and identification of the particular key container that is associated with the certificate.
A Certificate Store can reside in many places, the registry, a file, memory, and more. MSMQ stores the internal certificate in a Certificate Store under HKCU\Software\Microsoft\MSMQ\CertStore. The associated keys are located in HKCU\Software\Microsoft\Cryptography\UserKeys\MSMQ. These locations also should not be relevant to any application. MSMQ always works with Crypto API, never directly with the keys themselves.
MSMQ also stores the public key of a Queue Manager (QM) in the Message Queue Information Store (MQIS). Applications can retrieve this public key by calling MQGetMachineProperties() with PROPID_QM_ENCRYPTION_PK. The calling user should have "get security" permission granted on the computer in order to successfully retrieve the public key of the computer. PROPID_QM_ENCRYPTION_PK is unlike all other properties, where the user should have the "get permissions" permission on the computer.
MSMQ uses this public key internally to encrypt messages. The QM retrieves the public key of the destination computer from MQIS. Because in most cases the QM runs under the local system account, the "get permissions" privilege should be granted to everyone; otherwise, the QM fails to retrieve the public key and thus fails to encrypt the messages.