Troubleshooting system state backup of a protected server in System Center Data Protection Manager 2007

Solution to Symptom 1: Make sure the PSdataSourceConfig.xml file is located in the DPM agent install path under the datasources folder (e.g. C:\Microsoft Data Protection Manager\DPM\datasources). The Agent Install path can be checked in the registry under: 

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Data Protection Manager\Setup\InstallPath

If the file is in the wrong path or missing all together, you can copy it from another protected server and male sure it's in the correct location.

See http://technet.microsoft.com/en-us/library/bb809015.aspx for more details.


Solution to Symptom 2:


PROTECTED SERVER APPLICATION EVENT LOG

Windows Server 2003:

Event Type: Error
Event Source: NTBackup
Event Category: None
Event ID: 8019
Date:
Time: 
User: N/A
Computer: 
Description:
End Operation: Warnings or errors were encountered.
Consult the backup report for more details.

Windows Server 2008:

Log Name: Microsoft-Windows-Backup
Source: Microsoft-Windows-Backup
Date: 
Event ID: 5
Task Category: None
Level: Error
Keywords:
User: SYSTEM
Computer: 
Description:
Backup started at '<time/date>' failed with following error code '2155348165'.

Windows Server 2003 ntbackup logs for system state backup are stored in C:\Document and Settings\Default User\Application Data\Microsoft\NTBackup.

The Log files will be named BackupXX.log, Backup01.log, and so forth, look at the time stamp of the file to determine which one matches the failed job.

Windows Server 2008 backup logs are located in the %windir%\logs\WindowsServerBackup folder

The log files are named "SystemStatebackup date time.log"

You can view these logs to help resolve any issues that occur with the system state backup.


EXAMPLE NTBACKUP BACKUP01.LOG

Backup Status
Operation: Backup
Active backup destination: File
Media name: "FS01.bkf created 12/24/2007 at 8:01 PM"

Volume shadow copy creation: Attempt 1.

Error returned while creating the volume shadow copy:0x80042316.

Error returned while creating the volume shadow copy:80042316
Aborting Backup.

----------------------

The operation did not successfully complete.

----------------------


Where 0x80042316 = VSS_E_SNAPSHOT_SET_IN_PROGRESS - so it looks like in this case if you had systemstate in a separate protection group, you would need to modify the protection group to start the systemstate recovery point at an earlier or later time so the snapshot creation does not collide with other protection groups that may be protecting data on the O.S. (C:) drive. 

NOTE: This is a known issue and is fixed in Windows 2008 R2.

NOTE: If you are protecting both systemstate and data on the Windows O.S. (C:) drive or any other volumes that contain systemstate component in the same protection group, then DPM should schedule the recovery points for each data source seperately and not try to backup the data on C: and Systemstate at the same time. Look to see if DPM incorrectly scheduled / started the systemstate backup job before the job for protected data on C: completed. As a workaround - If you cannot get the systemstate to succeed, try putting it in its own protection group and schedule it for an earlier time than the original protection group it came from. 

NOTE: If the DPM server itself gets into a condition that it cannot create a snapshot for the same reason, recovery point jobs will fail with this error:

Cannot create a recovery point because another recovery point creation is in progress. (ID 211)

SOLUTION:

See this article, but perhaps SQL backups are scheduled first, thus causing systemstate backup to fail.

971807 - Data Protection Manager 2007 jobs to protect a SQL Server may remain in synchronizing state

If you use vssadmin list shadows, you may or may not see shadow copies of type:Backup. To stop any in progress snapshots, try stopping and restarting the Volume Shadow Copy service. If that does not work, reboot the system, then retry your DPM job.

If the error in the backup report is different than 0x80042316 then follow the steps below:

TROUBLESHOOTING

1) Check the application and system event log of the Protected server to see why the backup of systemstate failed.
2) For Windows 2008 - Make sure Windows 2008 has the backup feature installed.

3) VSS errors are one of the most common problems encountered, see the following article for WSB troubleshooting

http://blogs.technet.com/filecab/archive/2009/09/16/diagnosing-failures-in-windows-server-backup-part-1-vss-spp-errors.aspx

4) Make sure there is enough free space on C: volume to hold the PS_SERVER_NAME.BKF file located in the C:\DPM_SYSTEM_STATE folder.

Note: For Windows Server 2008 the systemstate is stored in drive:\WindowsImagebackup folder. For Windows 2008, systemstate backup may require ~10GB or MORE free space.

5) You can move (redirect) the systemstate backup location to another volume by modifying the \Microsoft Data Protection Manager\DPM\datasources\PSdataSourceConfig.xml file. See http://technet.microsoft.com/en-us/library/bb809015.aspx for details.

If you move it, unprotect systemstate then re-protect it on the DPM server so it re-reads the new location from the PSdataSourceConfig.xml file.

6) Try to make a manual systemstate backup and troubleshoot any errors that occur.

On Windows Server 2003, DPM executes the C:\program file\Microsoft Data Protection Manager\DPM\bin\DPMSystemstatebackup.cmd <drive-letter:> - you can run that and see what errors occur. On Windows Server 2008, DPM executes the C:\program file\Microsoft Data Protection Manager\DPM\bin\LHSystemstatebackup.cmd <drive-letter:> - you can run that and see what errors occur.

For Windows Server 2008 you can also try the following native backup command. The backup logs are located in C:\Windows\logs\WindowsServerBackup folder.

Wbadmin start systemstatebackup -backuptarget:c:

NOTE: Shadow copies for the target drive cannot be redirected to another volume or backup will not start. Use vssadmin list shadowstorage to confirm.

7) For Windows Server 2008, make sure this registry setting is enabled (which is added when a DPM agent is installed to a Windows 2008 server).

HKLM\Systtem\CurrentControlSet\Services\wbengine\SystemStateBackup\AllowSSBToAnyVolume:REG_DWORD:0x1

Once a manual NTBACKUP / Wbadmin Systemstate backups work, re-try the creating the recovery point.


Solution to Symptom 3:

A) Find out the writer ID of the writer you wish to exclude. You can use vssadmin list writers for this. The DPM writer is always going to be: {0074a271-4f24-43e0-9ec7-28a44189cb80}

B) Open regedit, the under the regkey "HKLM\Software\Microsoft\Windows NT\CurrentVersion\SPP", create a new MULTI_SZ-value with the name
"DisabledWriters".

C) Enter the writer guid with braces {0074a271-4f24-43e0-9ec7-28a44189cb80} as the value.

D) Shut down wbengine service if it’s running.

E) Try to create a new systemstate backup now, it should succeed.


Solution to Symptom 4:

This issue is typically caused by locating the system state component on a volume larger than 2 TB in size. All volumes containing a system state component like Sysvol and AD must be located on a volume less that 2TB in order to be protected.