Tag Archives: backup

Veeam backup and Replication – Failed to call RPC function ‘Vss.TruncateSqlLogs’

Today i have this warning in my Veeam Backup and Replication backup job log, for an SQL server:
Failed to finalize guest processing. Details: Failed to call RPC function ‘Vss.TruncateSqlLogs’: Error code: 0x80004005. Failed to invoke func [TruncateSqlLogs]: Unspecified error. Failed to process ‘TruncateSQLLog’ command. Failed to truncate SQL server transaction logs for instances: . See guest helper log.

I found this in the event viewer on the SQL server:
BACKUP failed to complete the command BACKUP LOG [databasename]. Check the backup application log for detailed messages.

So i check the veeam backup log file on the SQL server:
C:\ProgramData\Veeam\Backup
File VeeamGuestHelper_Lastdatefile.log

Where i see this:

19-12-2022 02:08:56 5080 Database found: [databasename]. Recovery model: 1. Is readonly: false. State: 0.
19-12-2022 02:13:56 5080 WARN Cannot truncate SQL logs for database: [databasename]. Code = 0x80040e31
19-12-2022 02:13:56 5080 WARN Code meaning = IDispatch error #3121
19-12-2022 02:13:56 5080 WARN Source = Microsoft OLE DB Driver for SQL Server
19-12-2022 02:13:56 5080 WARN Description = Query timeout expired
19-12-2022 02:13:56 5080 WARN No OLE DB Error Information found: hr = 0x80004005

What i did to fix this error is that i give the user: NT AUTHORITY\SYSTEM the rights db_backupoperator rights to the specific database, that fault.

And also change this timeout value in regedit:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication]
"SqlExecTimeout"=dword:00000600
"SqlLogBackupTimeout"=dword:00003600
"SqlConnectionTimeout"=dword:00000300

And also under WOW6432Node

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\VeeaM\Veeam Backup and Replication]
"SqlLogBackupTimeout"=dword:00003600
"SqlExecTimeout"=dword:00000600
"SqlConnectionTimeout"=dword:00000300

Then try to running the job again.

MSSQL Backup History Query

To see MSSQL history Query, run this SQL query. It will show all backups between the 1 january 2021 and 1 february 2021, order by Database and then Last backup time.

It will also show if the backup is a CopyOnly job.

;WITH CTE_Backup AS
(
SELECT  database_name,is_copy_only,backup_start_date,type,physical_device_name
       ,Row_Number() OVER(PARTITION BY database_name,BS.type
        ORDER BY backup_start_date DESC) AS RowNum
FROM    msdb..backupset BS
JOIN    msdb.dbo.backupmediafamily BMF
ON      BS.media_set_id=BMF.media_set_id
AND		BS.backup_start_date >= '01/01/2021'
AND		BS.backup_start_date <= '02/02/2021'
)
SELECT      D.name
           ,ISNULL(CONVERT(VARCHAR,backup_start_date),'No backups') AS last_backup_time
           ,D.recovery_model_desc
           ,state_desc,
            CASE WHEN type ='D' THEN 'Full database'
            WHEN type ='I' THEN 'Differential database'
            WHEN type ='L' THEN 'Log'
            WHEN type ='F' THEN 'File or filegroup'
            WHEN type ='G' THEN 'Differential file'
            WHEN type ='P' THEN 'Partial'
            WHEN type ='Q' THEN 'Differential partial'
            ELSE 'Unknown' END AS backup_type
           ,physical_device_name,CTE.is_copy_only
FROM        sys.databases D
LEFT JOIN   CTE_Backup CTE
ON          D.name = CTE.database_name

ORDER BY    D.name,last_backup_time

mssql get the backup information with copy_only information.

I need to find a way to see if the backup chain where broken. On my search I found this could script, created by “hot2use”
I have found this script on a forum.
https://dba.stackexchange.com/questions/204883/how-to-tell-if-a-backup-log-chain-is-broken

The script is very cool, as it will tell you the backup information, from a database. All you need to do is change this ‘<DATABASENAME>’ to yours database name, example ‘MASTER’.
It will also include if a job where COPY ONLY, and where the backup is save.

You can see in the screenshot below, that someone have run a full backup and save it on the D:\ drive, but it where a copy only so the backup is not broken.

/* ==================================================================
 Author......:  hot2use 
 Date........:  25.04.2018
 Version.....:  0.1
 Server......:  localhost (first created for)
 Database....:  msdb
 Owner.......:  -
 Table.......:  various
 Type........:  Script
 Name........:  ADMIN_Retrieve_Backup_History_Information.sql
 Description.:  Retrieve backup history information from msdb database
 ............   
 ............   
 ............       
 History.....:   0.1    h2u First created
 ............       
 ............       
================================================================== */
SELECT /* Columns for retrieving information */
       -- CONVERT(CHAR(100), SERVERPROPERTY('Servername')) AS SRVNAME, 
       msdb.dbo.backupset.database_name,
       msdb.dbo.backupset.backup_start_date,
       msdb.dbo.backupset.backup_finish_date,
       -- msdb.dbo.backupset.expiration_date, 

       CASE msdb.dbo.backupset.type
            WHEN 'D' THEN 'Full'
            WHEN 'I' THEN 'Diff'
            WHEN 'L' THEN 'Log'
       END  AS backup_type,
       -- msdb.dbo.backupset.backup_size / 1024 / 1024 as [backup_size MB],  
       msdb.dbo.backupmediafamily.logical_device_name,
       msdb.dbo.backupmediafamily.physical_device_name,
       -- msdb.dbo.backupset.name AS backupset_name,
       -- msdb.dbo.backupset.description,
       msdb.dbo.backupset.is_copy_only,
       msdb.dbo.backupset.is_snapshot,
       msdb.dbo.backupset.checkpoint_lsn,
       msdb.dbo.backupset.database_backup_lsn,
       msdb.dbo.backupset.differential_base_lsn,
       msdb.dbo.backupset.first_lsn,
       msdb.dbo.backupset.fork_point_lsn,
       msdb.dbo.backupset.last_lsn
FROM   msdb.dbo.backupmediafamily
       INNER JOIN msdb.dbo.backupset
            ON  msdb.dbo.backupmediafamily.media_set_id = msdb.dbo.backupset.media_set_id 

        /* ----------------------------------------------------------------------------
        Generic WHERE statement to simplify selection of more WHEREs    
        -------------------------------------------------------------------------------*/
WHERE  1 = 1

       /* ----------------------------------------------------------------------------
       WHERE statement to find Device Backups with '{' and date n days back
       ------------------------------------------------------------------------------- */
       -- AND     physical_device_name LIKE '{%'

       /* -------------------------------------------------------------------------------
       WHERE statement to find Backups saved in standard directories, msdb.dbo.backupfile AS b 
       ---------------------------------------------------------------------------------- */
       -- AND     physical_device_name  LIKE '[fF]:%'                          -- STANDARD F: Backup Directory
       -- AND     physical_device_name  NOT LIKE '[nN]:%'                      -- STANDARD N: Backup Directory

       -- AND     physical_device_name  NOT LIKE '{%'                          -- Outstanding Analysis
       -- AND     physical_device_name  NOT LIKE '%$\Sharepoint$\%' ESCAPE '$' -- Sharepoint Backs up to Share
       -- AND     backupset_name NOT LIKE '%Galaxy%'                           -- CommVault Sympana Backup


       /* -------------------------------------------------------------------------------
       WHERE Statement to find backup information for a certain period of time, msdb.dbo.backupset AS b 
       ---------------------------------------------------------------------------------- 
       AND    (CONVERT(datetime, msdb.dbo.backupset.backup_start_date, 102) >= GETDATE() - 7)  -- 7 days old or younger
       AND    (CONVERT(datetime, msdb.dbo.backupset.backup_start_date, 102) <= GETDATE())  -- n days old or older

       */

       /* -------------------------------------------------------------------------------
       WHERE Statement to find backup information for (a) given database(s) 
       ---------------------------------------------------------------------------------- */
       AND database_name IN ('<DATABASENAME>') -- database names
       -- AND     database_name IN ('rtc')  -- database names

        /* -------------------------------------------------------------------------------
        ORDER Clause for other statements
        ---------------------------------------------------------------------------------- */
        --ORDER BY        msdb.dbo.backupset.database_name, msdb.dbo.backupset.backup_finish_date -- order clause

        ---WHERE msdb..backupset.type = 'I' OR  msdb..backupset.type = 'D'
ORDER BY
       --2,

       2       DESC,
       3       DESC 

CAPI2 Error in event viewer when taking backup

When taking backup of a SQL server i got this error in my event viewer, every times the backup is running:
Event Viewer Error:

Cryptographic Services failed while processing the OnIdentity() call in the System Writer Object.

Details:
AddLegacyDriverFiles: Unable to back up image of binary Microsoft Link-Layer Discovery Protocol.

System Error:
Access is denied.
.

Solution:

During backup a VSS process running under NETWORK_SERVICE account calls cryptcatsvc!CSystemWriter::AddLegacyDriverFiles(), which enumerates all the drivers records in Service Control Manager database and tries opening each one of them. , The function fails on MSLLDP record with “Access Denied” error.

Turned out it fails because MSLLDP driver’s security permissions do not allow NETWORK_SERVICE to access the driver record.

The binary security descriptor for the record is located here:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MsLldp\Security

It should be modified, I used SC.EXE and Sysinternals’ ACCESSCHK.EXE to fix it.

The original security descriptor looked like below:

>accesschk.exe -c mslldp

mslldp
  RW NT AUTHORITY\SYSTEM
  RW BUILTIN\Administrators
  RW S-1-5-32-549       <- these are server operators
  R  NT SERVICE\NlaSvc

No service account is allowed to access MSLLDP driver

The security descriptor for the drivers that were processed successfully looked this way:

>accesschk.exe -c mup

mup
  RW NT AUTHORITY\SYSTEM
  RW BUILTIN\Administrators
  R  NT AUTHORITY\INTERACTIVE
  R  NT AUTHORITY\SERVICE  <- this gives access to services

How to add access rights for NT AUTHORITY\SERVICE to MSLLDP service:

1. Run: SC sdshow MSLLDP

You’ll get something like below (SDDL language is documented on MSDN):

D:(D;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;BG)(A;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;SY)(A;;CCDCLCSWRPDTLOCRSDRCWDWO;;;BA)(A;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;SO)(A;;LCRPWP;;;S-1-5-80-3141615172-2057878085-1754447212-2405740020-3916490453)S:(AU;FA;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;WD)

2. Run: SC sdshow MUP

You’ll get:

D:(A;;CCLCSWRPWPDTLOCRRC;;;SY)(A;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;BA)(A;;CCLCSWLOCRRC;;;IU)(A;;CCLCSWLOCRRC;;;SU)S:(AU;FA;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;WD)

3. Take NT AUTHORITY\ SERVICE entry, which is (A;;CCLCSWLOCRRC;;;SU) and add it to the original MSLLDP security descriptor properly, right before the last S:(AU… group.

4. Apply the new security descriptor to MSLLDP service :

sc sdset MSLLDP D:(D;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;BG)(A;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;SY)(A;;CCDCLCSWRPDTLOCRSDRCWDWO;;;BA)(A;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;SO)(A;;LCRPWP;;;S-1-5-80-3141615172-2057878085-1754447212-2405740020-3916490453)(A;;CCLCSWLOCRRC;;;SU)S:(AU;FA;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;WD)


5. Check the result:

>accesschk.exe -c mslldp

mslldp
  RW NT AUTHORITY\SYSTEM
  RW BUILTIN\Administrators
  RW S-1-5-32-549
  R  NT SERVICE\NlaSvc
  R  NT AUTHORITY\SERVICE

6. Run you backup app, the error is gone for my Home Server backup.

!!! Do not forget to use your security descriptor for MSLLDP driver since I guess there can be some rare cases when its different for your machine. Do not copy my SDDL descriptions, just in case. And backup the old descriptor just in case !!!

Thanks to szz743
https://answers.microsoft.com/en-us/windows/forum/windows8_1-hardware/cryptographic-services-failed-while-processing-the/c4274af3-79fb-4412-8ca5-cee721bda112

Ahsay restore Microsoft SQL (MSSQL) server database

If you want to restore a Microsoft SQL database, from a fuld backup, and put some transaction log file into it.  (This backup has backup by ahsay)

This is the why to do it:

  1. Start Microsoft SQL Server Management Studio
  2. Right click on Databases, and select restore database

  3. Type a database “To database”, and select from device , and press the butten with the 3 dots (…)

  4. Select file, and then click Add

  5. Select the database file you want to restore, and click ok

  6. Then select options, and click “Leave the database non-operational, and do not roll back uncommitted transactions. (Restore with NORECOVERY), then click OK
  7. Start a new query, and type this (Change it to yours database and files).
  8. RESTORE LOG RestoreTest
    FROM DISK = ‘F:\RestoreTest\2011-09-13(09-00-00)_LOG_RestoreTest_2011.bak’
    WITH NORECOVERYRESTORE LOG Farma_test
    FROM DISK = ‘F:\RestoreTest\2011-09-13(10-00-00)_LOG_RestoreTest_2011.bak’
    WITH NORECOVERY

    RESTORE LOG Farma_test
    FROM DISK = ‘F:\RestoreTest\2011-09-13(11-00-00)_LOG_RestoreTest_2011.bak’
    WITH NORECOVERY

    RESTORE LOG Farma_test
    FROM DISK = ‘F:\RestoreTest\2011-09-13(12-00-00)_RestoreTest_2011.bak’
    WITH NORECOVERY

  9. When you have takende the sekund last backup file, just type this:
  10. RESTORE LOG Farma_test
    FROM DISK = ‘F:\RestoreTest\2011-09-13(13-00-00)_RestoreTest_2011.bak’
    WITH RECOVERY

Backup of Exchange 2010 database level and document level with Ahsay

Installation of the Ashay Client:

Download the Ahsay Client:
http://eval.ahsay.com/obs/download/obm-win.exe

After the installation please apply the latest hotfix.
http://download.ahsay.com/support/hot-fixes/55/obm-win-hotfix.zip

Put the content of the hotfix into the folder: c:\program files\ahsayobm\

Disable Continous Backup services

Database level:
To backup the database level, just select Exchange Database Level, and choise Exchange 2010

Document level:
First install Microsoft Exchange Server MAPI Client and Collaboration Data Objects, download it from here: http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=1004
When you want to create a document level backup of Exchange 2007 or 2010 in ahsay you first need to run ahsay in 32bit java environment.
Please run c:\Program Files\AhsayOBM\bin\RunOBC32.bat
This will startup the Ahsay client in 32bit java environment. Then create a new backup job and set type to: MS Exchange Mail Level Backup, and give it a name.
If you get this: “DOMAIN\Administrator” might have insufficient permissions to the mailbox, please follow this steps.
a. Please first check that the Online Backup Scheduler is running under an acount that have access to the Exchange Mailbox, the “Local System” account does not have access to this, change it for to an account that have access, i use administrator.
1. Open [Control Panel] -> [Administrative Tools] -> [Services] -> [Online Backup Scheduler] -> [Log on]
2. Select the [This Account] option
3. Enter the Login Credentials
4. Restart the [Online Backup Scheduler] service
b. Please check that the account your have use is mail enabled. (Has an mailbox enable, and is not hidden from the default address book).
c. If the suggestions above cannot resolve the issue, please also verify if the MAPI profile used by OBM is configured properly. You could do so by following the instructions below:
1. Download the MAPI editor (MfcMapi.exe)
2. Open the MAPI editor
3. Select [Profile], and then [Show Profile]
4. Right click on the [Online Backup Manager] profile
5. Select [Open Profile]
6. Right click on the [Microsoft Exchange Server] entry and then select [Configure Service]
7. Enter the correct information, and then [Apply]
8. Once done, please restart the OBM client

Now select witch mailbox and public folders you want to backup with document level backup, and click next
Select the scheduled you want to run, and click next
Select the encrypting and click ok, now the backup is setup, and you can close this window, remember to click Save settings.

If you ran into any problems, look here:
If you get this: “DOMAIN\Administrator” might have insufficient permissions to the mailbox, please follow this steps.

a. Please first check that the Online Backup Scheduler is running under an acount that have access to the Exchange Mailbox, the “Local System” account does not have access to this, change it for to an account that have access, i use administrator.

Open [Control Panel] -> [Administrative Tools] -> [Services] -> [Online Backup Scheduler] -> [Log on]
Select the [This Account] option
Enter the Login Credentials
Restart the [Online Backup Scheduler] service

b. Please check that the account your have use is mail enabled. (Has an mailbox enable, and is not hidden from the default address book).
c. If the suggestions above cannot resolve the issue, please also verify if the MAPI profile used by OBM is configured properly. You could do so by following the instructions below:
1. Download the MAPI editor (MfcMapi.exe)
2. Open the MAPI editor
3. Select [Profile], and then [Show Profile]
4. Right click on the [Online Backup Manager] profile
5. Select [Open Profile]
6. Right click on the [Microsoft Exchange Server] entry and then select [Configure Service]
7. Enter the correct information, and then [Apply]
8. Once done, please restart the OBM client

Arcserve R12 and Exchange

If you have follow my guide howto to backup exchange from Arcserve R12, but still have problems with backup the exchange server.
Try to search in global address book, in outlook if you can find the user this way, if you can’t that the problems.
In our setup we got this problem on our Hosted Exchange server, the problems was resolve with adding a user and putting it in our own global address book.

Ahsay: Howto move user to a new “user home” folder

  1. Logon to the backup server [Administration Console]
  2. Under the [Manage System] page, enter another [User Home] (in this case, E:\User) in the [New] textbox provided and press the [Update] button
  3. Shutdown the backup service from [Control Panel] -> [Administrative Tools] -> [Services] -> [Ahsay Offsite Backup Server]
  4. Move the user directory to the new user home (e.g. D:\user\xxxx -> E:\User\xxxx)
  5. Startup the backup service from [Control Panel] -> [Administrative Tools] -> [Services] -> [Ahsay Offsite Backup Server]

[ad]