Yearly Archives: 2022

Veeam backup and Replication – Failed to call RPC function ‘Vss.TruncateSqlLogs’

Today i have this warning in my Veeam Backup and Replication backup job log, for an SQL server:
Failed to finalize guest processing. Details: Failed to call RPC function ‘Vss.TruncateSqlLogs’: Error code: 0x80004005. Failed to invoke func [TruncateSqlLogs]: Unspecified error. Failed to process ‘TruncateSQLLog’ command. Failed to truncate SQL server transaction logs for instances: . See guest helper log.

I found this in the event viewer on the SQL server:
BACKUP failed to complete the command BACKUP LOG [databasename]. Check the backup application log for detailed messages.

So i check the veeam backup log file on the SQL server:
C:\ProgramData\Veeam\Backup
File VeeamGuestHelper_Lastdatefile.log

Where i see this:

19-12-2022 02:08:56 5080 Database found: [databasename]. Recovery model: 1. Is readonly: false. State: 0.
19-12-2022 02:13:56 5080 WARN Cannot truncate SQL logs for database: [databasename]. Code = 0x80040e31
19-12-2022 02:13:56 5080 WARN Code meaning = IDispatch error #3121
19-12-2022 02:13:56 5080 WARN Source = Microsoft OLE DB Driver for SQL Server
19-12-2022 02:13:56 5080 WARN Description = Query timeout expired
19-12-2022 02:13:56 5080 WARN No OLE DB Error Information found: hr = 0x80004005

What i did to fix this error is that i give the user: NT AUTHORITY\SYSTEM the rights db_backupoperator rights to the specific database, that fault.

And also change this timeout value in regedit:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication]
"SqlExecTimeout"=dword:00000600
"SqlLogBackupTimeout"=dword:00003600
"SqlConnectionTimeout"=dword:00000300

And also under WOW6432Node

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\VeeaM\Veeam Backup and Replication]
"SqlLogBackupTimeout"=dword:00003600
"SqlExecTimeout"=dword:00000600
"SqlConnectionTimeout"=dword:00000300

Then try to running the job again.

Expand an EXT4, on a SCSI device

If you have vmware ESXi or Microsoft Hyper-v, you can expand a disk, from the hypervisor console, very simple. But afterwards the partition the servers is not expand.
It is always a good idea to create a Snapshot/checkpoint before. Just remember, that the disc can’t be expanded in Hyper-v when there are a checkpoint. So first expand the disk in Hyper-v and then create a snapshot.

To do this you first need to rescan the disk for change

echo "1" > /sys/class/block/sda/device/rescan

Replace sda with the disk name.
Then you can run fdisk -l /dev/sda, to see the disk information, if you got an error on the top like in the screenshot below

Then you need to repair the disk partition table.

parted -l

After you have check the partition table, you need to delete the old partition, and recreate it

This is done by type fdisk /dev/sda

In the fdisk, you type d to delete, choice the partition, in the example is two. (Beaware that this guide can only be used to expand the last partition on the disk, if you have multiple)
Then you type n to create a new one, and just goes for the default once settings.
In the end type N to not remove the signature.

Then you w to save the settings

Then we just have to resize the ext4 information on the disk, this is done by the command resize2fs

Afterwards you can check that the server have been expand by the command df -h

Here we have expand the disk from 195GB to 295GB.

Get Certificate from remote computers

If you need to get all certificates, from a list of remote computers, you can use this script:

$Servers = "kennethdalbjerg-dc01",
           "kennethdalbjerg-dc02",
           "kennethdalbjerg-exch01",
           "kennethdalbjerg-fs01"
 
  $Results = @()
  $Results = Invoke-Command -cn $Servers {
          $Certs = @{} | Select Certificate,Expired
          $Cert = Get-ChildItem Cert:\LocalMachine\My
          If($Cert){
              $Certs.Certificate = $Cert.subject
              $Certs.Expired = $Cert.NotAfter
          }
          Else{
              $Certs.Certificate = " - "
              $Certs.Expired = " - "
          }
          $Certs
  } | Select-Object @{n='ServerName';e={$_.pscomputername}},Certificate,Expired
 
  #Display results in console
  $Results | Sort-Object Expired -Descending
 
  #Save results to CSV file
  $Results | Sort-Object Expired -Descending | Export-Csv -Path C:\users\$env:username\desktop\cert_results.csv -NoTypeInformation -Force
 
  #Open results in new window
  $Results | Sort-Object Expired -Descending | Out-GridView -Title "Final results"

Found groups that a 365 contacts is member of

If you need to found out whitch groups (mailling groups) a contact is member of in Microsoft 365, then you case use these powershell commands:

$getsessions = Get-PSSession | Select-Object -Property State, Name
$isconnected = (@($getsessions) -like '@{State=Opened; Name=ExchangeOnlineInternalSession*').Count -gt 0
If ($isconnected -ne "True") {
Connect-ExchangeOnline
}

$dn = (Get-MailContact "[email protected]").DistinguishedName
Get-Recipient -Filter "Members -eq '$dn'"

Replace [email protected] with yours mail address.

SNMP OID list for iLO4

Here is a list of SNMP OID list for iLO4, to used in yours monitoring software like Atera.

Fans:
.1.3.6.1.4.1.232.6.2.6.7.1.2.0 (Fan Index)
.1.3.6.1.4.1.232.6.2.6.7.1.3.0 (Fan Locale (1=other, 2=unknown, 3=system, 4=systemBoard, 5=ioBoard, 6=cpu, 7=memory, 8=storage, 9=removable media, 10=power supply, 11=ambent, 12=chassis, 13=bridge card, 14=management board, 15=backplane, 16=network slot, 17=blade slot, 18=virtual)
.1.3.6.1.4.1.232.6.2.6.7.1.4.0 (Fan Present (1=other, 2=absent, 3=present)
.1.3.6.1.4.1.232.6.2.6.7.1.5.0 (Fan Present (1=other, 2=tachOutput, 3=spinDetect)
.1.3.6.1.4.1.232.6.2.6.7.1.6.0 (Fan Speed (1=other, 2=normal, 3=high)
.1.3.6.1.4.1.232.6.2.6.7.1.9.0 (Fan Condition (1=other, 2=ok, 3=degraded, 4=failed)

Temperature:
.1.3.6.1.4.1.232.6.2.6.8.1.2.0 (Temperature Sensor Index)
.1.3.6.1.4.1.232.6.2.6.8.1.3.0 (Temperature Sensor Locale (1=other, 2=unknown, 3=system, 4=systemBoard, 5=ioBoard, 6=cpu, 7=memory, 8=storage, 9=removable media, 10=power supply, 11=ambent, 12=chassis, 13=bridge card)
.1.3.6.1.4.1.232.6.2.6.8.1.7.0 (Threshold Type (1=other, 5=blowout, 9=caution, 15=critical, 16=noreaction)
.1.3.6.1.4.1.232.6.2.6.8.1.4.0 (Temperature Celsius)
.1.3.6.1.4.1.232.6.2.6.8.1.5.0 (TemperatureThreshold)
.1.3.6.1.4.1.232.6.2.6.8.1.6.0 (TemperatureCondition)

CPU:
.1.3.6.1.4.1.232.1.2.2.1.1.1 (CPU Index)
.1.3.6.1.4.1.232.1.2.2.1.1.3 (CPU Name)
.1.3.6.1.4.1.232.1.2.2.1.1.4 (CPU Speed in MHz)
.1.3.6.1.4.1.232.1.2.2.1.1.5 (CPU Step)
.1.3.6.1.4.1.232.1.2.2.1.1.6 (CPU status (1=unknown, 2=ok, 3=degraded, 4=failed, 5=disabled)
.1.3.6.1.4.1.232.1.2.2.1.1.15 (Number of enabled CPU cores)
.1.3.6.1.4.1.232.1.2.2.1.1.25 (Number of available CPU threads)
.1.3.6.1.4.1.232.1.2.2.1.1.26 (CPU power status (1=unknown, 2=Low Powered, 3=Normal Powered, 4=High Powered)

Logical Drives:
.1.3.6.1.4.1.232.3.2.3.1.1.2.0 (Logical Drive Index)
.1.3.6.1.4.1.232.3.2.3.1.1.1.0 (Logical Drive Controller)
.1.3.6.1.4.1.232.3.2.3.1.1.3.0 (Logical Drive Fault Tolerance (1=other, 2=none, 3=RAID 1/RAID 1+0 (Mirroring), 4=RAID 4 (Data Guard), 5=RAID 5 (Distributed Data Guard), 7=RAID 6 (Advanced Data Guarding), 8=RAID 50, 9=RAID 60, 10=RAID 1 ADM (Advanced Data Mirroring), 11=RAID 10 ADM (Advanced Data Mirroring with Striping))
.1.3.6.1.4.1.232.3.2.3.1.1.9.0 (Logical Drive Size in Mb)
.1.3.6.1.4.1.232.3.2.3.1.1.4.0 (Logical Drive Status (1=other, 2=ok, 3=Failed, 4=Unconfigured, 5=Recovering, 6=Ready Rebuild, 7=Rebuilding, 8=Wrong Drive, 9=Bad Connect, 10=Overheating, 11=Shutdown, 12=Expanding, 13=Not Available, 14=Queued For Expansion, 15=Multi-path Access Degraded, 16=Erasing, 17=Predictive Spare Rebuild Ready, 18=Rapid Parity Initialization In Progress, 19=Rapid Parity Initialization Pending, 20=No Access – Encrypted with No Controller Key, 21=Unencrypted to Encrypted Transformation in Progress, 22=New Logical Drive Key Rekey in Progress, 23=No Access – Encrypted with Controller Encryption Not Enabled, 24=Unencrypted To Encrypted Transformation Not Started, 25=New Logical Drive Key Rekey Request Received)
.1.3.6.1.4.1.232.3.2.3.1.1.11.0 (Logical Drive Condition (1=other, 2=ok, 3=degraded, 4=failed)

Drives:
.1.3.6.1.4.1.232.3.2.5.1.1.2.0 (Drive Index)
.1.3.6.1.4.1.232.3.2.5.1.1.5.0 (Drive Bay)
.1.3.6.1.4.1.232.3.2.5.1.1.64.0 (Drive Location)
.1.3.6.1.4.1.232.3.2.5.1.1.3.0 (Drive Vendor)
.1.3.6.1.4.1.232.3.2.5.1.1.51.0 (Drive Serial Number)
.1.3.6.1.4.1.232.3.2.5.1.1.45.0 (Drive Size in Mb)
.1.3.6.1.4.1.232.3.2.5.1.1.65.0 (Drive Link Rate (1=other, 2=1.5Gbps, 3=3.0Gbps, 4=6.0Gbps, 5=12.0Gbps))
.1.3.6.1.4.1.232.3.2.5.1.1.70.0 (Drive Current Temperature)
.1.3.6.1.4.1.232.3.2.5.1.1.71.0 (Drive Temperature Threshold)
.1.3.6.1.4.1.232.3.2.5.1.1.72.0 (Drive Maximum Temperature)
.1.3.6.1.4.1.232.3.2.5.1.1.6.0 (Drive Status (1=Other, 2=Ok, 3=Failed, 4=Predictive Failure, 5=Erasing, 6=Erase Done, 7=Erase Queued, 8=SSD Wear Out, 9=Not Authenticated)
.1.3.6.1.4.1.232.3.2.5.1.1.37.0 (Drive Condition (1=other, 2=ok, 3=degraded, 4=failed)
.1.3.6.1.4.1.232.3.2.5.1.1.9.0 (Drive Reference Time in hours)

iLO NIC:
.1.3.6.1.4.1.232.9.2.5.2.1.1 (iLO location)
.1.3.6.1.4.1.232.9.2.5.1.1.2 (iLO NIC model)
.1.3.6.1.4.1.232.9.2.5.1.1.4 (iLO NIC MAC)
.1.3.6.1.4.1.232.9.2.5.1.1.5 (iLO NIC IPv4)
.1.3.6.1.4.1.232.9.2.5.1.1.9 (iLO NIC speed)
.1.3.6.1.4.1.232.9.2.5.1.1.14 (iLO NIC FQDN)
.1.3.6.1.4.1.232.9.2.5.2.1.2 (Tx bytes)
.1.3.6.1.4.1.232.9.2.5.2.1.3 (Tx packets)
.1.3.6.1.4.1.232.9.2.5.2.1.6 (Tx discard packets)
.1.3.6.1.4.1.232.9.2.5.2.1.7 (Tx error packets)
.1.3.6.1.4.1.232.9.2.5.2.1.9 (Rx bytes)
.1.3.6.1.4.1.232.9.2.5.2.1.10 (Rx packets)
.1.3.6.1.4.1.232.9.2.5.2.1.13 (Rx discard packets)
.1.3.6.1.4.1.232.9.2.5.2.1.14 (Rx error packets)
.1.3.6.1.4.1.232.9.2.5.2.1.15 (Rx unknown packets)

Memory:
.1.3.6.1.4.1.232.6.2.14.13.1.1 (Memory Index)
.1.3.6.1.4.1.232.6.2.14.13.1.13 (Location)
.1.3.6.1.4.1.232.6.2.14.13.1.9 (Manufacturer)
.1.3.6.1.4.1.232.6.2.14.13.1.10 (Part Number)
.1.3.6.1.4.1.232.6.2.14.13.1.6 (Size in Kbytes)
.1.3.6.1.4.1.232.6.2.14.13.1.8 (Memory Technology)
.1.3.6.1.4.1.232.6.2.14.13.1.7 (Memory Type)
.1.3.6.1.4.1.232.6.2.14.13.1.19 (Memory status (1=other, 2=notPresent, 3=present, 4=good, 5=add, 6=upgrade, 7=missing, 8=doesNotMatch, 9=notSupported, 10=badConfig, 11=degraded, 12=spare, 13=partial)
.1.3.6.1.4.1.232.6.2.14.13.1.20 (Memory condition (1=other, 2=ok, 3=degraded, 4=degradedModuleIndexUnknown)

Extend a raidset with STORCLI CLI

You can use this command to see the status of the Raid Controller and Raidset.

storcli /c0 show
Generating detailed summary of the adapter, it may take a while to complete.

Controller = 0
Status = Success
Description = None

Product Name = Intel(R) RAID Controller RS3DC080
Serial Number = SK94678290
SAS Address =  500605b00f7ef7e0
PCI Address = 00:b5:00:00
System Time = 03/18/2022 15:44:57
Mfg. Date = 11/24/19
Controller Time = 03/18/2022 15:44:29
FW Package Build = 24.21.0-0126
BIOS Version = 6.36.00.3_4.19.08.00_0x06180203
FW Version = 4.680.00-8519
Driver Name = lsi-mr3
Driver Version = 7.702.13.00
Current Personality = RAID-Mode
Vendor Id = 0x1000
Device Id = 0x5D
SubVendor Id = 0x8086
SubDevice Id = 0x9360
Host Interface = PCI-E
Device Interface = SATA-3G
Bus Number = 181
Device Number = 0
Function Number = 0
Drive Groups = 1

TOPOLOGY :
========

-----------------------------------------------------------------------------
DG Arr Row EID:Slot DID Type  State BT       Size PDC  PI SED DS3  FSpace TR
-----------------------------------------------------------------------------
 0 -   -   -        -   RAID5 Optl  N    5.451 TB dflt N  N   dflt N      N
 0 0   -   -        -   RAID5 Optl  N    5.451 TB dflt N  N   dflt N      N
 0 0   0   252:4    8   DRIVE Onln  N  930.390 GB dflt N  N   dflt -      N
 0 0   1   252:5    9   DRIVE Onln  N  930.390 GB dflt N  N   dflt -      N
 0 0   2   252:6    10  DRIVE Onln  N  930.390 GB dflt N  N   dflt -      N
 0 0   3   252:7    11  DRIVE Onln  N  930.390 GB dflt N  N   dflt -      N
 0 0   4   252:0    15  DRIVE Onln  N  930.390 GB dflt N  N   dflt -      N
 0 0   5   252:1    12  DRIVE Onln  N  930.390 GB dflt N  N   dflt -      N
 0 0   6   252:2    13  DRIVE Onln  N  930.390 GB dflt N  N   dflt -      N
-----------------------------------------------------------------------------

DG=Disk Group Index|Arr=Array Index|Row=Row Index|EID=Enclosure Device ID
DID=Device ID|Type=Drive Type|Onln=Online|Rbld=Rebuild|Dgrd=Degraded
Pdgd=Partially degraded|Offln=Offline|BT=Background Task Active
PDC=PD Cache|PI=Protection Info|SED=Self Encrypting Drive|Frgn=Foreign
DS3=Dimmer Switch 3|dflt=Default|Msng=Missing|FSpace=Free Space Present
TR=Transport Ready

Virtual Drives = 1

VD LIST :
=======

-------------------------------------------------------------
DG/VD TYPE  State Access Consist Cache Cac sCC     Size Name
-------------------------------------------------------------
0/0   RAID5 Optl  RW     Yes     RWBD  -   ON  5.451 TB
-------------------------------------------------------------

EID=Enclosure Device ID| VD=Virtual Drive| DG=Drive Group|Rec=Recovery
Cac=CacheCade|OfLn=OffLine|Pdgd=Partially Degraded|Dgrd=Degraded
Optl=Optimal|RO=Read Only|RW=Read Write|HD=Hidden|TRANS=TransportReady|B=Blocked|
Consist=Consistent|R=Read Ahead Always|NR=No Read Ahead|WB=WriteBack|
AWB=Always WriteBack|WT=WriteThrough|C=Cached IO|D=Direct IO|sCC=Scheduled
Check Consistency

Physical Drives = 8

PD LIST :
=======

-------------------------------------------------------------------------------------
EID:Slt DID State DG       Size Intf Med SED PI SeSz Model                   Sp Type
-------------------------------------------------------------------------------------
252:0    15 Onln   0 930.390 GB SATA SSD N   N  512B Samsung SSD 870 QVO 1TB U  -
252:1    12 Onln   0 930.390 GB SATA SSD N   N  512B Samsung SSD 870 QVO 1TB U  -
252:2    13 Onln   0 930.390 GB SATA SSD N   N  512B Samsung SSD 870 QVO 1TB U  -
252:3    14 UGood  - 930.390 GB SATA SSD N   N  512B Samsung SSD 870 QVO 1TB U  -
252:4     8 Onln   0 930.390 GB SATA SSD N   N  512B Samsung SSD 860 EVO 1TB U  -
252:5     9 Onln   0 930.390 GB SATA SSD N   N  512B Samsung SSD 860 EVO 1TB U  -
252:6    10 Onln   0 930.390 GB SATA SSD N   N  512B Samsung SSD 860 EVO 1TB U  -
252:7    11 Onln   0 930.390 GB SATA SSD N   N  512B Samsung SSD 860 EVO 1TB U  -
-------------------------------------------------------------------------------------

EID=Enclosure Device ID|Slt=Slot No.|DID=Device ID|DG=DriveGroup
DHS=Dedicated Hot Spare|UGood=Unconfigured Good|GHS=Global Hotspare
UBad=Unconfigured Bad|Onln=Online|Offln=Offline|Intf=Interface
Med=Media Type|SED=Self Encryptive Drive|PI=Protection Info
SeSz=Sector Size|Sp=Spun|U=Up|D=Down|T=Transition|F=Foreign
UGUnsp=Unsupported|UGShld=UnConfigured shielded|HSPShld=Hotspare shielded
CFShld=Configured shielded|Cpybck=CopyBack|CBShld=Copyback Shielded


Cachevault_Info :
===============

---------------------------------------------------------
Model  State   Temp Mode MfgDate    Next Learn
---------------------------------------------------------
CVPM02 Optimal 23C  -    2020/04/09 2022/03/31  00:41:59
---------------------------------------------------------

Here you can see under PD List, that drive 252:3 14 stand as UGood. Thats the new disk i have put into the server, and have been detected by the raidset.

To include this disk into the array that you see under VD List, you need to do this

 storcli /c0/v0 start migrate type=raid5 option=add drives=252:3

This will add the disk to the raidset. This can takes hours and even days, if you have a big raidset.
You can check the status with this command:

storcli /c0/v0 show migrate
Controller = 0
Status = Success
Description = None


VD Operation Status :
===================

------------------------------------------------------------
VD Operation      Progress% Status      Estimated Time Left
------------------------------------------------------------
 0 Reconstruction         2 In progress 9 Hours 52 Minutes
------------------------------------------------------------

DG=Arrays | Slot=Drive Bay No|VD=Virtual Drive/Logical Drive|EID=Enclosure Device ID

Here you can see that it will take 9 hours and 52 minutes to complete the Reconstruction

When it done with the reconstuction, the command from before will output this:

VD Operation Status :
===================

-----------------------------------------------------------
VD Operation Progress% Status          Estimated Time Left 
-----------------------------------------------------------
 0 Migrate   -         Not in progress -                   
-----------------------------------------------------------

And then you can check the array with the command from the top of this guide

storcli /c0 show

After this you need to change the partition table, and then expand the volume.

MSSQL – Always On availability group – Get sync status

If you wan to know the sync status of a always On availability group database replication, this command can be used:

SELECT 
	ar.replica_server_name, 
	adc.database_name, 
	ag.name AS ag_name, 
	drs.is_local, 
	drs.is_primary_replica, 
	drs.synchronization_state_desc, 
	drs.is_commit_participant, 
	drs.synchronization_health_desc, 
	drs.recovery_lsn, 
	drs.truncation_lsn, 
	drs.last_sent_lsn, 
	drs.last_sent_time, 
	drs.last_received_lsn, 
	drs.last_received_time, 
	drs.last_hardened_lsn, 
	drs.last_hardened_time, 
	drs.last_redone_lsn, 
	drs.last_redone_time, 
	drs.log_send_queue_size, 
	drs.log_send_rate, 
	drs.redo_queue_size, 
	drs.redo_rate, 
	drs.filestream_send_rate, 
	drs.end_of_log_lsn, 
	drs.last_commit_lsn, 
	drs.last_commit_time
FROM sys.dm_hadr_database_replica_states AS drs
INNER JOIN sys.availability_databases_cluster AS adc 
	ON drs.group_id = adc.group_id AND 
	drs.group_database_id = adc.group_database_id
INNER JOIN sys.availability_groups AS ag
	ON ag.group_id = drs.group_id
INNER JOIN sys.availability_replicas AS ar 
	ON drs.group_id = ar.group_id AND 
	drs.replica_id = ar.replica_id
ORDER BY 
	ag.name, 
	ar.replica_server_name, 
	adc.database_name;