Fix file backup interruption on vShere 7.0.x after restore

Symptoms

vCenter has been restored from file backup after failed update and the file based backup start and stop few seconds with this message :

BackupManager encountered an exception. See logs for details. https://zhsvc3.skyguide.corp/appliance/support-bundle

You can check on the logs on /var/log/vmware/sso/vmware-identity-sts.log if you found this entry :

2023-02-10T14:24:46.239Z INFO sts[81:tomcat-http--44] [CorId=a6bc3d94-4a4f-4692-9809-1dc9962d0513] [com.vmware.identity.sts.InvalidCredentialsException] Censored exception

com.vmware.identity.sts.InvalidCredentialsException: Solution user's certificate does not match the one in BST!

This issue can be solved with the recreation of the solutions user's with this script from the vmware support (many thanks for this) available bellow

/!\ stop the vCenter or the both vCenter's if linked and do a cold snapshot and restart

connect to vcenter using SSH

transfer the file to the /tmp folder on the vcenter

make the file executable by running

chmod +x “Recreate_Solution_Users7.0.X.sh”

run the script: ./Recreate_Solution_Users7.0.X

Restart services of the VCSA.

#!/bin/bash #Cesar Badilla Tuesday, February 22, 2022 7:46:30 AM #Note: this Script needs to be run in all VCSA's in SSO. DOMAIN=$(/opt/likewise/bin/lwregshell list_values '[HKEY_THIS_MACHINE\Services\vmafd\Parameters]'| grep DomainName| awk '{print $4}'| tr -d '"') SOLUSERS=(hvc,wcp,vpxd-extension,vpxd,vsphere-webclient,machine) echo "####################################################################################" echo "####################################################################################" echo "This Script removes and recreates all Solution Users from Current VCSA 7.0.x." echo "If Enhanced Linked Mode(ELM) please make sure you run this script on Each VCSA of the SSO domain that is missing a solution user." echo "####################################################################################" echo "####################################################################################" SSOPASS="" while [[ -z $SSOPASS ]]; do read -s -p "Please enter the administrator@$DOMAIN password: " SSOPASS; echo; done MACHINE=$(/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost) #Remove all solution users from SSO domain read -p "Do you want to remove all Solution Users in SSO domain (Y|y|N|n)" -n 1 -r if [[ ! $REPLY =~ ^[Yy]$ ]] then exit 1 fi echo "####################################################################################" echo "Deleting Solution Users..." echo "####################################################################################" for i in $(/usr/lib/vmware-vmafd/bin/dir-cli service list --login administrator@$DOMAIN --password "$SSOPASS"| awk -F '.' '{print $2}' | awk '{print $1}'|grep -i $MACHINE); do echo y | /usr/lib/vmware-vmafd/bin/dir-cli service delete --name $i --login administrator@$DOMAIN --password "$SSOPASS"; done # Recreate Solution users for 7.0.x VCSA's. echo "####################################################################################" echo "Exporting new Solution Users Certificates" echo "####################################################################################" rm -rf /etc/certs/wcp && mkdir /etc/certs/wcp 2>/dev/null /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store wcp --alias wcp --output /etc/certs/wcp/wcp.crt rm -rf /etc/certs/hvc && mkdir /etc/certs/hvc 2>/dev/null /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store hvc --alias hvc --output /etc/certs/hvc/hvc.crt rm -rf /etc/certs/machine && mkdir /etc/certs/machine 2>/dev/null /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store machine --alias machine --output /etc/certs/machine/machine.crt rm -rf /etc/certs/vpxd && mkdir /etc/certs/vpxd 2>/dev/null /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd --alias vpxd --output /etc/certs/vpxd/vpxd.crt rm -rf /etc/certs/vpxd-extension && mkdir /etc/certs/vpxd-extension 2>/dev/null /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /etc/certs/vpxd-extension/vpxd-extension.crt rm -rf /etc/certs/vsphere-webclient && mkdir /etc/certs/vsphere-webclient 2>/dev/null /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vsphere-webclient --alias vsphere-webclient --output /etc/certs/vsphere-webclient/vsphere-webclient.crt echo "####################################################################################" read -p "Do you want to recreate all Solution Users in current VCSA. (Y|y|N|n)" -n 1 -r if [[ ! $REPLY =~ ^[Yy]$ ]] then exit 1 fi echo "####################################################################################" echo "Recreating new Solution Users" #Recreating new Solution Users. echo "The new Solution Users are: " for i in $(ls /etc/certs/); do (/usr/lib/vmware-vmafd/bin/dir-cli service create --name $i-$MACHINE --cert /etc/certs/$i/$i.crt --ssogroups ComponentManager.Administrators,SystemConfiguration.Administrators,LicenseService.Administrators --ssoadminrole Administrator --wstrustrole ActAsUser --login administrator@$DOMAIN --password "$SSOPASS"); done echo "####################################################################################" echo "The new Solution Users are: " /usr/lib/vmware-vmafd/bin/dir-cli service list --login administrator@$DOMAIN --password "$SSOPASS" echo "####################################################################################" echo "####################################################################################" echo "Please restart services of this VCSA." echo "####################################################################################"