KB ID: 115900

Challenge

A hardware failure or system crash on the BDRSuite Backup Server machine can result in the loss of both the BDRSuite application and its backend database (PostgreSQL server).

Solution

In case of backup server failure/crash, If the backup data is stored on a separate storage device and the database dump files remain intact, you can rebuild or reconstruct the BDRSuite Backup Server from scratch.

Based on your requirement, you can perform rebuild of any of the following:

1. Rebuild the entire backup server
   
2. Rebuild the specific backup job
   
3. Rebuild the main database only
   

Follow the steps below to rebuild the BDRSuite Backup Server to its previous state (before failure) with all the backup jobs.

Note: The following steps are applicable only if you are using BDRSuite Backup Server v5.3 and above. After completing all the necessary steps, the rebuilding process may take some time to complete. This includes rebuilding jobs and making them available for restoration.

The following are required for rebuilding the BDRSuite Backup Server:

1. Backup Data (Mandatory)

1. Backup storage targets that hold backup data. All the backup data are stored inside "sgstorage" folder on the storage drives attached earlier. When more than one drive is configured for storage, ensure you have all of them.
   
2. For example, when E and H drives are configured as storage drives, then make sure you have both E:\sgstorage, H:\sgstorage.
   

2. Database Dump File (Mandatory):

1. By default, BDRSuite Backup Server configurations are stored in the database. This is taken as a dump and stored in the backup repository.
   
2. The dump file is stored inside the "sgstorage" folder of the 'first volume/drive' that has been configured as the backup repository in your previous setup. You will have the last three days dump which you can identify based on the timestamp added to the dump folder name. Make sure to use the latest dump for rebuild.
   

Database Dump File name format is as follows: SGDatabasexxxxxxxxxxxx.sql. Example: SGDatabase202301110000.sql

Database Dump file Location (example):

Windows:<repositorylocation>\sgstorage\<reponame>\VembuDBDumps\SGDatabasexxxxxxxxxx_xx.sql

E:\sgstorage\DefaultRepo\VembuDBDumps\SGDatabase_2022_06_01_00_00.sql

Ubuntu - <repository location="">/sgstorage/DefaultRepo/SGDatabase/VembuDBDumps/SGDatabasexxxxxxxxxx_xx.sql

/sgstorage/DefaultRepo/SGDatabase/VembuDBDumps/SGDatabase_2022_06_01_00_00.sql

3. BDR ID & Product Version:

You need to install the same backup server version and also, you need to provide the same BDR ID of your old backup server while installing the new backup server.

1. Login to your Vembu Portal account.
   
2. Navigate to Account -> Registered BDR Servers. Under BDRSuite (On-Premise) -> Registered Backup & Offsite DR Servers, you can find the BDR ID and Version of the backup server you want to rebuild.
3. Navigate to the Tab: Downloads. Select the version of your old backup server and download it.

4. Rebuild Script:

Make sure you have downloaded the Rebuild Script. Click to download the Rebuild Script zip file.

Steps to Rebuild Backup Server

Rebuilding the BDRSuite Backup Server involves the following processes:

1. Install BDRSuite Backup Server & Update the configuration file.
   
2. Attach the backup data & database dump file to the backup server.
   
3. Download and Run Rebuild Scripts.
   

Step 1: Install BDRSuite Backup Server & Update the configuration file

1. Download and Install the same version of the BDRSuite Backup Server which was running earlier on your machine. Refer to prerequisites section 3rd point.
   
2. Stop the BDRSuite service.
   
3. Open ‘[BDRSuite-Home]/conf/SGConfiguration.conf’ file and set the previously used BDR ID (old server) in the field - StoreGrid ID="Enter the BDR ID".
   
4. As mentioned in the prerequisites section (refer to 3rd point), you can get the BDR ID from Vembu Portal (if already registered) or you can refer to the value from the SGConfiguration.conf file from the previous BDRSuite Backup Server installation (if you have that old conf folder).
   

Step 2: Attach the backup data & database dump file to the same location on the backup server

Follow the steps below to attach backup data & database dump files to the backup server:

1. Verify that all necessary drives are present. As already mentioned in prerequisites (1st point), if you have configured multiple drives as backup storage targets, make sure you have all of them.
   

Note: All the backup data and database dump file will be available in the folder "sgstorage" on the drives that have been configured as repository. It is important to ensure that backup data storage folder need to be only with the name "sgstorage".

1. Finally, copy the existing backup data & database dump file to the new server. Follow the steps below if you were using any of the following storage targets previously.
   

1. Local Volume/External Hard Drive: If the backup data was on local volume in the previous server, make sure to use the same drive letter in the new server too. If the backup data was stored on an external hard drive, attach the drive to the same volume it was attached to on the previous server. For example, if it was connected to the E drive, make sure to connect it to the E drive on the new server.
   
2. SAN LUNs as Local Volumes on backup server: If the backup data was stored on a SAN and mounted as local volumes on the previous server, mount the volumes on the new server using the same drive letter as they were mounted on the previous server.
   
3. Network Drives (File Shares directly added to the backup server) - Make sure to add the network drive using the same drive letter as it was mapped to on the previous BDRSuite Backup Server.
   
4.  In case, if you want to use a different drive letter or storage- follow the steps in this article to attach the storage to the new server: https://www.bdrsuite.com/pdf/kb/steps-to-migrate-backup-storage-to-a-different-drive.pdf
   

Step 3: Download and Run Rebuild Scripts

Click to download the Rebuild Script zip file. The zip file contains:

1. requirements.txt
   
2. rebuild.json - The rebuild.json file contains the default installation directories of the BDRSuite Backup Server. **Note:** Open the rebuild.json in any text editor and edit the following if you have customized the following fields during installation: 
   
1. If you've changed the default installation location, then edit the windowsbdrlocation or linuxbdrlocation.
   
2. For Linux BDRSuite Server: If you've changed the PostgreSQL port number during installation, update the field: postgres_portnumber.
   
3. rebuild.py
   

Steps to run the Rebuild Automation Script (rebuild.py)

Note: Before you run the rebuild script, make sure all the BDRSuite Services are running.

Step 1: Open command prompt with administrative privileges.

Step 2: Change the working directory to where you have download the python script folder and run the file using the command given below:

Windows

BDRSuite v5.3 to v5.5:

C:\Rebuild >> "C:\Program Files\Vembu\BackupForAWS\Python37\python.exe" rebuild.py

BDRSuite v5.6 & above:

C:\Rebuild >> "C:\Program Files\Vembu\VembuBDR\Python37\python.exe" rebuild.py

Linux

BDRSuite v5.3 & above:

home/rebuild >> "/home/vembubdr/Vembu/BackupForAWS/Python37/bin/python" rebuild.py

Step 3: After the script has run, you will be asked to select one of the rebuild options to continue.

You can choose one of the following options based on your requirement and proceed.

Option 1: Rebuild the Main Database Only.

Option 2: Rebuild the Entire BDRSuite Server (No Change in Backup Repository Locations).

Option 3: Rebuild Specific Backup Job.

Option 1: Rebuild the Main Database Only (Skip this option, if you are planning to rebuild the entire backup server)

1. This option allows you to rebuild the BDRSuite Backup Server main database alone (Server & Backup configurations). Then you can manually rebuild a specific backup job.
   
2. For this option, you will need the Main database dump. Note: The main database dump will be stored on the backup repository automatically. You can find the dump file inside the "sgstorage" folder of the first volume/drive that has been configured as repository. Refer to prerequisite section - 2nd point.
   
3. Once the script has run, you will be asked the following details. To rebuild the main database only, follow the steps below:
   

Please enter the value of the Rebuild Type You wish to proceed with: 1

Type "1" and press enter to choose the rebuild type as "Rebuild Main Database Only"

Please specify the type of server for which you wish to perform rebuild:

1. BDRSuite Backup Server Rebuild.
   
2. BDRSuite OffsiteDR Server Rebuild.
   

Type "1" and press enter to choose to rebuild the "BDRSuite Backup Server"

Please enter the full path of the main database dump file location to perform rebuild:

Refer to prerequisites section - 2nd point for the database dump location and provide the location in the mentioned format.

Example:

Windows: Please enter full path of the main database dump file location to perform rebuild:

E:\sgstorage\automationbvtpool\VembuDBDumps\SGDatabase_2023_01_24_00_00.sql

Linux: Please enter full path of the main database dump file location to perform rebuild:

/sgstorage/automationbvtpool/VembuDBDumps/SGDatabase_2023_01_24_00_00.sql

Option 2: Rebuild the Entire BDRSuite Server (No Change in Backup Repository Locations)

This option allows you to reconstruct entire BDRSuite Backup Server if the BDRSuite machine is lost due to corruption or other reasons.

To rebuild backup server using this option, you will need the following;

1. Backup data: Please note that the drive letters of the backup storage locations should not be changed. (Refer step 2: Attach the backup data & database dump file to the same location on the backup server).
   
2. Main database dump (Refer prerequisite 2nd point).
   

Please enter the value of the Rebuild Type You wish to proceed with:

Type "2" and press enter to choose the rebuild type as "Rebuild the Entire BDRSuite Server"

Please specify the type of server for which you wish to perform rebuild:

1. BDRSuite Backup Server Rebuild
   
2. BDRSuite OffsiteDR Server Rebuild
   

[1 or 2]:

Type "1" and press enter to choose to rebuild the "BDRSuite Backup Server"

Please enter the full path of the main database dump file location to perform rebuild:

Refer to prerequisites section - 2nd point for the database dump location and provide the location in the mentioned format.

Example:

Windows: Please enter full path of the main database dump file location to perform rebuild:

E:\sgstorage\automationbvtpool\VembuDBDumps\SGDatabase_2023_01_24_00_00.sql

Linux: Please enter full path of the main database dump file location to perform rebuild:

/sgstorage/automationbvtpool/VembuDBDumps/Recentdump.sql

Option 3: Rebuild Specific Backup Job

This option will be helpful if you have many backup jobs and you want to rebuild a particular backup immediately if the shared DB for a particular backup job is corrupted.

Once the script has run, you will be asked the following details. To rebuild the specific backup job, follow the steps below:

Please enter the value of the Rebuild Type You wish to proceed with:

Type "3" and press enter to choose the rebuild type as "Rebuild Specific Backup Job"

Please specify the type of server for which you wish to perform rebuild:

1. BDRSuite Backup Server Rebuild.
   
2. BDRSuite OffsiteDR Server Rebuild.
   

[1 or 2]:

Type "1" and press enter to choose to rebuild the "BDRSuite Backup Server"

Enter the name of the backup job that needs to be rebuild:

Here you need to enter the backup job name that you want to rebuild.

Example: Enter the name of the backup job that needs to be rebuild: vmware-job-1

Enter the name of the client to which the backup job is assigned:

Here you need to enter the BDR ID of the BDRSuite Server/Client from which the backup job was configured.

Example: Enter the name of the client to which the backup job is assigned: backupserver01

Step 4: After completing all the necessary steps and successful script execution, you can log in to the backup server. The backup jobs will be listed and the rebuild process for each will start.

Note: The rebuilding process may take some time to complete. By default, 10 rebuild tasks will be active. Once the rebuild is successful for a backup job, you can initiate recovery for the VMs/Hosts in the particular backup job.