Configuring ExaCC backups of an Oracle Database

This post covers how to configure your backups of an ExaCC database beyond the web interface. 

First off the documentation can be found below, along with using the "--help" option at the command line with "bkup_api"

Configuration - https://docs.oracle.com/en/cloud/cloud-at-customer/exadata-cloud-at-customer/exacc/customize-backup-configuration-using-bkup_api.html

Backup execution - https://docs.oracle.com/en/cloud/cloud-at-customer/exadata-cloud-at-customer/exacc/create-demand-backup.html#GUID-2370EA04-3141-4D02-B328-5EE9A10F66F2

Step #1 - Configure backup settings in ExaCC

The next step is to configure my database to be backed up using the tooling. This is pretty straightforward. I click on the "edit backup" button and fill in the information for my database and save it.  In my case I am using ZFS, and the need to make sure that I change my container to the container where the ZFS is configured.

NOTE : The backup strategy is a Weekly L0 (full) backup every Sunday, and a daily L1 (differential incremental backup) on all other days. The time the backup is scheduled can be found in either the backup settings, or by looking at the crontab file.

Then I just wait until I see complete. If I click on the work requests, I can see the progress until it's finished.

Step #2 - Update the settings to use my RMAN catalog.

First I need to get what the current settings are for my database (dbsg2) and save them in a config file so I can update them.

I log into the first node, and su to root.

Once there I execute "get config --all" and save all the settings to a file that I can update.

NOTE : I an creating a new file under the bakup_api/cfg directory to make it easy to find.

$ sudo su -
Last login: Thu May  6 11:43:46 PDT 2021 on pts/0
[root@ecc ~]## /var/opt/oracle/bkup_api/bkup_api get config --all --file=/var/opt/oracle/bkup_api/cfg/dbsg2.cfg --dbname dbsg2
DBaaS Backup API V1.5 @2021 Multi-Oracle home
-> Action : get_config
-> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_92303612_20210506125612.006275.log
File /var/opt/oracle/bkup_api/cfg/dbsg2.cfg created

Now I am going to edit it and make some changes.

I changed to RMAN catalog settings to use my catalog.

NOTE: The entry has to be the connect string, not a tnsnames.ora entry.

#### This section is applicable when using a rman catalog ####
# Enables RMAN catalog. Can be set to yes or no.
bkup_use_rcat=yes

## Below parameters are required if rman catalog is enabled
# RMAN catalog user
bkup_rcat_user=rco


# RMAN catalog password
#bkup_rcat_passwd=RMan19c#_

# RMAN catalog conn string
bkup_rcat_conn=ecc-scan.bgrenn.com:1521:rmanpdb.bgrenn.com

Now I am going to commit (set) the changes using the "set config" command

# /var/opt/oracle/bkup_api/bkup_api set config --file=/var/opt/oracle/bkup_api/cfg/dbsg2.cfg --dbname dbsg2 
DBaaS Backup API V1.5 @2021 Multi-Oracle home
-> Action : set_config
-> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_b800281f_20210506130824.084259.log
cfgfile : /var/opt/oracle/bkup_api/cfg/dbsg2.cfg
Using configuration file: /var/opt/oracle/bkup_api/cfg/dbsg2.cfg
API::Parameters validated.
UUID d0845ea0aea611eb98fb52540068a695 for this set_config(configure-backup)
** process started with PID: 86143
** see log file for monitor progress
-------------------------------------

And after a few minutes, I am going to check and make sure it was successful by using the configure_status command

/var/opt/oracle/bkup_api/bkup_api configure_status --dbname dbsg2
DBaaS Backup API V1.5 @2021 Multi-Oracle home
-> Action : configure_status
-> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_fa81558e_20210507060019.504831.log
* Last registered operation: 2021-05-07 12:58:41 UTC 
* Configure backup status: finished
**************************************************
* API History: API steps
  API:: NEW PROCESS 120531
*
* RETURN CODE:0
##################################################

Everything looks good !  It removed my configuration file (which is good because it had the password in it).  

I found that 2 things happened as part of adding an RMAN catalog

1. The password  for the RMAN catalog user is now stored in the wallet file.
2. There is an entry in my tnsnames file on all nodes for "CATALOG" which points to the rman catalog.

NOTE: When part of this process is to register the database with the RMAN catalog. You do not have to manually register the database in the catalog.

Step #3 - Take a manual backup

Now logged in as OPC, and becoming Root, and can run a special backup using bkup_api

# /var/opt/oracle/bkup_api/bkup_api bkup_start --dbname=dbsg2
DBaaS Backup API V1.5 @2021 Multi-Oracle home
-> Action : bkup_start
-> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_9458c30f_20210510084341.430481.log
UUID 7f6622f8b1a611eb865552540068a695 for this backup
** process started with PID: 336757
** see log file for monitor progress
-------------------------------------

I can see the status while it's running

/var/opt/oracle/bkup_api/bkup_api bkup_status --dbname=dbsg2
DBaaS Backup API V1.5 @2021 Multi-Oracle home
-> Action : bkup_status
-> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_46545e6f_20210510084812.014419.log
(' Warning: unable to get current configuration of:', 'catalog')
* Current backup settings:
* Last registered Bkup: 05-10 15:44 UTC API::336757:: Starting dbaas backup process
* Bkup state: running
**************************************************
* API History: API steps
  API:: NEW PROCESS 336757
  API:: Starting dbaas backup process
*
* RETURN CODE:0
##################################################

And I waited a few minutes, and I can see it was successful.

# /var/opt/oracle/bkup_api/bkup_api bkup_status --dbname=dbsg2
DBaaS Backup API V1.5 @2021 Multi-Oracle home
-> Action : bkup_status
-> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_8acd03e3_20210510085129.207757.log
(' Warning: unable to get current configuration of:', 'catalog')
* Current backup settings:
* Last registered Bkup: 05-10 15:44 UTC API::336757:: Starting dbaas backup process
* Bkup state: running
**************************************************
* API History: API steps
  API:: NEW PROCESS 336757
  API:: Starting dbaas backup process
*************************************************
* Backup steps
 -> 2021-05-10 08:44:20.651787 - API:: invoked with args : -dbname=dbsg2 -uuid=7f6622f8b1a611eb865552540068a695 -level1 
 -> 2021-05-10 08:44:23.458698 - API:: Wallet is in open AUTOLOGIN state
 -> 2021-05-10 08:44:24.204793 - API:: Oracle database state is up and running
 -> 2021-05-10 08:44:25.686134 - API:: CATALOG SETTINGS 
 -> 2021-05-10 08:45:19.767284 - API:: DB instance: dbsg2
 -> 2021-05-10 08:45:19.767424 - API:: Validating the backup repository ...... 
 -> 2021-05-10 08:46:38.263401 - API::      All backup pieces are ok
 -> 2021-05-10 08:46:38.263584 - API:: Validating the TDE wallet ...... 
 -> 2021-05-10 08:46:41.842706 - API:: TDE check successful.
 -> 2021-05-10 08:46:42.446560 - API:: Performing incremental backup to shared storage
 -> 2021-05-10 08:46:42.448228 - API:: Executing rman instructions
 -> 2021-05-10 08:49:21.161884 - API:: ....... OK
 -> 2021-05-10 08:49:21.162089 - API:: Incremental backup to shared storage is Completed
 -> 2021-05-10 08:49:21.163822 - API:: Starting backup of config files
 -> 2021-05-10 08:49:21.699197 - API:: Determining the oracle database id
 -> 2021-05-10 08:49:21.726308 - API::  DBID: 2005517379
 -> 2021-05-10 08:49:22.040891 - API:: Creating directories to store config files
 -> 2021-05-10 08:49:22.085476 - API:: Enabling RAC exclusions for config files.
 -> 2021-05-10 08:49:22.114211 - API:: Compressing config files into tar files
 -> 2021-05-10 08:49:22.173842 - API:: Uploading config files to NFS location
 -> 2021-05-10 08:49:22.222493 - API:: Removing temporary location /var/opt/oracle/log/dbsg2/obkup/7f6622f8b1a611eb865552540068a695.
 -> 2021-05-10 08:49:22.224071 - API:: Config files backup ended successfully
 -> 2021-05-10 08:49:26.052494 - API:: All requested tasks are completed
*
* RETURN CODE:0
##################################################

Step #4 - Check my periodic backups

Now it's been a few days (I started on Thursday and it's now Monday).

I am going to check on the incremental backups, and the archive log backups.

There are 2 ways I can do this.

Using the bkup_api command to list the backups that have run.

# /var/opt/oracle/bkup_api/bkup_api list --dbname=dbsg2
DBaaS Backup API V1.5 @2021 Multi-Oracle home
-> Action : list
-> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_eddcd4e1_20210510064145.497707.log
-> Listing all backups
  Backup Tag             Completion Date (UTC)            Type          keep    
----------------------   -----------------------      -----------    --------
   TAG20210506T123203     05/06/2021 19:32:03       full        False
   TAG20210506T131438     05/06/2021 20:14:38       incremental        False
   TAG20210507T012240     05/07/2021 08:22:40       incremental        False
   TAG20210508T012315     05/08/2021 08:23:15       incremental        False
   TAG20210509T012438     05/09/2021 08:24:38       full        False
   TAG20210510T012322     05/10/2021 08:23:22       incremental        False

Using the RMAN catalog

Backup Type         Encrypted Tag                                Backup Piece                                                 Backup Time           Day Of Week
-------------------- --------- --------------------------------- ------------------------------------------------------------ -------------------- --------------------
Full L0              YES       DBAAS_FULL_BACKUP20210506122626     /backup/dbaas_bkup_DBSG2_2005517379_0dvu5rp2_13_1          05/06/21 12:29:32    THURSDAY
Differential L1      YES       DBAAS_INCR_BACKUP20210506131110     /backup/dbaas_bkup_DBSG2_2005517379_2avu5ud1_74_1          05/06/21 13:14:18    THURSDAY
Differential L1      YES       DBAAS_INCR_BACKUP20210507011926     /backup/dbaas_bkup_DBSG2_2005517379_72vu792b_226_1         05/07/21 01:22:27    FRIDAY
Differential L1      YES       DBAAS_INCR_BACKUP20210508011939     /backup/dbaas_bkup_DBSG2_2005517379_lbvu9tf3_683_1         05/08/21 01:22:51    SATURDAY
Full L0              YES       DBAAS_FULL_BACKUP20210509011940     /backup/dbaas_bkup_DBSG2_2005517379_u3vuchr8_963_1         05/09/21 01:22:59    SUNDAY
Differential L1      YES       DBAAS_INCR_BACKUP20210510011940     /backup/dbaas_bkup_DBSG2_2005517379_6rvuf672_1243_1        05/10/21 01:22:49    MONDAY

NOTE: I can see that a periodic L1 (differential) is executed at 1:22 AM, every day but Sunday where a Full backup is executed.

Now to look at archive log backups -- I am going to show a subset.

Again I can use the bkup_api "list_jobs" command and see all the backup jobs that have been run (which include archive logs).

# /var/opt/oracle/bkup_api/bkup_api list_jobs --dbname dbsg2 | more
DBaaS Backup API V1.5 @2021 Multi-Oracle home
-> Action : list_jobs
-> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_b2532724_20210510070545.552300.log
UUID                             | DATE                | STATUS  | TAG                 | ACTION              
e7ad1ef6aea011eb9c8252540068a695 | 2021-05-06 19:26:23 | success | TAG20210506T123203  | create-backup-full  
03616d68aea211eba5aa52540068a695 | 2021-05-06 19:34:12 | success | TAG20210506T123516  | archivelog-backup   
33fae162aea611eba0ed52540068a695 | 2021-05-06 20:04:12 | success | TAG20210506T130518  | archivelog-backup   
267c21daaea711eb9d3852540068a695 | 2021-05-06 20:11:07 | success | TAG20210506T131438  | create-backup-incremental
650fd222aeaa11ebb58652540068a695 | 2021-05-06 20:34:12 | success | TAG20210506T133516  | archivelog-backup   
961831e4aeae11ebb0d452540068a695 | 2021-05-06 21:04:11 | success | TAG20210506T140517  | archivelog-backup   
c6919f28aeb211eb957e52540068a695 | 2021-05-06 21:34:12 | success | TAG20210506T143518  | archivelog-backup   
f7ce0d0caeb611eb97c552540068a695 | 2021-05-06 22:04:12 | success | TAG20210506T150522  | archivelog-backup   
286e8ea6aebb11eb864c52540068a695 | 2021-05-06 22:34:11 | success | TAG20210506T153516  | archivelog-backup   
598f77eeaebf11eb92c052540068a695 | 2021-05-06 23:04:11 | success | TAG20210506T160518  | archivelog-backup   
89f4919aaec311eb9a9452540068a695 | 2021-05-06 23:34:11 | success | TAG20210506T163516  | archivelog-backup   
bb5ba95eaec711ebb1ed52540068a695 | 2021-05-07 00:04:11 | success | TAG20210506T170518  | archivelog-backup   
 

Step #5 - On demand backups 

Now that I have my database configured, I am going to demonstrate some of the options you can add to your backup.

I am going to create a keep backup and give it a tag using bkup_start

$ /var/opt/oracle/bkup_api/bkup_api bkup_start --dbname=dbsg2 --keep --tag=Maymonthlybackup
DBaaS Backup API V1.5 @2021 Multi-Oracle home
-> Action : bkup_start
-> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_7d923417_20210507113940.052080.log
UUID 958a58beaf6311eba98a52540068a695 for this backup
** process started with PID: 262102
** see log file for monitor progress
-------------------------------------

Now to list it.

$ /var/opt/oracle/bkup_api/bkup_api list --dbname dbsg2 --keep
DBaaS Backup API V1.5 @2021 Multi-Oracle home
-> Action : list
-> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_19714a18_20210507114254.007083.log
-> Listing all backups
  Backup Tag                           Completion Date (UTC)      Type          keep    
----------------------                 -----------------------   -----------    --------
   Maymonthlybackup20210507T113125     05/07/2021 18:31:25       keep-forever   True

Step #6 - Restore my database

The last step I'm going to do in my database is to restore it to a previous point in time.

Below is what you see in the console.

NOTE - If you chose a specific time it will be in UTC time.

I pick a time to restore to, and click on the 'Restore Database' option. I can follow the process by looking at 'Workload Requests'.

Step #7 - Validating backups

A great feature of the command tool is the ability to validate backups that have been taken.  This is easy to do with the 'bkup_api reval_start' command.

I started my validate for my database dbbsg and I saved the uuid to monitor it.

# /var/opt/oracle/bkup_api/bkup_api reval_start --dbname=dbbsg
DBaaS Backup API V1.5 @2021 Multi-Oracle home
-> Action : reval_start
-> logfile: /var/opt/oracle/log/dbbsg/bkup_api_log/bkup_api_d0647aa8_20210511032638.300613.log
UUID 5f204c4cb24311eb887252540068a695 for restore validation
** process started with PID: 15281
** Backup Request uuid     : 5f204c4cb24311eb887252540068a695

Now to monitor it using the uuid until it's done, and I can see it completed successfully.

# /var/opt/oracle/bkup_api/bkup_api --uuid=5f204c4cb24311eb887252540068a695 --dbname=dbbsg
DBaaS Backup API V1.5 @2021 Multi-Oracle home
@ STARTING CHECK STATUS 5f204c4cb24311eb887252540068a695
[ REQUEST TICKET ]
[UUID    ->  5f204c4cb24311eb887252540068a695 
[DBNAME  ->  dbbsg 
[STATE   ->  success 
[ACTION  ->  start-restore-validate 
[STARTED ->  2021-05-11 10:26:39 UTC 
[ENDED   ->  2021-05-11 10:28:00 UTC 
[PID     ->  15281 
[TAG     ->  None 
[PCT     ->  0.0 
[LOG     ->  2021-05-11 03:26:39.780830 - API:: invoked with args : -dbname=dbbsg -reval=default  
[LOG     ->  2021-05-11 03:26:42.324669 - API:: Wallet is in open AUTOLOGIN state 
[LOG     ->  2021-05-11 03:26:42.996885 - API:: Oracle database state is up and running 
[LOG     ->  2021-05-11 03:28:00.857565 - API:: ....... OK 
[LOG     ->  2021-05-11 03:28:00.857645 - API:: Restore Validation is Completed 
[ END TICKET ]

Step #8 - Restoring/listing/backups with API

There are many options to restoring with the API for both the "database" which consists of the CDB and all PDBs, or just a specific PDB.

Below are some of the commands that help with this.

NOTE: All commands are executed using "bkup_api" from /var/opt/oracle/bkup_api as "oracle"

Command	Options	Description
bkup_start		Start new special backup now
bkup_start	--keep	Create keep backup
bkup_start	--level0	Perform a new FULL level 0 backup
bkup_start	--level1	Perform a new level1 incremental backup
bkup_start	--cron	Creates an incremntal backup through Cron
bkup_chkcfg		Verifies that backups have been configured
bkup_status		Shows the status of the most recent backup
list		Shows the list of the most recent backups
reval_start		Starts a restore validation of datafiles
archreval_start		Starts a revalidation of archive logs
recover_start	--latest	Recover from latest backup
recover_start	--scn	Recover to SCN #
recover_start	--b	Recover using a specific backup tag and defuzzy to archivelog following
recover_start	-t	Recover to time. Specify --nonutc to use a non-UTC timestamp
recover_status		Show status of most recent recover of this database

With recovery you can also just recover a single PDB

• --pdb={pdbname} - Recovery just a single PDB

You can also specify if the config files should be restored

• --cfgfiles - store the configuration files (controlfiles, spfiles etc) along with database files.

Step #9 - Configuration changes

You can execute the "bkup_api get config --dbname={dbname}" to create a file containing the  current configuration.  In that file you can see some of the other changes you can be.

Below is what I see it using the version at the time of writing this.

Config Parameter	Settings	Description
bkup_cron_entry	yes/no	Enable/Disable automatic backups
bkup_archlog_cron_entry	yes/no	Enable automatic archive log cleanup when not using tooling
bkup_cfg_files	yes/no	Enable backup of Config files
bkup_daily_time	hh24:mi	Time to execute daily backup
bkup_archlog_frequency	15,20,30…	How many minutes apart to execute archive log backups
bkup_disk	yes/no	Backups to the FRA
bkup_disk_recovery_window	1-14	Recover window of FRA
bkup_oss_xxx		Backup settings when backing up to Object Store in Public Cloud
bkup_zdlra_xx		Backup settings when backing up to a ZDLRA
bkup_nfs_xxx		Backup settings when backing up to NFS
bkup_set_section_size	yes/no	Set to yes to over ride the default setting
bkup_section_size		Value for Over riding the default setting for section size
bkup_channels_node	xx	Number of channels to be used by RMAN
bkup_use_rcat	yes/no	If you are using an RMAN catalog
bkup_rcat_xxx		RMAN catalog settings

Step #10 - Scheduled backups

Backups are scheduled in the crontab on the first node of a cluster. You can view schedule by executing "sudo su - " to become root, and look at the /etc/crontab file.

Below is what is there for my database (dbsg2)

# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name  command to be executed

15 * * * * oracle /var/opt/oracle/misc/backup_db_wallets.pl
15 * * * * oracle /var/opt/oracle/dbaascli/dbaascli tde backup --alldb
19 1 * * * oracle /var/opt/oracle/bkup_api/bkup_api bkup_start --cron --dbname=dbsg2
4,34 * * * * oracle /var/opt/oracle/bkup_api/bkup_api bkup_archlogs --cron --dbname=dbsg2

The jobs that are scheduled to execute are.

1. backup_db_wallets.pl - Every 15 minutes this script executes from the Crontab.  This script goes through the list of databases (regardless if database backups have been turned on) and it makes a copy of the SEPS wallet file in the current wallet location adding the current date/time. The old copy is removed and only one back exists.

The following 2 settings are in my configuration file (/var/opt/oracle/creg/dbsg2.ini) are used as the source location of the wallet, and the location for the backup

 wallet_loc=/var/opt/oracle/dbaas_acfs/dbsg2/db_wallet

 wallet_loc_bak=/u02/app/oracle/admin/dbsg2/db_wallet

NOTE: This wallet is used for storing user credentials and is an autologin wallet.

I can see the credentials stored. In my case it is both the "sys" password, and the password for "rco". If I was using OSS (object store) my login credential would be stored in this wallet, and if I backed up to ZDLRA, this wallet would contain my connection to the ZDLRA(s) I was backing up to.

2: CATALOG rco

1: dbsg2 sys

2. /var/opt/oracle/dbaascli/dbaascli tde backup --alldb - Every 15 minutes this script executes from the Crontab.  This script goes through the list of databases (regardless if database backups have been turned on) and it makes a copy of the TDE wallet file in the $ORACLE_BASE directory.

The location is $ORACLE_BASE/{db_name}/tde_wallet/tde/

The output from this script is in /var/opt/oracle/log/misc/backup

3. /var/opt/oracle/bkup_api/bkup_api bkup_start --cron --dbname={mydb}- Every at 1:19 AM,  This time is determined in the bkup_api configuration using the parameter  "bkup_daily_time".  This is the same API that is called to perform an on-demand backup from the command line, but with a '--cron' parameter also.

4. /var/opt/oracle/bkup_api/bkup_api bkup_archlogs --dbname={mydb} Every 30 minutes this script is executed based on the 'bkup_archlog_frequency' bkup_api configuration setting. This script will backup my archive logs to the backup location.