Monday, May 10, 2021

Configuring ExaCC backups of an Oracle Database

This post covers how to configure your backups of an ExaCC database beyond the web interface. 


First off the documentation can be found below, along with using the "--help" option at the command line with "bkup_api"

Configuration - https://docs.oracle.com/en/cloud/cloud-at-customer/exadata-cloud-at-customer/exacc/customize-backup-configuration-using-bkup_api.html

Backup execution - https://docs.oracle.com/en/cloud/cloud-at-customer/exadata-cloud-at-customer/exacc/create-demand-backup.html#GUID-2370EA04-3141-4D02-B328-5EE9A10F66F2



    Step #1 - Configure backup settings in ExaCC

    The next step is to configure my database to be backed up using the tooling. This is pretty straightforward. I click on the "edit backup" button and fill in the information for my database and save it.  In my case I am using ZFS, and the need to make sure that I change my container to the container where the ZFS is configured.

    NOTE : The backup strategy is a Weekly L0 (full) backup every Sunday, and a daily L1 (differential incremental backup) on all other days. The time the backup is scheduled can be found in either the backup settings, or by looking at the crontab file.



    Then I just wait until I see complete. If I click on the work requests, I can see the progress until it's finished.



    Step #2 - Update the settings to use my RMAN catalog.

    First I need to get what the current settings are for my database (dbsg2) and save them in a config file so I can update them.

    I log into the first node, and su to root.
    Once there I execute "get config --all" and save all the settings to a file that I can update.

    NOTE : I an creating a new file under the bakup_api/cfg directory to make it easy to find.

    $ sudo su -
    Last login: Thu May  6 11:43:46 PDT 2021 on pts/0
    [root@ecc ~]## /var/opt/oracle/bkup_api/bkup_api get config --all --file=/var/opt/oracle/bkup_api/cfg/dbsg2.cfg --dbname dbsg2
    DBaaS Backup API V1.5 @2021 Multi-Oracle home
    -> Action : get_config
    -> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_92303612_20210506125612.006275.log
    File /var/opt/oracle/bkup_api/cfg/dbsg2.cfg created
    
    

    Now I am going to edit it and make some changes.

    I changed to RMAN catalog settings to use my catalog.
    NOTE: The entry has to be the connect string, not a tnsnames.ora entry.

    #### This section is applicable when using a rman catalog ####
    # Enables RMAN catalog. Can be set to yes or no.
    bkup_use_rcat=yes
    
    ## Below parameters are required if rman catalog is enabled
    # RMAN catalog user
    bkup_rcat_user=rco
    
    
    # RMAN catalog password
    #bkup_rcat_passwd=RMan19c#_
    
    # RMAN catalog conn string
    bkup_rcat_conn=ecc-scan.bgrenn.com:1521:rmanpdb.bgrenn.com
    
    
    

    Now I am going to commit (set) the changes using the "set config" command
    # /var/opt/oracle/bkup_api/bkup_api set config --file=/var/opt/oracle/bkup_api/cfg/dbsg2.cfg --dbname dbsg2 
    DBaaS Backup API V1.5 @2021 Multi-Oracle home
    -> Action : set_config
    -> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_b800281f_20210506130824.084259.log
    cfgfile : /var/opt/oracle/bkup_api/cfg/dbsg2.cfg
    Using configuration file: /var/opt/oracle/bkup_api/cfg/dbsg2.cfg
    API::Parameters validated.
    UUID d0845ea0aea611eb98fb52540068a695 for this set_config(configure-backup)
    ** process started with PID: 86143
    ** see log file for monitor progress
    -------------------------------------
    
    


    And after a few minutes, I am going to check and make sure it was successful by using the configure_status command

    
    /var/opt/oracle/bkup_api/bkup_api configure_status --dbname dbsg2
    DBaaS Backup API V1.5 @2021 Multi-Oracle home
    -> Action : configure_status
    -> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_fa81558e_20210507060019.504831.log
    * Last registered operation: 2021-05-07 12:58:41 UTC 
    * Configure backup status: finished
    **************************************************
    * API History: API steps
      API:: NEW PROCESS 120531
    *
    * RETURN CODE:0
    ##################################################
    
    
    Everything looks good !  It removed my configuration file (which is good because it had the password in it).  
    I found that 2 things happened as part of adding an RMAN catalog
    1. The password  for the RMAN catalog user is now stored in the wallet file.
    2. There is an entry in my tnsnames file on all nodes for "CATALOG" which points to the rman catalog.

    NOTE: When part of this process is to register the database with the RMAN catalog. You do not have to manually register the database in the catalog.

    Step #3 - Take a manual backup

    Now logged in as OPC, and becoming Root, and can run a special backup using bkup_api


    # /var/opt/oracle/bkup_api/bkup_api bkup_start --dbname=dbsg2
    DBaaS Backup API V1.5 @2021 Multi-Oracle home
    -> Action : bkup_start
    -> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_9458c30f_20210510084341.430481.log
    UUID 7f6622f8b1a611eb865552540068a695 for this backup
    ** process started with PID: 336757
    ** see log file for monitor progress
    -------------------------------------
    
    

    I can see the status while it's running

    /var/opt/oracle/bkup_api/bkup_api bkup_status --dbname=dbsg2
    DBaaS Backup API V1.5 @2021 Multi-Oracle home
    -> Action : bkup_status
    -> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_46545e6f_20210510084812.014419.log
    (' Warning: unable to get current configuration of:', 'catalog')
    * Current backup settings:
    * Last registered Bkup: 05-10 15:44 UTC API::336757:: Starting dbaas backup process
    * Bkup state: running
    **************************************************
    * API History: API steps
      API:: NEW PROCESS 336757
      API:: Starting dbaas backup process
    *
    * RETURN CODE:0
    ##################################################
    
    

    And I waited a few minutes, and I can see it was successful.

    # /var/opt/oracle/bkup_api/bkup_api bkup_status --dbname=dbsg2
    DBaaS Backup API V1.5 @2021 Multi-Oracle home
    -> Action : bkup_status
    -> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_8acd03e3_20210510085129.207757.log
    (' Warning: unable to get current configuration of:', 'catalog')
    * Current backup settings:
    * Last registered Bkup: 05-10 15:44 UTC API::336757:: Starting dbaas backup process
    * Bkup state: running
    **************************************************
    * API History: API steps
      API:: NEW PROCESS 336757
      API:: Starting dbaas backup process
    *************************************************
    * Backup steps
     -> 2021-05-10 08:44:20.651787 - API:: invoked with args : -dbname=dbsg2 -uuid=7f6622f8b1a611eb865552540068a695 -level1 
     -> 2021-05-10 08:44:23.458698 - API:: Wallet is in open AUTOLOGIN state
     -> 2021-05-10 08:44:24.204793 - API:: Oracle database state is up and running
     -> 2021-05-10 08:44:25.686134 - API:: CATALOG SETTINGS 
     -> 2021-05-10 08:45:19.767284 - API:: DB instance: dbsg2
     -> 2021-05-10 08:45:19.767424 - API:: Validating the backup repository ...... 
     -> 2021-05-10 08:46:38.263401 - API::      All backup pieces are ok
     -> 2021-05-10 08:46:38.263584 - API:: Validating the TDE wallet ...... 
     -> 2021-05-10 08:46:41.842706 - API:: TDE check successful.
     -> 2021-05-10 08:46:42.446560 - API:: Performing incremental backup to shared storage
     -> 2021-05-10 08:46:42.448228 - API:: Executing rman instructions
     -> 2021-05-10 08:49:21.161884 - API:: ....... OK
     -> 2021-05-10 08:49:21.162089 - API:: Incremental backup to shared storage is Completed
     -> 2021-05-10 08:49:21.163822 - API:: Starting backup of config files
     -> 2021-05-10 08:49:21.699197 - API:: Determining the oracle database id
     -> 2021-05-10 08:49:21.726308 - API::  DBID: 2005517379
     -> 2021-05-10 08:49:22.040891 - API:: Creating directories to store config files
     -> 2021-05-10 08:49:22.085476 - API:: Enabling RAC exclusions for config files.
     -> 2021-05-10 08:49:22.114211 - API:: Compressing config files into tar files
     -> 2021-05-10 08:49:22.173842 - API:: Uploading config files to NFS location
     -> 2021-05-10 08:49:22.222493 - API:: Removing temporary location /var/opt/oracle/log/dbsg2/obkup/7f6622f8b1a611eb865552540068a695.
     -> 2021-05-10 08:49:22.224071 - API:: Config files backup ended successfully
     -> 2021-05-10 08:49:26.052494 - API:: All requested tasks are completed
    *
    * RETURN CODE:0
    ##################################################
    
    


    Step #4 - Check my periodic backups


    Now it's been a few days (I started on Thursday and it's now Monday).
    I am going to check on the incremental backups, and the archive log backups.

    There are 2 ways I can do this.

    Using the bkup_api command to list the backups that have run.

    # /var/opt/oracle/bkup_api/bkup_api list --dbname=dbsg2
    DBaaS Backup API V1.5 @2021 Multi-Oracle home
    -> Action : list
    -> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_eddcd4e1_20210510064145.497707.log
    -> Listing all backups
      Backup Tag             Completion Date (UTC)            Type          keep    
    ----------------------   -----------------------      -----------    --------
       TAG20210506T123203     05/06/2021 19:32:03       full        False
       TAG20210506T131438     05/06/2021 20:14:38       incremental        False
       TAG20210507T012240     05/07/2021 08:22:40       incremental        False
       TAG20210508T012315     05/08/2021 08:23:15       incremental        False
       TAG20210509T012438     05/09/2021 08:24:38       full        False
       TAG20210510T012322     05/10/2021 08:23:22       incremental        False
    
    

    Using the RMAN catalog

    Backup Type         Encrypted Tag                                Backup Piece                                                 Backup Time           Day Of Week
    -------------------- --------- --------------------------------- ------------------------------------------------------------ -------------------- --------------------
    Full L0              YES       DBAAS_FULL_BACKUP20210506122626     /backup/dbaas_bkup_DBSG2_2005517379_0dvu5rp2_13_1          05/06/21 12:29:32    THURSDAY
    Differential L1      YES       DBAAS_INCR_BACKUP20210506131110     /backup/dbaas_bkup_DBSG2_2005517379_2avu5ud1_74_1          05/06/21 13:14:18    THURSDAY
    Differential L1      YES       DBAAS_INCR_BACKUP20210507011926     /backup/dbaas_bkup_DBSG2_2005517379_72vu792b_226_1         05/07/21 01:22:27    FRIDAY
    Differential L1      YES       DBAAS_INCR_BACKUP20210508011939     /backup/dbaas_bkup_DBSG2_2005517379_lbvu9tf3_683_1         05/08/21 01:22:51    SATURDAY
    Full L0              YES       DBAAS_FULL_BACKUP20210509011940     /backup/dbaas_bkup_DBSG2_2005517379_u3vuchr8_963_1         05/09/21 01:22:59    SUNDAY
    Differential L1      YES       DBAAS_INCR_BACKUP20210510011940     /backup/dbaas_bkup_DBSG2_2005517379_6rvuf672_1243_1        05/10/21 01:22:49    MONDAY
    
    
    

    NOTE: I can see that a periodic L1 (differential) is executed at 1:22 AM, every day but Sunday where a Full backup is executed.

    Now to look at archive log backups -- I am going to show a subset.

    Again I can use the bkup_api "list_jobs" command and see all the backup jobs that have been run (which include archive logs).


    # /var/opt/oracle/bkup_api/bkup_api list_jobs --dbname dbsg2 | more
    DBaaS Backup API V1.5 @2021 Multi-Oracle home
    -> Action : list_jobs
    -> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_b2532724_20210510070545.552300.log
    UUID                             | DATE                | STATUS  | TAG                 | ACTION              
    e7ad1ef6aea011eb9c8252540068a695 | 2021-05-06 19:26:23 | success | TAG20210506T123203  | create-backup-full  
    03616d68aea211eba5aa52540068a695 | 2021-05-06 19:34:12 | success | TAG20210506T123516  | archivelog-backup   
    33fae162aea611eba0ed52540068a695 | 2021-05-06 20:04:12 | success | TAG20210506T130518  | archivelog-backup   
    267c21daaea711eb9d3852540068a695 | 2021-05-06 20:11:07 | success | TAG20210506T131438  | create-backup-incremental
    650fd222aeaa11ebb58652540068a695 | 2021-05-06 20:34:12 | success | TAG20210506T133516  | archivelog-backup   
    961831e4aeae11ebb0d452540068a695 | 2021-05-06 21:04:11 | success | TAG20210506T140517  | archivelog-backup   
    c6919f28aeb211eb957e52540068a695 | 2021-05-06 21:34:12 | success | TAG20210506T143518  | archivelog-backup   
    f7ce0d0caeb611eb97c552540068a695 | 2021-05-06 22:04:12 | success | TAG20210506T150522  | archivelog-backup   
    286e8ea6aebb11eb864c52540068a695 | 2021-05-06 22:34:11 | success | TAG20210506T153516  | archivelog-backup   
    598f77eeaebf11eb92c052540068a695 | 2021-05-06 23:04:11 | success | TAG20210506T160518  | archivelog-backup   
    89f4919aaec311eb9a9452540068a695 | 2021-05-06 23:34:11 | success | TAG20210506T163516  | archivelog-backup   
    bb5ba95eaec711ebb1ed52540068a695 | 2021-05-07 00:04:11 | success | TAG20210506T170518  | archivelog-backup   
     
    

    Step #5 - On demand backups 

    Now that I have my database configured, I am going to demonstrate some of the options you can add to your backup.

    I am going to create a keep backup and give it a tag using bkup_start

    $ /var/opt/oracle/bkup_api/bkup_api bkup_start --dbname=dbsg2 --keep --tag=Maymonthlybackup
    DBaaS Backup API V1.5 @2021 Multi-Oracle home
    -> Action : bkup_start
    -> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_7d923417_20210507113940.052080.log
    UUID 958a58beaf6311eba98a52540068a695 for this backup
    ** process started with PID: 262102
    ** see log file for monitor progress
    -------------------------------------
    
    

    Now to list it.

    $ /var/opt/oracle/bkup_api/bkup_api list --dbname dbsg2 --keep
    DBaaS Backup API V1.5 @2021 Multi-Oracle home
    -> Action : list
    -> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_19714a18_20210507114254.007083.log
    -> Listing all backups
      Backup Tag                           Completion Date (UTC)      Type          keep    
    ----------------------                 -----------------------   -----------    --------
       Maymonthlybackup20210507T113125     05/07/2021 18:31:25       keep-forever   True
    
    

    Step #6 - Restore my database


    The last step I'm going to do in my database is to restore it to a previous point in time.

    Below is what you see in the console.
    NOTE - If you chose a specific time it will be in UTC time.


    I pick a time to restore to, and click on the 'Restore Database' option. I can follow the process by looking at 'Workload Requests'.




    Step #7 - Validating backups


    A great feature of the command tool is the ability to validate backups that have been taken.  This is easy to do with the 'bkup_api reval_start' command.

    I started my validate for my database dbbsg and I saved the uuid to monitor it.

    # /var/opt/oracle/bkup_api/bkup_api reval_start --dbname=dbbsg
    DBaaS Backup API V1.5 @2021 Multi-Oracle home
    -> Action : reval_start
    -> logfile: /var/opt/oracle/log/dbbsg/bkup_api_log/bkup_api_d0647aa8_20210511032638.300613.log
    UUID 5f204c4cb24311eb887252540068a695 for restore validation
    ** process started with PID: 15281
    ** Backup Request uuid     : 5f204c4cb24311eb887252540068a695
    
    

    Now to monitor it using the uuid until it's done, and I can see it completed successfully.

    # /var/opt/oracle/bkup_api/bkup_api --uuid=5f204c4cb24311eb887252540068a695 --dbname=dbbsg
    DBaaS Backup API V1.5 @2021 Multi-Oracle home
    @ STARTING CHECK STATUS 5f204c4cb24311eb887252540068a695
    [ REQUEST TICKET ]
    [UUID    ->  5f204c4cb24311eb887252540068a695 
    [DBNAME  ->  dbbsg 
    [STATE   ->  success 
    [ACTION  ->  start-restore-validate 
    [STARTED ->  2021-05-11 10:26:39 UTC 
    [ENDED   ->  2021-05-11 10:28:00 UTC 
    [PID     ->  15281 
    [TAG     ->  None 
    [PCT     ->  0.0 
    [LOG     ->  2021-05-11 03:26:39.780830 - API:: invoked with args : -dbname=dbbsg -reval=default  
    [LOG     ->  2021-05-11 03:26:42.324669 - API:: Wallet is in open AUTOLOGIN state 
    [LOG     ->  2021-05-11 03:26:42.996885 - API:: Oracle database state is up and running 
    [LOG     ->  2021-05-11 03:28:00.857565 - API:: ....... OK 
    [LOG     ->  2021-05-11 03:28:00.857645 - API:: Restore Validation is Completed 
    [ END TICKET ]
    
    

    Step #8 - Restoring/listing/backups with API

    There are many options to restoring with the API for both the "database" which consists of the CDB and all PDBs, or just a specific PDB.

    Below are some of the commands that help with this.
    NOTE: All commands are executed using "bkup_api" from /var/opt/oracle/bkup_api as "oracle"


    Command Options Description
    bkup_start   Start new special backup now
    bkup_start --keep Create keep backup
    bkup_start --level0 Perform a new FULL level 0 backup 
    bkup_start --level1 Perform a new level1 incremental backup
    bkup_start --cron Creates an incremntal backup through Cron
    bkup_chkcfg   Verifies that backups have been configured
    bkup_status   Shows the status of the most recent backup
    list   Shows the list of the most recent backups
    reval_start   Starts a restore validation of datafiles
    archreval_start   Starts a revalidation of archive logs
    recover_start --latest Recover from latest backup
    recover_start --scn Recover to SCN #
    recover_start --b Recover using a specific backup tag and defuzzy to archivelog following
    recover_start -t Recover to time. Specify --nonutc to use a non-UTC timestamp
    recover_status   Show status of most recent recover of this database


    With recovery you can also just recover a single PDB
    • --pdb={pdbname} - Recovery just a single PDB
    You can also specify if the config files should be restored
    • --cfgfiles - store the configuration files (controlfiles, spfiles etc) along with database files.

    Step #9 - Configuration changes

    You can execute the "bkup_api get config --dbname={dbname}" to create a file containing the  current configuration.  In that file you can see some of the other changes you can be.
    Below is what I see it using the version at the time of writing this.

    Config Parameter Settings Description
    bkup_cron_entry yes/no Enable/Disable automatic backups
    bkup_archlog_cron_entry yes/no Enable automatic archive log cleanup when not using tooling
    bkup_cfg_files yes/no Enable backup of Config files
    bkup_daily_time hh24:mi Time to execute daily backup
    bkup_archlog_frequency 15,20,30… How many minutes apart to execute archive log backups
    bkup_disk yes/no Backups to the FRA
    bkup_disk_recovery_window 1-14 Recover window of FRA
    bkup_oss_xxx   Backup settings when backing up to Object Store in Public Cloud
    bkup_zdlra_xx   Backup settings when backing up to a ZDLRA
    bkup_nfs_xxx   Backup settings when backing up to NFS
    bkup_set_section_size yes/no Set to yes to over ride the default setting
    bkup_section_size   Value for Over riding the default setting for section size
    bkup_channels_node xx Number of channels to be used by RMAN
    bkup_use_rcat yes/no If you are using an RMAN catalog
    bkup_rcat_xxx   RMAN catalog settings

    Step #10 - Scheduled backups


    Backups are scheduled in the crontab on the first node of a cluster. You can view schedule by executing "sudo su - " to become root, and look at the /etc/crontab file.
    Below is what is there for my database (dbsg2)

    # Example of job definition:
    # .---------------- minute (0 - 59)
    # |  .------------- hour (0 - 23)
    # |  |  .---------- day of month (1 - 31)
    # |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
    # |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
    # |  |  |  |  |
    # *  *  *  *  * user-name  command to be executed
    
    15 * * * * oracle /var/opt/oracle/misc/backup_db_wallets.pl
    15 * * * * oracle /var/opt/oracle/dbaascli/dbaascli tde backup --alldb
    19 1 * * * oracle /var/opt/oracle/bkup_api/bkup_api bkup_start --cron --dbname=dbsg2
    4,34 * * * * oracle /var/opt/oracle/bkup_api/bkup_api bkup_archlogs --cron --dbname=dbsg2
    
    
    The jobs that are scheduled to execute are.

    1. backup_db_wallets.pl - Every 15 minutes this script executes from the Crontab.  This script goes through the list of databases (regardless if database backups have been turned on) and it makes a copy of the SEPS wallet file in the current wallet location adding the current date/time. The old copy is removed and only one back exists.

    The following 2 settings are in my configuration file (/var/opt/oracle/creg/dbsg2.ini) are used as the source location of the wallet, and the location for the backup

     wallet_loc=/var/opt/oracle/dbaas_acfs/dbsg2/db_wallet
     wallet_loc_bak=/u02/app/oracle/admin/dbsg2/db_wallet

    NOTE: This wallet is used for storing user credentials and is an autologin wallet.
    I can see the credentials stored. In my case it is both the "sys" password, and the password for "rco". If I was using OSS (object store) my login credential would be stored in this wallet, and if I backed up to ZDLRA, this wallet would contain my connection to the ZDLRA(s) I was backing up to.
    2: CATALOG rco
    1: dbsg2 sys

    2. /var/opt/oracle/dbaascli/dbaascli tde backup --alldb - Every 15 minutes this script executes from the Crontab.  This script goes through the list of databases (regardless if database backups have been turned on) and it makes a copy of the TDE wallet file in the $ORACLE_BASE directory.
    The location is $ORACLE_BASE/{db_name}/tde_wallet/tde/

    The output from this script is in /var/opt/oracle/log/misc/backup

    3. /var/opt/oracle/bkup_api/bkup_api bkup_start --cron --dbname={mydb}- Every at 1:19 AM,  This time is determined in the bkup_api configuration using the parameter  "bkup_daily_time".  This is the same API that is called to perform an on-demand backup from the command line, but with a '--cron' parameter also.

    4. /var/opt/oracle/bkup_api/bkup_api bkup_archlogs --dbname={mydb} Every 30 minutes this script is executed based on the 'bkup_archlog_frequency' bkup_api configuration setting. This script will backup my archive logs to the backup location.

    6 comments:

    1. Hi,
      thank You for this great post, I'm on the same page, but have no idea how to restore such backup to totaly different host/exadata. Seems like api does not allow this.
      Any ideas maybe ?
      Regards
      Greg

      ReplyDelete
      Replies
      1. The way I look at the API, is that it can make things easy, but if you want you can bypass it. You can still restore and recover to a different (manually). If it is the same database, you can create a database and drop most of it. If it is a different database, than you need to use an auxiliary. Things will change over time however.

        Delete
    2. Thanks, very helpful. Just one thing: You mentioned "Every 15 minutes this script executes from the Crontab.", but the script executes once per hour only (15 mins past the full hour).
      Regards
      Clemens

      ReplyDelete
      Replies
      1. My understanding is that the script is executed every 15 minutes when using ExaCC, but executes every how when using DBCS (Base DB).

        Delete
    3. Hi Hello
      I think I need to correct it says: Every 15 minutes this script executes from the Crontab , It should be every hour at the 15th minute

      ReplyDelete