ZFSSA replication can be used to create locked offsite backups. In this post I will show you how to take advantage of the new "Locked Snapshot" feature of ZFSSA and the ZFS Image in OCI to create an offsite backup strategy to OCI.
If you haven't heard of the locked snapshot feature of ZFSSA I blogged about here. In this post I am going to take advantage of this feature and show you how you can leverage it to provide a locked backup in the Oracle Cloud using the ZFS image available in OCI.
In order to demonstrate this I will start by following the documentation to create a ZFS image in OCI as my destination. Here is a great place to start with creating the virtual ZFS appliance in OCI.
Step 1 - Configure remote replication from source ZFSSA to ZFS appliance in OCI.
By enabling the "Remote Replication" service with a named destination, "downstream_zfs" in my example, I can now replicate to my ZFS appliance in OCI.
Step 2 - Ensure the source project/share has "Enable retention policy for Scheduled Snapshots" turned on
For my example I created a new project "Blogtest". On the "snapshots" tab I put a checkmark next to "Enable retention policy for Scheduled Snapshots". By checking this, the project will adhere to preventing the deletion of any locked snapshots. This property is replicated to the downstream and will cause the replicated project shares to also adhere to locking snapshots. This can also be set at the individual share level if you wish to control the configuration of locked snapshots for individual shares.
Below you can see where this is enabled for snapshots created within the project.
Step 3 - Create a snapshot schedule with "locked" snapshots
The next step is to create locked snapshots. This can be done at the project level (affecting all shares) or at the share level. In my example below I gave the scheduled snapshots a label "daily_snaps". Notice for my example I am only keeping only 1 snapshot and I am locking the snapshot at the source. In order for the snapshot to be locked at the destination
Retention Policy MUST be enabled for the share (or inherited from the project).
The source snapshot MUST be locked when it is created
Step 4 - Add replication to downstream ZFS in OCI
The next step is to add replication to the project configuration to replicate the shares to my ZFS in OCI. Below you can see the target is my "downstream_zfs" that I configured in the "Remote Replication" service.
You can also see that I am telling the replication to "include snapshots", which are my locked snapshots, and also to "Retain user snapshots on target". Under "Disaster Recovery" you can see that I am telling the downstream to keep a 30 day recovery point. Even though I am only keeping 1 locked snapshot on the source, I want to keep 30 days of recovery on the downstream in OCI.
Step 5 - Configure snapshots to replicate
In this step I am updating the replication action to replicate the locked scheduled snapshot to the downstream. Notice that I changed the number of snapshots from 1 (on the source) to 30 on the destination, and I am keeping the snapshot retention locked. This will ensure that the daily locked snapshot taken on the source will replicate to the destination as a locked snapshot, and 30 snapshots on the destination will remain locked. The 31st snapshot is no longer needed.
Step 6 - Configure the replication schedule
The last step is to configure the replication schedule. This ensures that on a daily basis the snapshots that are configured to be replicated will be replicated regularly to the downstream. You can make this more aggressive than daily if you wish the downstream to be more in sync in the primary. In my example below I configured the replication to occur every 10 minutes. This means that the downstream should have all updates as of 10 minutes ago or less. If I need to go back in time, I will have daily snapshots for the last 30 days that are locked and cannot be removed.
Step 7 - Validate the replication
Now that I have everything configured I am going to take a look at the replicated snapshots on my destination. I navigate to "shares" and I look under "replicat" and find my share. By clicking on the pencil and looking at the "snapshots" tab I can see my snapshot replicated over.
And when I click on the pencil next to the snapshot I can see that the snapshot is locked and I can't unlock it.
From there I can clone the snap and create a local snapshot, back it up to object storage, or reverse the replication if needed.
Migrating an Oracle database from on-premise to OCI is especially challenging when the database is quite large. In this blog post I will walk through the steps to migrate to OCI leveraging an on-disk local backup copied to object storage.
The basic steps to perform this task are on on the image above.
Step #1 - Upload backup pieces to object storage.
The first step to migrate my database (acmedb) is to copy the RMAN backup pieces to the OCI object storage using the OCI Client tool.
In order to make this easier, I am breaking this step into a few smaller steps.
Step #1A - Take a full backup to a separate location on disk
This can also be done by moving the backup pieces, or creating them with a different backup format. By creating the backup pieces in a separate directory, I am able to take advantage of the bulk upload feature of the OCI client tool. The alternative is to create an upload statement for each backup piece.
For my RMAN backup example (acmedb) I am going to change the location of the disk backup and perform a disk backup. I am also going to compress my backup using medium compression (this requires the ACO license). Compressing the backup sets allows me to make the backup pieces as small as possible when transferring to the OCI object store.
Below is the output from my RMAN configuration that I am using for the backup.
RMAN> show all;
RMAN configuration parameters for database with db_unique_name ACMEDBP are:
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE DEVICE TYPE DISK PARALLELISM 4 BACKUP TYPE TO COMPRESSED BACKUPSET;
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT '/acmedb/ocimigrate/backup_%d_%U';
CONFIGURE COMPRESSION ALGORITHM 'MEDIUM' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE;
I created a new level 0 backup including archive logs and below is the "list backup summary" output showing the backup pieces.
List of Backups
===============
Key TY LV S Device Type Completion Time #Pieces #Copies Compressed Tag
------- -- -- - ----------- --------------- ------- ------- ---------- ---
4125 B A A DISK 21-JUN-22 1 1 YES TAG20220621T141019
4151 B A A DISK 21-JUN-22 1 1 YES TAG20220621T141201
4167 B 0 A DISK 21-JUN-22 1 1 YES TAG20220621T141202
4168 B 0 A DISK 21-JUN-22 1 1 YES TAG20220621T141202
4169 B 0 A DISK 21-JUN-22 1 1 YES TAG20220621T141202
4170 B 0 A DISK 21-JUN-22 1 1 YES TAG20220621T141202
4171 B 0 A DISK 21-JUN-22 1 1 YES TAG20220621T141202
4172 B 0 A DISK 21-JUN-22 1 1 YES TAG20220621T141202
4173 B 0 A DISK 21-JUN-22 1 1 YES TAG20220621T141202
4174 B 0 A DISK 21-JUN-22 1 1 YES TAG20220621T141202
4175 B 0 A DISK 21-JUN-22 1 1 YES TAG20220621T141202
4176 B 0 A DISK 21-JUN-22 1 1 YES TAG20220621T141202
4208 B A A DISK 21-JUN-22 1 1 YES TAG20220621T141309
4220 B F A DISK 21-JUN-22 1 1 YES TAG20220621T141310
From the output you can see that there are a total of 14 backup pieces
3 Archive log backup sets (two created before the backup of datafiles, and one after).
TAG20220621T141019
TAG20220621T141201
TAG20220621T141309
10 Level 0 datafile backups
TAG20220621T141202
1 controlfile backup
TAG20220621T141310
Step #1B - Create the bucket in OCI and configure OCI Client
Now we need a bucket to upload the 14 RMAN backup pieces to.
Before I can upload the objects, I need to download and configure the OCI Client tool. You can find the instructions to do this here.
Once the client tool is installed I can create the bucket and verify that the OCI Client tool is configured correctly.
The command to create the bucket is.
Below is the output when I ran it for my compartment and created the bucket "acmedb_migrate"
Step #2 - Create the manifest for the backup pieces.
The next step covers creating the "metadata.xml" for each object which is the manifest the the RMAN library uses to read the backup pieces.
Again this is broken down into a few different steps.
Step #2A - Download an configure the Oracle Database Cloud Backup Module.
The link for the instructions (which includes the download can be found here.
I executed the jar file which downloads/created the following files.
libopc.so - This is the library used by the Cloud Backup module, and I downloaded it into "/home/oracle/ociconfig/lib/" on my host
acmedb.ora - This is the configuration file for my database backup. This was created in "/home/oracle/ociconfig/config/" on my host
This information is used to allocate the channel in RMAN for the manifest.
Step #2b - Generate the manifest create for each backup piece.
The next step is to dynamically create the script to build the manifest for each backup piece. This needs to be done for each backup piece, and the command is
The script I am using to complete this uses backup information from the controlfile of the database, and narrows the backup pieces to just the pieces in the directory I created for this backup.
Step #2c - Execute the script with an allocated channel.
The next step is to execute the script in RMAN within a run block after allocating a channel to the bucket in object storage. This needs to be done for each backup piece. You create a run block with one channel allocation followed by "send" commands.
NOTE: This does not have be executed on the host that generated the backups. In the example below, I set my ORACLE_SID to "dummy" and performed create manifest with the "dummy" instance started up nomount.
Below is an example of allocating a channel to the object storage and creating the manifest for one of the backup pieces.
export ORACLE_SID=dummy
rman target /
RMAN> startup nomount;
startup failed: ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/u01/app/oracle/product/19c/dbhome_1/dbs/initdummy.ora'
starting Oracle instance without parameter file for retrieval of spfile
Oracle instance started
Total System Global Area 1073737792 bytes
Fixed Size 8904768 bytes
Variable Size 276824064 bytes
Database Buffers 780140544 bytes
Redo Buffers 7868416 bytes
RMAN> run {
allocate channel t1 device type sbt parms='SBT_LIBRARY=/home/oracle/ociconfig/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/ociconfig/config/acmedb.ora)';
send channel t1 'export backuppiece backup_RADB_3r10k6ec_123_1_1';
}
2> 3> 4>
allocated channel: t1
channel t1: SID=19 device type=SBT_TAPE
channel t1: Oracle Database Backup Service Library VER=23.0.0.1
sent command to channel: t1
released channel: t1
Step #2d - Validate the manifest is created.
I logged into the OCI console, and I can see that there is a directory called "sbt_catalog". This is the directory containing the manifest files. Within this directory you will find a subdirectory for each backup piece. And within those subdirectories you will find a "metadata.xml" object containing the manifest.
Step #3 - Catalog the backup pieces.
The next step covers cataloging the backup pieces in OCI. You need to download the controlfile backup from OCI and start up mount the database.
Again this is broken down into a few different steps.
Step #3A - Download an configure the Oracle Database Cloud Backup Module.
The link for the instructions (which includes the download can be found here.
Again, you need to configure the backup module (or you can copy the files from your on-premise host).
Step #3b - Catalog each backup piece.
The next step is to dynamically create the script to build the catalog each backup piece. This needs to be done for each backup piece, and the command is
"catalog device type 'sbt_tape' backuppiece <object name>';
The script I am using to complete this uses backup information from the controlfile of the database, and narrows the backup pieces to just the pieces in the directory I created for this backup.
Step #3c - Execute the script with a configured channel.
I created a configure channel command, and cataloged the backup pieces that in the object store.
RMAN> CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' PARMS 'SBT_LIBRARY=/home/oracle/ociconfig/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/ociconfig/config/acmedb.ora)';
run {
catalog device type 'sbt_tape' backuppiece 'backup_RADB_3r10k6ec_123_1_1';
catalog device type 'sbt_tape' backuppiece 'backup_RADB_3s10k6hh_124_1_1';
catalog device type 'sbt_tape' backuppiece 'backup_RADB_3t10k6hj_125_1_1';
catalog device type 'sbt_tape' backuppiece 'backup_RADB_3u10k6hj_126_1_1';
catalog device type 'sbt_tape' backuppiece 'backup_RADB_3v10k6hj_127_1_1';
catalog device type 'sbt_tape' backuppiece 'backup_RADB_4010k6hj_128_1_1';
catalog device type 'sbt_tape' backuppiece ' backup_RADB_4110k6hk_129_1_1';
catalog device type 'sbt_tape' backuppiece 'backup_RADB_4210k6id_130_1_1';
catalog device type 'sbt_tape' backuppiece 'backup_RADB_4310k6ie_131_1_1';
catalog device type 'sbt_tape' backuppiece 'backup_RADB_4410k6ie_132_1_1';
catalog device type 'sbt_tape' backuppiece 'backup_RADB_4510k6jh_133_1_1';
}
old RMAN configuration parameters:
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' PARMS 'SBT_LIBRARY=/home/oracle/ociconfig/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/ociconfig/config/acmedb.ora)';
new RMAN configuration parameters:
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' PARMS 'SBT_LIBRARY=/home/oracle/ociconfig/lib/libopc.so ENV=(OPC_PFILE=/home/oracle/ociconfig/config/acmedb.ora)';
new RMAN configuration parameters are successfully stored
starting full resync of recovery catalog
full resync complete
RMAN>
RMAN> 2> 3> 4> 5> 6> 7> 8> 9> 10> 11> 12> 13>
allocated channel: ORA_SBT_TAPE_1
channel ORA_SBT_TAPE_1: SID=406 device type=SBT_TAPE
channel ORA_SBT_TAPE_1: Oracle Database Backup Service Library VER=23.0.0.1
allocated channel: ORA_SBT_TAPE_2
channel ORA_SBT_TAPE_2: SID=22 device type=SBT_TAPE
channel ORA_SBT_TAPE_2: Oracle Database Backup Service Library VER=23.0.0.1
allocated channel: ORA_SBT_TAPE_3
channel ORA_SBT_TAPE_3: SID=407 device type=SBT_TAPE
...
...
...
channel ORA_SBT_TAPE_4: SID=23 device type=SBT_TAPE
channel ORA_SBT_TAPE_4: Oracle Database Backup Service Library VER=23.0.0.1
channel ORA_SBT_TAPE_1: cataloged backup piece
backup piece handle=backup_RADB_4510k6jh_133_1_1 RECID=212 STAMP=1107964867
RMAN>
Step #3d - List the backups pieces cataloged
I performed a list backup summary to view the newly cataloged tape backup pieces.
RMAN> list backup summary;
List of Backups
===============
Key TY LV S Device Type Completion Time #Pieces #Copies Compressed Tag
------- -- -- - ----------- --------------- ------- ------- ---------- ---
4220 B F A DISK 21-JUN-22 1 1 YES TAG20220621T141310
4258 B A A SBT_TAPE 21-JUN-22 1 1 YES TAG20220621T141019
4270 B A A SBT_TAPE 21-JUN-22 1 1 YES TAG20220621T141201
4282 B 0 A SBT_TAPE 21-JUN-22 1 1 YES TAG20220621T141202
4292 B 0 A SBT_TAPE 21-JUN-22 1 1 YES TAG20220621T141202
4303 B 0 A SBT_TAPE 21-JUN-22 1 1 YES TAG20220621T141202
4315 B 0 A SBT_TAPE 21-JUN-22 1 1 YES TAG20220621T141202
4446 B 0 A SBT_TAPE 21-JUN-22 1 1 YES TAG20220621T141202
4468 B 0 A SBT_TAPE 21-JUN-22 1 1 YES TAG20220621T141202
4490 B 0 A SBT_TAPE 21-JUN-22 1 1 YES TAG20220621T141202
4514 B 0 A SBT_TAPE 21-JUN-22 1 1 YES TAG20220621T141202
4539 B 0 A SBT_TAPE 21-JUN-22 1 1 YES TAG20220621T141202
RMAN>
Step #4 - Restore the database.
The last step is restore the cataloged backup pieces. Remember you might have to change the location of the datafiles.
The process above can be used to upload and catalog both additional archive logs (to bring the files forward) and incremental backups to bring the database forward.
This post is going to go a little deeper on how to quickly download objects from the OCI object store onto your host.
In my example, I needed to download RMAN disk backup files that were copied to the Object Store in OCI.
I have over 10 TB of RMAN backup pieces, so I am going to create an ACFS mount point to store them on.
1) Create ACFS mount point
Creating the mount point is made up of multiple small steps that are documented here. This is a link to the 19c documentation so note it is subject to change over time.
Use ASMCMD to create a volume on the data disk
group of 20 TB
- Start ASMCMD connected to the Oracle ASM instance. You must be a user in the OSASM operating system group.
- Create the volume "volume1" on the "data" disk group
ASMCMD [+] > volcreate -G data -s 20G volume1
Use ASMCMD to list the volume information NOTE: my volume name is volume1-123
ASMCMD [+] > volinfo -G data volume1
Diskgroup Name: DATA
Volume Name: VOLUME1
Volume Device: /dev/asm/volume1-123
State: ENABLED
...
SQL> SELECT volume_name, volume_device FROM V$ASM_VOLUME
WHERE volume_name ='VOLUME1';
VOLUME_NAME VOLUME_DEVICE
----------------- --------------------------------------
VOLUME1 /dev/asm/volume1-123
Create the file system with mkfs from the volume "/dev/asm/volume1-123"
$ /sbin/mkfs -t acfs /dev/asm/volume1-123
mkfs.acfs: version = 19.0.0.0.0
mkfs.acfs: on-disk version = 46.0
mkfs.acfs: volume = /dev/asm/volume1-123
mkfs.acfs: volume size = 21474836480 ( 20.00 GB )
mkfs.acfs: Format complete.
The next step is to look at the objects I want to copy to my new ACFS file system. The format of accessing the object store in the commands is
"rclone {command} [connection name]:{bucket/partial object name - optional}.
NOTE: For all examples my connection name is oci_s3
I am going to start with the simplest command list buckets (lsd).
NOTE: We are using the s3 interface to view the objects in the namespace. There is a single namespace space for the entire tenancy. With OCI there is the concept of "compartments" which can be used to separate applications and users. The S3 interface does not have this concept, which means that all buckets are visible.
rclone lsd - This is the simplest command to list the buckets, and as I noted previously, it lists all buckets, not just my bucket.
If I want to list what is within my bucket (bsgbucket) I can list that bucket. In this case it treats the flat structure of the object name as if it is a file system, and lists only the top "directories" within my bucket.
3) Use rclone to copy the objects to my local file system.
There are 2 command you can use to copy the files from the object store to the local file system.
copy - This is as you expect. It copies the files to the local file system and overwrites the local copy
sync - This syncronizes the local file system with the objects in the object store, and will not copy down the object if it already has a local copy.
In my case I am going to use the sync command. This will allow me to re-start copying the objects and it will ignore any objects that were previously successfully copies.
Below is the command I am using to copy (synchronize) the objects from my bucket in the object store (oci_s3:bsgbucket) to the local filesystem (/home/opc/acfs).
-vv This option to rclone gives me "verbose" output so I can see more of what is being copied as the command is executed.
-P This option to rclone gives me feedback on how much of the object has downloaded so far to help me monitor it.
--multi-threaded-streams 12 This option to rclone breaks larger objects into chunks to increase the concurrency.
--transfers 64 This option to rclone allows for 64 concurrent transfers to occur. This increases the download throughput
oci-s3:bsgbucket - This is the source to copy/sync
/home/opc/acfs - this is the destination to copy/.sync with
Finally, this is the what the command looks like when it is executing.
opc@rlcone-test rclone]$ ./rclone -vv sync -P --multi-thread-streams 12 --transfers 64 oci_s3:bsgbucket /home/opc/acfs
2021/08/15 00:15:32 DEBUG : rclone: Version "v1.56.0" starting with parameters ["./rclone" "-vv" "sync" "-P" "--multi-thread-streams" "12" "--transfers" "64" "oci_s3:bsgbucket" "/home/opc/acfs"]
2021/08/15 00:15:32 DEBUG : Creating backend with remote "oci_s3:bsgbucket"
2021/08/15 00:15:32 DEBUG : Using config file from "/home/opc/.config/rclone/rclone.conf"
2021/08/15 00:15:32 DEBUG : Creating backend with remote "/home/opc/acfs"
2021-08-15 00:15:33 DEBUG : sbt_catalog/DTA_BACKUP_MYDB_4601d1ph_134_1_1/metadata.xml: md5 = 505fc1fdce141612c262c4181a9122fc OK
2021-08-15 00:15:33 INFO : sbt_catalog/DTA_BACKUP_MYDB_4601d1ph_134_1_1/metadata.xml: Copied (new)
2021-08-15 00:15:33 DEBUG : expdat.dmp: md5 = f97060f5cebcbcea3ad6fadbda136f4e OK
2021-08-15 00:15:33 INFO : expdat.dmp: Copied (new)
2021-08-15 00:15:33 DEBUG : Local file system at /home/opc/acfs: Waiting for checks to finish
2021-08-15 00:15:33 DEBUG : Local file system at /home/opc/acfs: Waiting for transfers to finish
2021-08-15 00:15:33 DEBUG : file_chunk/2985366474/MYDB/backuppiece/2021-06-14/DTA_BACKUP_MYDB_4601d1ph_134_1_1/yHqtjSE51L3B/0000000001: Starting multi-thread copy with 2 parts of size 160.875Mi
2021-08-15 00:15:33 DEBUG : file_chunk/2985366474/MYDB/backuppiece/2021-06-14/DTA_BACKUP_MYDB_4601d1ph_134_1_1/yHqtjSE51L3B/0000000001: multi-thread copy: stream 2/2 (168689664-337379328) size 160.875Mi starting
2021-08-15 00:15:33 DEBUG : file_chunk/2985366474/MYDB/backuppiece/2021-06-14/DTA_BACKUP_MYDB_4601d1ph_134_1_1/yHqtjSE51L3B/0000000001: multi-thread copy: stream 1/2 (0-168689664) size 160.875Mi starting
2021-08-15 00:15:33 DEBUG : file_chunk/2985366474/MYDB/backuppiece/2021-06-14/DTA_BACKUP_MYDB_4d01d1uq_141_1_1/lS9Sdnka2nD0/metadata.xml: md5 = 0a8eccc1410e1995e36fa2bfa0bf7a70 OK
2021-08-15 00:15:33 INFO : file_chunk/2985366474/MYDB/backuppiece/2021-06-14/DTA_BACKUP_MYDB_4d01d1uq_141_1_1/lS9Sdnka2nD0/metadata.xml: Copied (new)
2021-08-15 00:15:33 DEBUG : file_chunk/2985366474/MYDB/backuppiece/2021-06-14/DTA_BACKUP_MYDB_4601d1ph_134_1_1/yHqtjSE51L3B/metadata.xml: md5 = 505fc1fdce141612c262c4181a9122fc OK
2021-08-15 00:15:33 INFO : file_chunk/2985366474/MYDB/backuppiece/2021-06-14/DTA_BACKUP_MYDB_4601d1ph_134_1_1/yHqtjSE51L3B/metadata.xml: Copied (new)
2021-08-15 00:15:33 DEBUG : sbt_catalog/DTA_BACKUP_MYDB_4d01d1uq_141_1_1/metadata.xml: md5 = 0a8eccc1410e1995e36fa2bfa0bf7a70 OK
2021-08-15 00:15:33 INFO : sbt_catalog/DTA_BACKUP_MYDB_4d01d1uq_141_1_1/metadata.xml: Copied (new)
2021-08-15 00:15:33 INFO : file_chunk/2985366474/MYDB/backuppiece/2021-06-14/DTA_BACKUP_MYDB_4d01d1uq_141_1_1/lS9Sdnka2nD0/0000000001: Copied (new)
2021-08-15 00:15:34 DEBUG : file_chunk/2985366474/MYDB/backuppiece/2021-06-14/DTA_BACKUP_MYDB_4601d1ph_134_1_1/yHqtjSE51L3B/0000000001: multi-thread copy: stream 1/2 (0-168689664) size 160.875Mi finished
Transferred: 333.398Mi / 356.554 MiByte, 94%, 194.424 MiByte/s, ETA 0s
Transferred: 6 / 7, 86%
Elapsed time: 2.0s
Transferring:
NOTE: it broke up the larger object into chunks, and you can see that it downloaded 2 chunks simultaneously. At the end you can see the file that it was in the middle of transferring.
Conclusion.
rclone is great alternative to the OCI CLI to manage your objects and download them. It has more intuitive commands (like "rclone ls"). And the best part is that it doesn't require python and special privleges to install.
The next step is to configure my database to be backed up using the tooling. This is pretty straightforward. I click on the "edit backup" button and fill in the information for my database and save it. In my case I am using ZFS, and the need to make sure that I change my container to the container where the ZFS is configured.
NOTE : The backup strategy is a Weekly L0 (full) backup every Sunday, and a daily L1 (differential incremental backup) on all other days. The time the backup is scheduled can be found in either the backup settings, or by looking at the crontab file.
Then I just wait until I see complete. If I click on the work requests, I can see the progress until it's finished.
Step #2 - Update the settings to use my RMAN catalog.
First I need to get what the current settings are for my database (dbsg2) and save them in a config file so I can update them.
I log into the first node, and su to root.
Once there I execute "get config --all" and save all the settings to a file that I can update.
NOTE : I an creating a new file under the bakup_api/cfg directory to make it easy to find.
$ sudo su -
Last login: Thu May 6 11:43:46 PDT 2021 on pts/0
[root@ecc ~]## /var/opt/oracle/bkup_api/bkup_api get config --all --file=/var/opt/oracle/bkup_api/cfg/dbsg2.cfg --dbname dbsg2
DBaaS Backup API V1.5 @2021 Multi-Oracle home
-> Action : get_config
-> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_92303612_20210506125612.006275.log
File /var/opt/oracle/bkup_api/cfg/dbsg2.cfg created
Now I am going to edit it and make some changes.
I changed to RMAN catalog settings to use my catalog.
NOTE: The entry has to be the connect string, not a tnsnames.ora entry.
#### This section is applicable when using a rman catalog ####
# Enables RMAN catalog. Can be set to yes or no.
bkup_use_rcat=yes
## Below parameters are required if rman catalog is enabled
# RMAN catalog user
bkup_rcat_user=rco
# RMAN catalog password
#bkup_rcat_passwd=RMan19c#_
# RMAN catalog conn string
bkup_rcat_conn=ecc-scan.bgrenn.com:1521:rmanpdb.bgrenn.com
Now I am going to commit (set) the changes using the "set config" command
# /var/opt/oracle/bkup_api/bkup_api set config --file=/var/opt/oracle/bkup_api/cfg/dbsg2.cfg --dbname dbsg2
DBaaS Backup API V1.5 @2021 Multi-Oracle home
-> Action : set_config
-> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_b800281f_20210506130824.084259.log
cfgfile : /var/opt/oracle/bkup_api/cfg/dbsg2.cfg
Using configuration file: /var/opt/oracle/bkup_api/cfg/dbsg2.cfg
API::Parameters validated.
UUID d0845ea0aea611eb98fb52540068a695 for this set_config(configure-backup)
** process started with PID: 86143
** see log file for monitor progress
-------------------------------------
And after a few minutes, I am going to check and make sure it was successful by using the configure_status command
/var/opt/oracle/bkup_api/bkup_api configure_status --dbname dbsg2
DBaaS Backup API V1.5 @2021 Multi-Oracle home
-> Action : configure_status
-> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_fa81558e_20210507060019.504831.log
* Last registered operation: 2021-05-07 12:58:41 UTC
* Configure backup status: finished
**************************************************
* API History: API steps
API:: NEW PROCESS 120531
*
* RETURN CODE:0
##################################################
Everything looks good ! It removed my configuration file (which is good because it had the password in it).
I found that 2 things happened as part of adding an RMAN catalog
The password for the RMAN catalog user is now stored in the wallet file.
There is an entry in my tnsnames file on all nodes for "CATALOG" which points to the rman catalog.
NOTE: When part of this process is to register the database with the RMAN catalog. You do not have to manually register the database in the catalog.
Step #3 - Take a manual backup
Now logged in as OPC, and becoming Root, and can run a special backup using bkup_api
# /var/opt/oracle/bkup_api/bkup_api bkup_start --dbname=dbsg2
DBaaS Backup API V1.5 @2021 Multi-Oracle home
-> Action : bkup_start
-> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_9458c30f_20210510084341.430481.log
UUID 7f6622f8b1a611eb865552540068a695 for this backup
** process started with PID: 336757
** see log file for monitor progress
-------------------------------------
I can see the status while it's running
/var/opt/oracle/bkup_api/bkup_api bkup_status --dbname=dbsg2
DBaaS Backup API V1.5 @2021 Multi-Oracle home
-> Action : bkup_status
-> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_46545e6f_20210510084812.014419.log
(' Warning: unable to get current configuration of:', 'catalog')
* Current backup settings:
* Last registered Bkup: 05-10 15:44 UTC API::336757:: Starting dbaas backup process
* Bkup state: running
**************************************************
* API History: API steps
API:: NEW PROCESS 336757
API:: Starting dbaas backup process
*
* RETURN CODE:0
##################################################
And I waited a few minutes, and I can see it was successful.
# /var/opt/oracle/bkup_api/bkup_api bkup_status --dbname=dbsg2
DBaaS Backup API V1.5 @2021 Multi-Oracle home
-> Action : bkup_status
-> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_8acd03e3_20210510085129.207757.log
(' Warning: unable to get current configuration of:', 'catalog')
* Current backup settings:
* Last registered Bkup: 05-10 15:44 UTC API::336757:: Starting dbaas backup process
* Bkup state: running
**************************************************
* API History: API steps
API:: NEW PROCESS 336757
API:: Starting dbaas backup process
*************************************************
* Backup steps
-> 2021-05-10 08:44:20.651787 - API:: invoked with args : -dbname=dbsg2 -uuid=7f6622f8b1a611eb865552540068a695 -level1
-> 2021-05-10 08:44:23.458698 - API:: Wallet is in open AUTOLOGIN state
-> 2021-05-10 08:44:24.204793 - API:: Oracle database state is up and running
-> 2021-05-10 08:44:25.686134 - API:: CATALOG SETTINGS
-> 2021-05-10 08:45:19.767284 - API:: DB instance: dbsg2
-> 2021-05-10 08:45:19.767424 - API:: Validating the backup repository ......
-> 2021-05-10 08:46:38.263401 - API:: All backup pieces are ok
-> 2021-05-10 08:46:38.263584 - API:: Validating the TDE wallet ......
-> 2021-05-10 08:46:41.842706 - API:: TDE check successful.
-> 2021-05-10 08:46:42.446560 - API:: Performing incremental backup to shared storage
-> 2021-05-10 08:46:42.448228 - API:: Executing rman instructions
-> 2021-05-10 08:49:21.161884 - API:: ....... OK
-> 2021-05-10 08:49:21.162089 - API:: Incremental backup to shared storage is Completed
-> 2021-05-10 08:49:21.163822 - API:: Starting backup of config files
-> 2021-05-10 08:49:21.699197 - API:: Determining the oracle database id
-> 2021-05-10 08:49:21.726308 - API:: DBID: 2005517379
-> 2021-05-10 08:49:22.040891 - API:: Creating directories to store config files
-> 2021-05-10 08:49:22.085476 - API:: Enabling RAC exclusions for config files.
-> 2021-05-10 08:49:22.114211 - API:: Compressing config files into tar files
-> 2021-05-10 08:49:22.173842 - API:: Uploading config files to NFS location
-> 2021-05-10 08:49:22.222493 - API:: Removing temporary location /var/opt/oracle/log/dbsg2/obkup/7f6622f8b1a611eb865552540068a695.
-> 2021-05-10 08:49:22.224071 - API:: Config files backup ended successfully
-> 2021-05-10 08:49:26.052494 - API:: All requested tasks are completed
*
* RETURN CODE:0
##################################################
Step #4 - Check my periodic backups
Now it's been a few days (I started on Thursday and it's now Monday).
I am going to check on the incremental backups, and the archive log backups.
There are 2 ways I can do this.
Using the bkup_api command to list the backups that have run.
# /var/opt/oracle/bkup_api/bkup_api list --dbname=dbsg2
DBaaS Backup API V1.5 @2021 Multi-Oracle home
-> Action : list
-> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_eddcd4e1_20210510064145.497707.log
-> Listing all backups
Backup Tag Completion Date (UTC) Type keep
---------------------- ----------------------- ----------- --------
TAG20210506T123203 05/06/2021 19:32:03 full False
TAG20210506T131438 05/06/2021 20:14:38 incremental False
TAG20210507T012240 05/07/2021 08:22:40 incremental False
TAG20210508T012315 05/08/2021 08:23:15 incremental False
TAG20210509T012438 05/09/2021 08:24:38 full False
TAG20210510T012322 05/10/2021 08:23:22 incremental False
Using the RMAN catalog
Backup Type Encrypted Tag Backup Piece Backup Time Day Of Week
-------------------- --------- --------------------------------- ------------------------------------------------------------ -------------------- --------------------
Full L0 YES DBAAS_FULL_BACKUP20210506122626 /backup/dbaas_bkup_DBSG2_2005517379_0dvu5rp2_13_1 05/06/21 12:29:32 THURSDAY
Differential L1 YES DBAAS_INCR_BACKUP20210506131110 /backup/dbaas_bkup_DBSG2_2005517379_2avu5ud1_74_1 05/06/21 13:14:18 THURSDAY
Differential L1 YES DBAAS_INCR_BACKUP20210507011926 /backup/dbaas_bkup_DBSG2_2005517379_72vu792b_226_1 05/07/21 01:22:27 FRIDAY
Differential L1 YES DBAAS_INCR_BACKUP20210508011939 /backup/dbaas_bkup_DBSG2_2005517379_lbvu9tf3_683_1 05/08/21 01:22:51 SATURDAY
Full L0 YES DBAAS_FULL_BACKUP20210509011940 /backup/dbaas_bkup_DBSG2_2005517379_u3vuchr8_963_1 05/09/21 01:22:59 SUNDAY
Differential L1 YES DBAAS_INCR_BACKUP20210510011940 /backup/dbaas_bkup_DBSG2_2005517379_6rvuf672_1243_1 05/10/21 01:22:49 MONDAY
NOTE: I can see that a periodic L1 (differential) is executed at 1:22 AM, every day but Sunday where a Full backup is executed.
Now to look at archive log backups -- I am going to show a subset.
Again I can use the bkup_api "list_jobs" command and see all the backup jobs that have been run (which include archive logs).
Now that I have my database configured, I am going to demonstrate some of the options you can add to your backup.
I am going to create a keep backup and give it a tag using bkup_start
$ /var/opt/oracle/bkup_api/bkup_api bkup_start --dbname=dbsg2 --keep --tag=Maymonthlybackup
DBaaS Backup API V1.5 @2021 Multi-Oracle home
-> Action : bkup_start
-> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_7d923417_20210507113940.052080.log
UUID 958a58beaf6311eba98a52540068a695 for this backup
** process started with PID: 262102
** see log file for monitor progress
-------------------------------------
Now to list it.
$ /var/opt/oracle/bkup_api/bkup_api list --dbname dbsg2 --keep
DBaaS Backup API V1.5 @2021 Multi-Oracle home
-> Action : list
-> logfile: /var/opt/oracle/log/dbsg2/bkup_api_log/bkup_api_19714a18_20210507114254.007083.log
-> Listing all backups
Backup Tag Completion Date (UTC) Type keep
---------------------- ----------------------- ----------- --------
Maymonthlybackup20210507T113125 05/07/2021 18:31:25 keep-forever True
Step #6 - Restore my database
The last step I'm going to do in my database is to restore it to a previous point in time.
Below is what you see in the console.
NOTE - If you chose a specific time it will be in UTC time.
I pick a time to restore to, and click on the 'Restore Database' option. I can follow the process by looking at 'Workload Requests'.
Step #7 - Validating backups
A great feature of the command tool is the ability to validate backups that have been taken. This is easy to do with the 'bkup_api reval_start' command.
I started my validate for my database dbbsg and I saved the uuid to monitor it.
# /var/opt/oracle/bkup_api/bkup_api reval_start --dbname=dbbsg
DBaaS Backup API V1.5 @2021 Multi-Oracle home
-> Action : reval_start
-> logfile: /var/opt/oracle/log/dbbsg/bkup_api_log/bkup_api_d0647aa8_20210511032638.300613.log
UUID 5f204c4cb24311eb887252540068a695 for restore validation
** process started with PID: 15281
** Backup Request uuid : 5f204c4cb24311eb887252540068a695
Now to monitor it using the uuid until it's done, and I can see it completed successfully.
# /var/opt/oracle/bkup_api/bkup_api --uuid=5f204c4cb24311eb887252540068a695 --dbname=dbbsg
DBaaS Backup API V1.5 @2021 Multi-Oracle home
@ STARTING CHECK STATUS 5f204c4cb24311eb887252540068a695
[ REQUEST TICKET ]
[UUID -> 5f204c4cb24311eb887252540068a695
[DBNAME -> dbbsg
[STATE -> success
[ACTION -> start-restore-validate
[STARTED -> 2021-05-11 10:26:39 UTC
[ENDED -> 2021-05-11 10:28:00 UTC
[PID -> 15281
[TAG -> None
[PCT -> 0.0
[LOG -> 2021-05-11 03:26:39.780830 - API:: invoked with args : -dbname=dbbsg -reval=default
[LOG -> 2021-05-11 03:26:42.324669 - API:: Wallet is in open AUTOLOGIN state
[LOG -> 2021-05-11 03:26:42.996885 - API:: Oracle database state is up and running
[LOG -> 2021-05-11 03:28:00.857565 - API:: ....... OK
[LOG -> 2021-05-11 03:28:00.857645 - API:: Restore Validation is Completed
[ END TICKET ]
Step #8 - Restoring/listing/backups with API
There are many options to restoring with the API for both the "database" which consists of the CDB and all PDBs, or just a specific PDB.
Below are some of the commands that help with this.
NOTE: All commands are executed using "bkup_api" from /var/opt/oracle/bkup_api as "oracle"
Command
Options
Description
bkup_start
Start new special backup now
bkup_start
--keep
Create keep backup
bkup_start
--level0
Perform a new FULL level 0 backup
bkup_start
--level1
Perform a new level1 incremental backup
bkup_start
--cron
Creates an incremntal backup through Cron
bkup_chkcfg
Verifies that backups have been configured
bkup_status
Shows the status of the most recent backup
list
Shows the list of the most recent backups
reval_start
Starts a restore validation of datafiles
archreval_start
Starts a revalidation of archive logs
recover_start
--latest
Recover from latest backup
recover_start
--scn
Recover to SCN #
recover_start
--b
Recover using a specific backup tag and defuzzy to archivelog following
recover_start
-t
Recover to time. Specify --nonutc to use a non-UTC timestamp
recover_status
Show status of most recent recover of this database
With recovery you can also just recover a single PDB
--pdb={pdbname} - Recovery just a single PDB
You can also specify if the config files should be restored
--cfgfiles - store the configuration files (controlfiles, spfiles etc) along with database files.
Step #9 - Configuration changes
You can execute the "bkup_api get config --dbname={dbname}" to create a file containing the current configuration. In that file you can see some of the other changes you can be.
Below is what I see it using the version at the time of writing this.
Config Parameter
Settings
Description
bkup_cron_entry
yes/no
Enable/Disable automatic backups
bkup_archlog_cron_entry
yes/no
Enable automatic archive log cleanup when not using tooling
bkup_cfg_files
yes/no
Enable backup of Config files
bkup_daily_time
hh24:mi
Time to execute daily backup
bkup_archlog_frequency
15,20,30…
How many minutes apart to execute archive log backups
bkup_disk
yes/no
Backups to the FRA
bkup_disk_recovery_window
1-14
Recover window of FRA
bkup_oss_xxx
Backup settings when backing up to Object Store in Public Cloud
bkup_zdlra_xx
Backup settings when backing up to a ZDLRA
bkup_nfs_xxx
Backup settings when backing up to NFS
bkup_set_section_size
yes/no
Set to yes to over ride the default setting
bkup_section_size
Value for Over riding the default setting for section size
bkup_channels_node
xx
Number of channels to be used by RMAN
bkup_use_rcat
yes/no
If you are using an RMAN catalog
bkup_rcat_xxx
RMAN catalog settings
Step #10 - Scheduled backups
Backups are scheduled in the crontab on the first node of a cluster. You can view schedule by executing "sudo su - " to become root, and look at the /etc/crontab file.
1. backup_db_wallets.pl- Every 15 minutes this script executes from the Crontab. This script goes through the list of databases (regardless if database backups have been turned on) and it makes a copy of the SEPS wallet file in the current wallet location adding the current date/time. The old copy is removed and only one back exists.
The following 2 settings are in my configuration file (/var/opt/oracle/creg/dbsg2.ini) are used as the source location of the wallet, and the location for the backup
NOTE: This wallet is used for storing user credentials and is an autologin wallet.
I can see the credentials stored. In my case it is both the "sys" password, and the password for "rco". If I was using OSS (object store) my login credential would be stored in this wallet, and if I backed up to ZDLRA, this wallet would contain my connection to the ZDLRA(s) I was backing up to.
2: CATALOG rco
1: dbsg2 sys
2. /var/opt/oracle/dbaascli/dbaascli tde backup --alldb- Every 15 minutes this script executes from the Crontab. This script goes through the list of databases (regardless if database backups have been turned on) and it makes a copy of the TDE wallet file in the $ORACLE_BASE directory.
The location is $ORACLE_BASE/{db_name}/tde_wallet/tde/
The output from this script is in /var/opt/oracle/log/misc/backup
3. /var/opt/oracle/bkup_api/bkup_api bkup_start --cron --dbname={mydb}- Every at 1:19 AM, This time is determined in the bkup_api configuration using the parameter "bkup_daily_time". This is the same API that is called to perform an on-demand backup from the command line, but with a '--cron' parameter also.
4. /var/opt/oracle/bkup_api/bkup_api bkup_archlogs --dbname={mydb} Every 30 minutes this script is executed based on the 'bkup_archlog_frequency' bkup_api configuration setting. This script will backup my archive logs to the backup location.
