Pre-authenticated URLs in OCI are fast becoming one of my favorite features of using object storage. In this blog post I will explain how I am using them for both:
- Storing the current copy of my backup scripts and dynamically pulling it from my central repository
- uploading all my logs files to a central location
PROBLEM:
The problem I was trying to solve, is that I wanted to create a script to run on all my database nodes to create a weekly archival backup.
Since I have databases that are both Base DB databases, and ExaCS I found that I was continuously making changes to my backup script. Sometimes it was checking something different in my environment, and sometimes it was improving the error checking.
Each time I made a change, I was going out to every DB host and copying the new copy of my script.
Eventually I realized that Pre-authenticated URLs could not only help me ensure all my DB hosts are running the current copy of my backup script, they could be the central log location.
Solution:
Solution #1 - Script repository
The first problem I wanted to solve, was that I wanted to configure a script repository that I could dynamically pull the most current copy of my scripts from. Since I am running in OCI, I was looking for a "Cloud Native" solution rather than using NFS mounts that are shared across all my DB hosts.
To complicate things, I have databases that are running in different tenancies.
Step #1 - Store scripts in a bucket
The first step was to create a bucket in OCI to store both the scripts and logs. Within that bucket, under "More Actions" I chose "Create New Folder" and I created 2 new folders, "logs" and "scripts".
Then within the scripts folder I uploaded by current scripts
- rman.sh - My executable script that will set the environment and call RMAN
- backup.rman - My RMAN script that contains the RMAN command to backup my database.
Step #2 - Create a Pre-Authenticated Request
The next step was to create a Pre-Authenticated request on the "scripts" folder. Next to the scripts folder I clicked on the 3 dots and chose "Create Pre-Authenticated Request".
On the window that came up, I changed the expiration to be 1 year in the future (the default is 7 days). I chose the "Objects with prefix" box so that I could download any scripts that I put in this folder to the DB hosts. I also made sure the "Access Type" is "Permit object reads on those with specified prefix".
I did not chose "Enable Object Listing".
These settings will allow me to download the scripts from this bucket using the Pre-Authenticated URL only. From this URL you will not be able to list the objects, or upload any changes.
Step #3 - Create wrapper script to download current scripts
Then using the Pre-Authenticated URL in a wrapper script, I download the current copies of the scripts to the host and then executed my execution script (rman.sh) with a parameter.
Below you can see that I am using curl to download my script (rman.sh) and storing it my local script directory (/home/oracle/archive_backups/scripts). I am doing the same thing for the RMAN command file.
Once I download the current scripts, I am executing the shell script (rman.sh) .
curl -X GET https://{my tenancy}.objectstorage.us-ashburn-1.oci.customer-oci.com/p/{actual URL is removed }/n/id20skavsofo/b/bgrenn/o/scripts/rman.sh --output /home/oracle/archive_backups/scripts/rman.sh
curl -X GET https://{my tenancy}.objectstorage.us-ashburn-1.oci.customer-oci.com/p/{actual URL is removed }/n/id20skavsofo/b/bgrenn/o/scripts/backup.rman --output /home/oracle/archive_backups/scripts/backup.rman
/home/oracle/archive_backups/scripts/rman.sh $1
Solution #2 - Log repository
The second problem I wanted to solve was to make it easy review the execution of my scripts. I don't want to go to each DB host and look at the log file. I want to have the logs stored in a central location that I can check. Again Pre-Authenticated URLs to the rescue !
Step #1 - Create the Pre-Authenticated URL
In the previous steps I already create a "logs" folder within the bucket. In this step I want to create a Pre-Authenticated URL like I did for the scripts, but in this case I want to use it to store the logs.
Like before I chose "Create Pre-Authenticated Request" for the "logs" folder.
This time, I am choosing "Permit object writes to those with the specified prefix". This will allow me to write my log files to this folder in the bucket, but not list the logs, or download any logs.
Step #2 - Upload the log output from my script
The nice thing was once I implemented Solution #1, and had all of my DB nodes already downloading the current script. I updated the script to add an upload to object storage of the log file and they will all use my new script.
In my script I already had 2 variables set
- NOW - The current date in "yyyymmdd" format
- LOGFILE - The name of the output log file from my RMAN backup.
Now all I had to do was to add a curl command to upload my log file to the bucket.
Note I ma using the NOW variable to create a new folder under "logs" with the data so that my script executions are organized by date.
curl --request PUT --upload-file /home/oracle/archive_backups/logs/${LOGFILE} https://{My tenancy}.objectstorage.us-ashburn-1.oci.customer-oci.com/p/{URL removed}/n/id20skavsofo/b/bgrenn/o/logs/${NOW}/${LOGFILE}
BONUS
If I wanted to get fancy I could have put my LOGS in a new bucket, and configured a lifecycle management rule to automatically delete logs after a period of time from the bucket.
