ZFSSA can be used to share data from your Oracle Database

 Data Sharing has become a big topic recently, and Oracle Cloud has added some new services to allow you to share data from an Autonomous Database.  But how do you do this with your on-premises database ? In this post I show you how to use ZFS as your data sharing platform.

Data Sharing

Being able to securely share data between applications is a critical feature in todays world.  The Oracle Database is often used to consolidate and summarize collected data, but is not always the platform for doing analysis.  The Oracle Database does have the capability to analyze data, but tools such as Jupyter Notebooks, Tableu, Power Bi, etc are typically the favorites of Data Scientists and data analysts.

The challenge is how to give access to specific pieces of data in the database without providing access to the database itself.  The most common solution is to use object storage and pre-authenticated URLs.  Data is extracted from the database based on the user and stored in object storage in a sharable format (JSON for example).  With this paradigm, and pictured above, you can create multiple datasets that contain a subset of the data specific to the users needs and authorization.  The second part is the use of a pre-authenticated URL.  This is a dynamically created URL that allows access to the object without authentication. Because it contains a long string of random characters, and is only valid for a specified amount of time, it can be securely shared with the user.

My Environment

For this post, I started with an environment I had previously configured to use DBMS_CLOUD.  My database is a 19.20 database.  In that database I used the steps specified in the MOS note and my blog (information can be found here) to use DBMS_CLOUD.

My ZFSSA environment is using 8.8.63, and I did all of my testing in OCI using compute instances.

For preparation to test I had

• Installed DBMS_CLOUD packages into my database using MOS note #2748362.1
• Downloaded the certificate for my ZFS appliance using my blog post and added them to wallet.
• Added the DNS/IP addresses to the DBMS_CLOUD_STORE table in the CDB.
• Created a user in my PDB with authority to use DBMS_CLOUD
• Created a user on ZFS to use for my object storage authentication (Oracle).
• Configured the HTTP service for OCI
• Added my public RSA key from my key par to the OCI service for authentication.
• Created a bucket

NOTE:  In order to complete the full test, there were 2 other items I needed to do.

1) Update the ACL to also access port 80.  The DBMS_CLOUD configuration sets ACLs to access websites using port 443.  During my testing I used port 80 (http vs https).

2) I granted execute on DBMS_CRYPTO to my database user for the testing.

Step #1 Create Object

The first step was to create an object from a query in the database.  This simulated pulling a subset of data (based on the user) and writing it to a object so that it could be shared.  To create the object I used the DBMS_CLOUD.EXPORT_DATA package.  Below is the statement I executed.

BEGIN
 DBMS_CLOUD.EXPORT_DATA(
       credential_name => 'ZFS_OCI2',  
    file_uri_list =>'https://zfs-s3.zfsadmin.vcn.oraclevcn.com/n/zfs_oci/b/db_back/o/shareddata/customer_sales.json',
    format => '{"type" : "JSON" }',
   query => 'SELECT OBJECT_NAME,OBJECT_TYPE,STATUS FROM user_objects');
END;
/

In this example:

• CREDENTIAL_NAME - Refers to my authentication credentials I had previously created in my database.
• FILE_URI_LIST - The name and location of the object I want to create on the ZFS object storage.
• FORMAT - The output is written in JSON format
• QUERY - This is the query you want to execute and store the results in the object storage.  

As you can see, it would be easy to create multiple objects that contain specific data by customizing the query, and naming the object appropriately.

In order to get the  proper name of the object I then selected the list of objects from object storage.

set pagesize 0
SET UNDERLINE =
col object_name format  a25
col created format  a20
select object_name,to_char(created,'MM/DD/YY hh24:mi:ss') created,bytes/1024  bytes_KB 
       from dbms_cloud.list_objects('ZFS_OCI2', 'https://zfs-s3.zfsadmin.vcn.oraclevcn.com/n/zfs_oci/b/db_back/o/shareddata/');


 customer_sales.json_1_1_1.json 12/19/23 01:19:51    3.17382813

From the output, I can see that my JSON file was named 'customer_sales.json_1_1_1.json'.

Step #2 Create Pre-authenticated URL

The Package I ran to do this is below. I am going to break down the pieces into multiple sections. Below is the full code.

DECLARE tenancy_ocid VARCHAR (100) ; user_ocid VARCHAR (100) ; fingerprint VARCHAR (100) ; rest_api VARCHAR (100) ; host VARCHAR (100) ; body VARCHAR (2000) ; URL varchar (20); Bucket varchar (20); compartment varchar (20); object_name varchar (200); content_sha256 VARCHAR (2000) ; content_sha256_header VARCHAR (2000) ; content_length VARCHAR (2000) ; content_length_header VARCHAR (200) ; headers VARCHAR (200) ; content_type_header VARCHAR (200) ; vardate VARCHAR (200) ; date_header VARCHAR (200) ; host_header VARCHAR (200) ; request_target VARCHAR (200) ; signing_string VARCHAR (2000) ; ip_str VARCHAR (2000) ; post_request VARCHAR (2000) ; pubkey VARCHAR (2000) ; prvkey VARCHAR (2000) ; sign_raw RAW (2000); signature varchar (2000); signature_final varchar (2000); authorization_header varchar (2000); position64 integer :=1; returnval BOOLEAN := false; sType PLS_INTEGER := DBMS_CRYPTO.SIGN_SHA256_RSA; kType PLS_INTEGER := DBMS_CRYPTO.KEY_TYPE_RSA; req utl_http.req; res utl_http.resp; buffer varchar2(4000); json_obj json_object_t; url_string varchar2(400); BEGIN /* ########################################################################################## # # # Set Variables # # # ########################################################################################## */ /* # The tenancy. */ tenancy_ocid := 'ocid1.tenancy.oc1..nobody'; user_ocid := 'ocid1.user.oc1..oracle'; /* # The fingerprint. */ fingerprint := '6d:f9:57:d5:ff:b1:c0:98:81:90:1e:6e:08:0f:d0:69'; /* # The URL */ URL := '/oci'; /* # The bucket */ bucket := 'db_back'; /* # The compartment/share on ZFS */ compartment := 'zfs_oci'; /* # The REST api you want to call, with any required paramters. */ rest_api := URL || '/n/' || compartment || '/b/' || bucket || '/p?compartmentId=' || compartment ; /* # The host you want to make the call against */ host := 'zfs-s3.zfsadmin.vcn.oraclevcn.com'; /* # Get the private key to sign the request */ prvkey := 'MIIEowIBAAKCAQEAzdfuSSwga9ziJTQK6RKB86VkBRnsIUrNGKQVKAkGaG5V+NjjJXvZt3a7p70NykHD4ZKO5ShbTBxYMCHc6uIjAJe/5moGog267K4q1Wa1WJIcL0/H5t/MWKMjToQ+qIz2K21zsOCAgf9YBY3XoQsK3RvPcCO2zbp0g0PZ5cfOUxNGQP7o2aPltymx6OrzMMXWfr7vgnMWeif86AAgdh4yfyQmNzjZFjwz1RDwYQYA50iLGSEVf0UIxDqxe8qCIMzPe5J8Drhw10NKB95ZSj++GKZ6wpv1ScClzuxS2kZYHqWfIYpAZal/Xmpr5o6GVyMyeAYxZ3TnCsXH7PhrnIbb9QIDAQABAoIBADKIlTYfgCIayuXpPdh0M7T14HUkJQUTUQCAn7UBajJPDAoYbMMyAYP5wR3v2qfRt130VeJKIwevWbMPTSXYuFEX8AOH+fLkvgQGyfszyMdir0qqImA6hvcnra5D5wIIARMuPp2or8O6RQiwBsTBajxX7HgSPHmLEbl7+c1gXPWlr4f1Qu7Fzas3plNjdkTRb+Dewr6bE1k/EVhEGm/LIHZn6bObx2pUZ2RHSVWuyZjqgLN7mcmC0z9tMumvdAVbi6SDNsp6wfk21kkywo5eKe2nYjqm4ezWCYX7ZLsjJg0jIzjO0Y7u1x/8lyvsgVuwU74SY84hOgJYEQy7MG9HZ+ECgYEA+1Qm6XT1H/YX9O+bZ3eEZH6AgTAq9PIgqWV8+1InvH4u6HT5OMccxc2RsxxCUXarv3jGJAme43jLaCeP/CkjBhcVpRQT1PLv4k8Sf4ABO3GLEP2qdM7sk/dJbFbDplO1sS0ZZh8AKcntLzN99trvZ+E9BvJXmBroyjstoJSUg20CgYEA0ata94Q9SOFleMK82Oflm+gs7zbuyebSmh6lBn8gal0szW5nOsKce48ht71nRFQU4rLCkglcjxH8s8+BSVDaoo05LeR8SDKpPV1qdd0itVFe+agHyOGuTlZnjh3u+B3xyRKn9tdSiUNopSZ1wMqy3p1/k8BgFjAAUvu6o+nD3akCgYATZaFDL1Wjy9h5fOyfhafgIatFTTRjhzRiHoUQ2Hxcndh89FovuZ5M3dq62hl8NA55TpbsPRnYMIByX2tLaNldJPb0sKH/M2i0LKoe0QBHWDIFc65uKXI00UcHZ4LeGWPScsqdCRN4Ji96Dx1PqgIEIGwXVyjPgnQo93xEAnOb2QKBgDjSaJ/FzdoPg7F7jTR3fEmJXYwi5gn/rTmpbm3ZTA6nW7n83qsVDNWLj8a85NymsNZTPdnYAaagaEzOwqn86JO/FCz0zB3A5Q7FX1G78WkUOVoNiLaPOu7uCbiBMj851kn47V4/MpYSl6ySRL1P8bPVGTsQmrnInn3ezNEZTESpAoGBAPRTl9d3l8LtlVD10qDyunMyvfeLXvBMSq4Nje7EfTT5wGwRUIFabvM/KSM0NMW1eaEKO2PxeTpcvVarJykbLh3caSra/ptJ0/Qm2F595iIuwNeAt3pMkpr+gVtp21re+vFw6JUw/3SUirXvPWPUogN+6h0/wlIcWQr7CCerY452'; /* # Set the current date/time to the correct format */ select to_char (systimestamp at time zone 'GMT','Dy, dd Mon yyyy hh24:mi:ss') || ' GMT' into vardate from dual; /* ########################################################################################## # # # Set rest API URL call # # # ########################################################################################## */ /* # The REST api you want to call, with any required paramters. */ rest_api := URL || '/n/' || compartment || '/b/' || bucket || '/p?compartmentId=' || compartment ; /* ########################################################################################## # # # Set The Post Request # # # ########################################################################################## */ post_request := 'POST http://' || host || rest_api || ' Date: ' || vardate; /* ########################################################################################## # # # Set The body as JSON # # # ########################################################################################## */ /* # the json file containing the data you want to POST to the rest endpoint */ object_name := 'shareddata/customer_sales.json_1_1_1.json'; body := '{' || chr(10) || ' "accessType": "ObjectRead",' || chr(10) || ' "bucketListingAction": "Deny",' || chr(10) || ' "bucketName": "' || bucket || '",' || chr(10) || ' "name": "mytestforjsonx1",' || chr(10) || ' "namespace": "' || compartment || '",' || chr(10) || ' "namespaceName": "' || compartment || '",' || chr(10) || ' "objectName": "' || object_name || '",' || chr(10) || ' "timeExpires": "2024-11-01T00:00:00+00:00"' || chr(10) || '}'; select utl_raw.cast_to_varchar2(utl_encode.base64_encode(dbms_crypto.hash (utl_i18n.string_to_raw (body || chr(10), 'AL32UTF8'),dbms_crypto.hash_sh256))) into content_sha256 from dual; /* ########################################################################################## # # # Set The signing string # # # ########################################################################################## */ content_sha256_header := 'x-content-sha256: ' || content_sha256 ; content_type_header := 'content-type: application/json'; content_length := length(body); content_length_header := 'content-length: ' || content_length ; date_header := 'date: ' || vardate; host_header := 'host: ' || host; headers := '(request-target) date host x-content-sha256 content-type content-length'; request_target := '(request-target): post ' || rest_api ; signing_string := request_target || chr(10) || date_header || chr(10) || host_header; signing_string := signing_string || chr(10) || content_sha256_header || chr(10) || content_type_header || chr(10) || content_length_header; sign_raw := DBMS_CRYPTO.SIGN ( src => UTL_I18N.STRING_TO_RAW(signing_string,'AL32UTF8'), prv_key => UTL_I18N.STRING_TO_RAW( prvkey, 'AL32UTF8'), pubkey_alg => kType, sign_alg => sType ); /* ########################################################################################## # # # encode the signing string in Base64 and build the final signature # # # ########################################################################################## */ select utl_raw.cast_to_varchar2(utl_encode.base64_encode(sign_raw)) into signature from dual; signature_final := ''; while position64 < length(signature) loop if position64 + 63 > length(signature) THEN signature_final := signature_final || substr(signature,position64,length(signature) - (position64-1)); else signature_final := signature_final || substr(signature,position64,64); end if; position64 :=position64 + 66; end loop; /* ########################################################################################## # # # Create the authorization header string including the signature string # # # ########################################################################################## */ authorization_header := 'Signature version="1",keyId="' || tenancy_ocid || '/' || user_ocid || '/' || fingerprint || '",algorithm="rsa-sha256",headers="(request-target) date host x-content-sha256 content-type content-length",signature="' || signature_final || '"'; /* ########################################################################################## # # # Send the post request to create the pre-authenticated URL # # # ########################################################################################## */ req := utl_http.begin_request('http://' || host || rest_api, 'POST',' HTTP/1.1'); utl_http.set_header(req, 'date', vardate); utl_http.set_header(req, 'x-content-sha256', content_sha256); utl_http.set_header(req, 'Content-Type', 'application/json'); utl_http.set_header(req, 'Content-Length', content_length); utl_http.set_header(req, 'Authorization', authorization_header); utl_http.write_text(req, body); /* ########################################################################################## # # # Read through the response and display the resulting URL # # # ########################################################################################## */ res := utl_http.get_response(req); begin loop utl_http.read_line(res, buffer); dbms_output.put_line(' '); json_obj := json_object_t.parse (buffer); url_string := json_obj.get('accessUri').to_string; dbms_output.put_line('URI to use : http://' || host || substr(url_string,2,length(url_string)-2)); dbms_output.put_line(' '); end loop; utl_http.end_response(res); exception when utl_http.end_of_body then utl_http.end_response(res); end; END; /

Step #2a Declare variables

The first part of the pl/sql package declares the variables that will be used in the package. Most of the variables are normal VARCHAR variables, but there a re a few other variable types that are specific to the packages used to encrypt and send the URL request.

• sType,kType - These are constants used to sign the URL request with  RSA 256 encryption
• utl_http.req,utl_http.resp - These are request and response types used when accessing the object storage
• json_obj - This type is used to extract the url from the resulting JSON code returned from the object storage call. 

Step #2b Set variables

In this section of code I set the authentication information along with the host, and the private key part of my RSA public/private key pair. 

I also set a variable with the current date time, in the correct GMT format.

NOTE: This date time stamp is compared with the date time on the ZFSSA. It must be within 5 minutes of the ZFSSA date/time or the request will be rejected.

Step #2c Set JSON body

In this section of code, I build the actual request for the pre-authenticated URL.  The parameters for this are...

• accessType - I set this to "ObjectRead" which allows me to create a URL that points to a specific object.  Other options are Write, and ReadWrite.
• bucketListingAction - I set this to "Deny",  This disallows the listing of objects.
• bucketName - Name of the bucket
• name - A name you give the request so that it can be identified later
• namespace/namepaceName - This is the ZFS share
• objectName - This is the object name on the share that I want the request to refer to. 
• timeExpires - This is when the request expires.

NOTE: I did not spend the time to do a lot of customization to this section.  You could easily make the object name a parameter that is passed to the package along with the bucketname etc. You could also dynamically set the expiration time based on sysdate.  For example you could have the request only be valid for 24 hours by dynamically setting the timeExpires.

The last step in this section of code is to create a sha256 digest of the JSON "body" that I am sending with the request.  I created it using the dbms_crypto.hash.

Step #2d Create the signing string

This section builds the signing string that is encrypted.  This string is set in a very specific format.  The string that is build contains.

(request-target): post /oci/n/{share name}/b/{bucket name}/p?compartment={compartment} 

date: {date in GMT}

host: {ZFS host}

x-content-sha256: {sha256 digest of the JSON body parameters}

content-type: application/json

content-length: {length of the JSON body parameters}

NOTE: This signing string has to be created with the line feeds.

The final step in this section is sign the signing string with the private key.

In order to sign the string the DBMS_CRYPTO.SIGN package is used.

Step #2e Build the signature from the signing string

This section takes the signed string that was built in the prior step and encodes the string in Base 64.  This section uses the utl_encode.base64_encode package to sign the raw string and it is then converted to a varchar.

Note: The resulting base64 encoded string is broken into 64 character sections.  After creating the encoded string, I loop through the string, and combine the 64 character sections into a single string.

This took the most time to figure out.

Step #2f Create the authorization header

This section dynamically builds the authorization header that is sent with the call.  This section includes the authentication parameters (tenancy OCID, User OCID, fingerprint), the headers (these must be in the order they are sent), and the signature that was created in the last 2 steps.

Step #2g Send a post request and header information

The next section sends the post call to the ZFS object storage followed by each piece of header information.  After header parameters are sent, then the JSON body is sent using the utl_http.write_text call.  

Step #2h Loop through the response

This section gets the response from the POST call, and loops through the response.  I am using the json_object_t.parse call to create a JSON type, and then use the json_obj.get to retrieve the unique request URI that is created.

Finally I display the resulting URI that can be used to retrieve the object itself.

Documentation

There were a few documents that I found very useful to give me the correct calls in order to build this package.

Signing request documentation : This document gave detail on the parameters needed to send get or post requests to object storage.  This document was very helpful to ensure that I had created the signature properly.

Http message signature format : This document gives detail on the signature itself and the format.

OCI rest call walk through : This post was the most helpful as it gave an example of a GET call and a PUT call. I was first able to create a GET call using this post, and then I built on it to create a GET call. 

File Retention on ZFS now supports expired file deletion/holds and changing permissions

 The latest release of ZFS (8.8.63) contains 2 new features associated with File Retention lock.

• File retention (deletion or hold) after file expiration
• Allow permission changes on retained files.

File retention on expiry policy

This new setting for projects/shares defaults to "off" which is the normal behavior of unlocking files, but leaving them on the filesystem.  In order to delete files you need to wait until the retention period expires, and then you can delete the file.

There are 2 new settings you can use to work with locked files to change this behavior.

Delete

When set to "Delete", files will be immediately deleted when their retention lock expires. This can be very useful if you want files to be automatically cleaned up at the end of their retention without having to create a deletion process.

There are a few items to be aware of the automatic deletion process.

1. DO NOT use this with an RMAN retention window.  Customers typically use a weekly full/daily incremental backup strategy with RMAN. With this strategy, a weeks worth of backups (all dependent on the oldest full backup) are deleted together.  Deleting backups as soon as they expire would delete backups too soon.  Even with archival backups I recommend letting RMAN perform the deletions, otherwise you risk having a file deleted too early.
2. Be careful changing this setting on an existing share.  This setting takes effect immediately and will affect ALL files that have a retention lock.  Any files that were locked, but their retention lock expired will be deleted when this setting is applied.

Hold

When set to "Hold", any files that have, or have had a retention lock will be affected.  This setting  immediately prevents the deletion of all retention locked files until the hold is removed regardless of when the lock is set to expire.  Keep in mind that while a hold is in place, the files still have a retention lock with an expiration date.

Removing the hold: When you remove the hold, the normal expiration date takes effect.  If you remove the hold by changing the expiry policy to "Delete", ALL files that have an expired retention will be immediately deleted.  If you change the expire policy to "Off", the files remain, and you must manually delete them.

NOTE: Be very careful when changing the Expiry Policy.  The new setting immediately affects existing files, not just new files going forward, unlike the other file retention settings.

Allow permission changes on retained files

What happens normally : When a file has a retention lock set, you are protecting this file from both being deleting AND from being updated. Because you were not allowed to update the file permissions, you were not able to change the settings from the default of -r--r-----+ while the file was locked.

This could be an issue depending on what type of file you are protecting. There are some cases where you want to make this file either

• An executable file, not just a read only file.
• readable by any user.

When you attempted to make this change to a locked file the update would fail with a "Operation not permitted".

-r--r-----+ 1 oracle oinstall 792 Dec  4  2023 testfile
[oracle@ssh-server rmanbackups]$ ls -al testfile
-r--r-----+ 1 oracle oinstall 792 Dec  4 20:26 testfile
[oracle@ssh-server rmanbackups]$ chmod 550 testfile
chmod: changing permissions of 'testfile': Operation not permitted
[oracle@ssh-server rmanbackups]$ chmod 444 testfile
chmod: changing permissions of 'testfile': Operation not permitted
[oracle@ssh-server rmanbackups]$

What this setting does: When you check the setting for "Allow permission changes on retained files" you are IMMEDIATELY able to change the permissions on files that are locked.  The files are still protected from making them writable, but you can adjust both the "r" - read and "x" execute settings for all users. 

NOTE: this setting does take affect immediately and will affect all currently locked files regardless of when they were created.

ZFS storing encryption keys in Oracle Key Vault (OKV)

 ZFS can be configured to use Oracle Key Vault (OKV)  as a KMIPs cluster to store it's encryption keys. In this blog post I will go through how to configure my ZFS replication pair to utilize my OKV cluster and take advantage of the Raw Crypto Replication mode introduced in 8.8.57.

OKV Cluster Environment:

First I am going to describe the environment I am using for my OKV cluster.

I have 2 OKV servers, OKVEAST1 ( IP:10.0.4.230)  and OKVEAST2 (IP: 10.0.4.254). These OKV servers are both running 21.6 (the current release as of writing this post).

ZFS replication Pair:

For my ZFS pair, I am using a pair of ZFS hosts that I have been running for awhile.  My first ZFS host is "testcost-a" (IP: 10.0.4.45)  and my second ZFS host is "zfs_s3"( IP: 10.0.4.206).  Both of these servers are running the 8.8.60 release.

For my replication, I already have "testcost-a" configured as my upstream, and "zfs_s3" configured as the downstream.

Steps to configure encryption using OKV

Documentation:

The documentation I am using to configure ZFS can be found in the 8.8.x Storage Administrators guide.  I did look through the documentation for OKV, and I didn't find anything specific that needs to be done when using OKV as a KMIP server.

Step #1 - Configure endpoints/wallets in OKV

The first step is to create 2 endpoints in OKV and assign a shared wallet between these 2 endpoints. 

 I am starting by creating a single wallet that I am going to use share the encryption keys between my 2 ZFS replication pairs.  I

The next step after creating the wallet is to create the 2 endpoints. Each ZFS host is an endpoint. Below is the screenshot for adding the first node.

After creating both endpoints I see them in the OKV console.

Then I click on each endpoint and ensure that 

• The default wallet for each endpoint is the "ZFS_ENCRYPTION_KEYS" wallet
• The endpoint has the ability to manage this wallet.

Then I go back to endpoint list in the console and save the "enrollment token" for each node and logout.

Server                    Enrollment Token

ZFS_S3        FdqkaimSpCUBfVqV

TESTCOST-A         uy59ercFNjBisU12

I then go to the main screen for OKV and click on the enrollment token download

Enter the Enrollment Token and click on "Submit Token"

You see that the token is validated. Then click on Enroll and it will download the token "okvclient.jar" which I am renaming to okvclient_{zfs server}.zip.  This will allow me to extract the certificates.

When completed, I have enrolled the endpoints and I am ready to add them to the ZFS.

Step #2 - Add the Certificates 

When I look at the .jar files that were created for the endpoints I can see all the files that are included in the endpoint enrollment. I need to add the certificates to the ZFS servers.  I can find those in the "ssl" directory contained in .jar file.

I start by uploading the "key.pem" for my first ZFS "testcost-a" in the Configuration=>SETTINGS=>Certificates=>System section of the BUI.

After uploading it I then add the "cert.pem" certificate in system also.

After uploading, I clicked in the pencil to see the details for the certificate.  

NOTE: The IP Address is the primary node in my OKV cluster.

Under Certificates=>Trusted I uploaded the CA.pem certificate.

After uploading this certificate, I click on the pencil and select "kmip" identifying this certificate to be used for the KMIP service.

The certificate should now appear as a trusted KMIP services certificate.

I can now upload the certificates for my other ZFS server (zfs_s3) the same way.

Step #3 - Add the OKV/KMIP service

I now navigate to the Shares=>ENCRYPTION=>KMIP section of the BUI to add the KMIP servers to my first ZFS host.  Because I have 2 possible KMIP servers (I am using an OKV cluster), I am going to uncheck the "Match Hostname against certificate subject" button.  I left the default to destroy the key when removing it from the ZFS.

I added the 2 OKV servers (if I had a more than 2 nodes in my cluster I would add those nodes also).  I added the port used for KMIP services on OKV (5696), and I chose the "Client TLS Authentication Certificate" I uploaded in the previous step (FLxULFbeMO).

I perform the same process on my second ZFS so that the paired ZFS servers are all configured to communicate with my OKV cluster to provide KMIP services.

NOTE: If you want to get the list of OKV hosts in the cluster you can look in the .jar file within the conf=>install.cfg file to see the OKV servers details. Below is the contents of my file.

Once I add the KMIP configuration to both of my ZFS servers I can look at my endpoints in OKV and see that they are both ENROLLED, and that OKV knows the IP address of my ZFS servers.

Step #4 - Add one or more keys.

On my upstream ZFS, I click on the "+" to add a new key and save it.

After adding it, the key appears in this section.

Step #5 - Add the keys to the shared wallet

I noticed that even though the wallet is the default wallet for the endpoints, the key did not get added to the wallet. I can see that both nodes have access to manage the wallet.

I clicked on the wallet, and then the "Add Contents", from there I am adding my new key to the wallet.

And now I login into the second ZFS (zfs_s3) and add the same key.  Make sure you add the same named key on the second ZFS so that they can match.

Step #6 - Create a new encrypted project/share

On my first ZFS (upstream - testcost-a) I am creating a new project and share that is encrypted using the key from the KMIP service.

Then within the share, I configure replication to my paired ZFS.

And now I am creating a share within this project.

Step #7 - Configure replication

Finally I configured replication from my project in my upstream (testcost-a) to my downstream (zfs_s3).  Below are the settings for my replication processing to send a snapshot every 10 minutes.  Notice that I made sure that I did NOT disable raw Crypto Mode (which is what I am using for this replication).  You can follow this link to learn more about Raw Crypto Replication.

Result:

I now have replication on my encrypted share working between my upstream and downstream.  With this new feature, the blocks are sent in their original encrypted format, and are stored on the downstream encrypted.  Since both ZFS servers can access the encryption key, both servers are able to decrypt the blocks.

I did test shutting down one of my OKV hosts, and found that the ZFS severs were able to successfully connect to the surviving node.

I even mounted the share, stored some files, replicated it, mounted a snapshot copy, and ensured that both ZFS servers presented the shares readable.

ZDLRA - Copy-to-cloud steps by step explained

 One of the best features of the ZDLRA is the ability to dynamically create a full Keep backup and send it to Cloud (ZFSSA or OCI) for archival storage.

Here is a great article by Oracle Product Manager Marco Calmasini that explains how to use this feature.

In this blog post, I will go through the RACLI steps that you execute, and explain what is happening with each step

The documentation I am started with is the 21.1 administrators Guide which can be here.  If you are on a more current release, then you can find the steps in chapter named "Archiving Backups to Cloud".

Deploying the OKV Client Software

To ensure that all the backup pieces are encrypted, you must use OKV (Oracle Key Vault) to manage the encryption keys that are being used by the ZDLRA.  Even if you are using TDE for the datafiles, the copy-to-cloud process encrypts ALL backup pieces including the backup of the controlfile, and spfile which aren't already encrypted.

I am not going to go through the detailed steps that are in the documentation to configure OKV, but I will just go through the high level processes.

The most important items to note on this sections are

• Both nodes of the ZDLRA are added as endpoints, and they should have a descriptive name that identifies them, and ties them together.
• A new endpoint group should be created with a descriptive name, and both nodes should be added to the new endpoint group.
• A new virtual wallet is created with a descriptive name, and this needs to both associated with the 2 endpoints, and be the default wallet for the endpoints.
• Both endpoints of the ZDLRA are enrolled through OKV and during the enrollment process a unique enrollment token file is created for each node. It is best to immediately rename the files to identify the endpoint it is associated with using the format <myhost>-okvclient.jar.
• Copy the enrollment token files to the /radump directory on the appropriate host.

NOTE: It is critical that you follow these directions exactly, and that each node has the appropriate enrollment token with the appropriate name before continuing.

#1 Add credential_wallet

racli add credential_wallet

Fri Jan 1 08:56:27 2018: Start: Add Credential Wallet

Enter New Keystore Password: <OKV_endpoint_password>

Confirm New Keystore Password:

Enter New Wallet Password: <ZDLRA_credential_wallet_password> 

Confirm New Wallet Password:

Re-Enter New Wallet Password:

Fri Jan 1 08:56:40 2018: End: Add Credential Wallet

The first step to configure the ZDLRA to talk to OKV is to have the ZDLRA create a password protected SEPS wallet file that contains the OKV password.

This step asks for 2 new passwords when executing

1. New Keystore Password - This password is the OKV endpoint password.  This password is used to communicate with OKV by the database, and can be used with okvutil to interact with OKV directly
2. New Wallet Password - This password is used to protect the wallet file itself that will contain the OKV keystore password.

This password file is shared across both nodes.

Update contents      -  "racli add credential"

Change password    - "racli alter credential_wallet"

#2 Add keystore

racli add keystore --type hsm --restart_db

RecoveryAppliance/log/racli.log

Fri Jan 1 08:57:03 2018: Start: Configure Wallets

Fri Jan 1 08:57:04 2018: End: Configure Wallets

Fri Jan 1 08:57:04 2018: Start: Stop Listeners, and Database

Fri Jan 1 08:59:26 2018: End: Stop Listeners, and Database

Fri Jan 1 08:59:26 2018: Start: Start Listeners, and Database

Fri Jan 1 09:02:16 2018: End: Start Listeners, and Database

The second step to configure the ZDLRA to talk to OKV is to have the ZDLRA database be configured to communicate with OKV. The Database on the ZDLRA will be configured to use the OKV wallet for encryption keys which requires a bounce of the database.  

Backout         - "racli remove keystore" 

Status            - "racli status keystore"

Update          - "racli alter keystore"

Disable          - "racli disable keystore"

Enable            - "racli enable keystore"

#3 Install okv_endpoint (OKV client software)

racli install okv_endpoint

23 20:14:40 2018: Start: Install OKV End Point [node01]

Wed August 23 20:14:43 2018: End: Install OKV End Point [node01]

Wed August 23 20:14:43 2018: Start: Install OKV End Point [node02]

Wed August 23 20:14:45 2018: End: Install OKV End Point [node02]

The third step to configure the ZDLRA to talk to OKV is to have the ZDLRA nodes (OKV endpoints) enrolled in OKV.  This step will install the OKV software on both nodes of the ZDLRA, and complete the enrollment of the 2 ZDLRA nodes with OKV.  The password that entered in step #1 for OKV is used during the enrollment process.

Status            - "racli status okv_endpoint"

NOTE: At the end of this step, the status command should return a status of online from both nodes.

Node: node02

Endpoint: Online

Node: node01

Endpoint: Online

#4 Open the Keystore

racli enable keystore

The fourth step to configure the ZDLRA to talk to OKV is to have the ZDLRA nodes open the encryption wallet in the database. This step will use the saved passwords from step #1 and open up the encryption wallet.

NOTE: This will need to be executed after any restarts of the database on the ZDLRA.

#5 Create a TDE master key for the ZDLRA in the Keystore

racli alter keystore --initialize_key

The final step to configure the ZDLRA to talk to OKV is to have the ZDLRA create the master encryption for the ZDLRA in the wallet.

Creating Cloud Objects for Copy-to-Cloud

These steps create the cloud objects necessary to send backups to a cloud location.

NOTE: If you are configuring multiple cloud locations, you may go through these steps for each location.

Configure public/private key credentials

Authentication with the object storage is done using an X.509 certificate.  The ZDLRA steps outlined in the documentation will generate a new pair of API signing keys and register the new set of keys.

You can also use any set of API keys that you previously generated by putting your private key in the shared location on the ZDLRA nodes..

In OCI each user can only have 3 sets of API keys, but the ZFSSA has no restrictions on the number of API signing keys that can be created.

Each "cloud_key" represents an API signing key pair, and each cloud_key contains 

1. pvt_key_path - Shared location on the ZDLRA where the private key is located
2. fingerprint      - fingerprint associated with the private key to identify which key to use.

You can use the same "cloud_key" to authenticate to multiple buckets, and even different cloud locations.

Documentation steps to create new key pair

#1 Add Cloud_key

racli add cloud_key --key_name=sample_key

Tue Jun 18 13:22:07 2019: Using log file /opt/oracle.RecoveryAppliance/log/racli.log

Tue Jun 18 13:22:07 2019: Start: Add Cloud Key sample_key

Tue Jun 18 13:22:08 2019: Start: Creating New Keys

Tue Jun 18 13:22:08 2019: Oracle Database Cloud Backup Module Install Tool, build 19.3.0.0.0DBBKPCSBP_2019-06-13

Tue Jun 18 13:22:08 2019: OCI API signing keys are created:

Tue Jun 18 13:22:08 2019:   PRIVATE KEY --> /raacfs/raadmin/cloud/key/sample_key/oci_pvt

Tue Jun 18 13:22:08 2019:   PUBLIC  KEY --> /raacfs/raadmin/cloud/key/sample_key/oci_pub

Tue Jun 18 13:22:08 2019: Please upload the public key in the OCI console.

Tue Jun 18 13:22:08 2019: End: Creating New Keys

Tue Jun 18 13:22:09 2019: End: Add Cloud Key sample_key

This step is used to generate a new set of API signing keys,

The output of this step is a shared set of files on the ZLDRA which are stored in:

/raacfs/raadmin/cloud/key/{key_name)/

In order to complete the cloud_key information, you need to add the public key to OCI, or to the ZFS and save the fingerprint that is associated with the public key. The fingerprint is used in the next step.

#2 racli alter cloud_key

racli alter cloud_key --key_name=sample_key --fingerprint=12:34:56:78:90:ab:cd:ef:12:34:56:78:90:ab:cd:ef

The fingerprint that is associated with the public key (from the previous step) is added to the ZDLRA cloud_key information so that it can be used for authentication.  

Both the private key, and the fingerprint are need to use the API signing key for credentials.

Using your own API signing key pair

#1 Add cloud_key

racli add cloud_key --key_name=KEY_NAME [--fingerprint=PUBFINGERPRINT --pvt_key_path=PVTKEYFILE]

You can add your own API signing keys to the ZDLRA by  using the "add cloud_key" command identifying both the private key file location (it is best to follow the format and location in the automated steps) and the fingerprint associated with the API signing keys.

It is assumed that the public key has already been added to OCI, or to the ZFSSA.

Status        - racli list cloud_key

Delete        - racli remove cloud_key

Update       - racli alter cloud_key

Documentation steps to create a new cloud_user 

This step is used to create the wallet entry on the ZDLRA that is used for authenticating to the object store.

This step combines the "cloud_key", which contains the API signing keys, the user login information and the compartment (on ZFSSA the compartment is the share ).

The cloud_user can be used for authentication with multiple buckets/locations that are identified as cloud_locations as long as they are within the same compartment (share on ZFSSA).

The format of the command to create a new cloud_user is below

racli add cloud_user 

--user_name=sample_user

--key_name=sample_key

--user_ocid=ocid1.user.oc1..abcedfghijklmnopqrstuvwxyz0124567901

--tenancy_ocid=ocid1.tenancy.oc1..abcedfghijklmnopqrstuvwxyz0124567902

--compartment_ocid=ocid1.compartment.oc1..abcedfghijklmnopqrstuvwxyz0124567903

The parameters for this command are

• user_name        - This is the username that is associated with the cloud_user to unique identify it.
• key_name         - This is name of the "cloud_key" identifying the API signing keys to be used.
• user_ocid          - This is the Username for authentication. In OCI this is the users OCID, in ZFS, this combines the ocid format with the username on the ZFSSA that owns the share.
• tenancy_ocid    - this is the tenancy OCID in OCI, on ZFSSA it is ignored
• compartment_ocid - this is the OCID, on ZFSSA it is the share

For more information on configuring the ZFSSA see

How to configure Zero Data Loss Recovery Appliance to use ZFS OCI Object Storage as a cloud repository (Doc ID 2761114.1)

List                - racli list  cloud_user

Delete            - racli remove  cloud_user

Update           - racli alter cloud_user

Documentation steps to create a new cloud_location 

This step is used to associate the cloud_user (used for authentication) with both the location and the bucket that is going to be used for backups.

racli add cloud_location

--cloud_user=<CLOUD_USER_NAME>

--host=https://<OPC_STORAGE_LOCATION>

--bucket=<OCI_BUCKET_NAME>

--proxy_port=<HOST_PORT>

--proxy_host=<PROXY_URL>

--proxy_id=<PROXY_ID>

--proxy_pass=<PROXY_PASS>

--streams=<NUM_STREAMS>

[--enable_archive=TRUE]

--archive_after_backup=<number>:[YEARS | DAYS]

[--retain_after_restore=<number_hours>:HOURS]

--import_all_trustcert=<X509_CERT_PATH>

--immutable
--temp_metadata_bucket=<metadata_bucket>  

 

I am going to go through the key items that need to be entered here.  I am going to skip over the PROXY information and certificate.

• cloud_user - This is the object store authentication information that was created in the previous steps.
• host - This the URL for the object storage location. On ZFS the namespace in the URL is the "share"
• bucket - This is the bucket where the backups will be sent. The bucket will be created if it doesn't exists. 
• streams - The maximum number of channels to use when sending backups to the cloud
• enable_archive - Not used with ZFS. With OCI the default TRUE allows you to set an archival strategy, FALSE will automatically put backups in archival storage.
• archive_after_restore - Not used with ZFS. Automatically configures an archival strategy in OCI
• retain_after_restore - Not used with ZFS. Sets the period of time that backups will remain in standard storage before returning to archival storage.
• immutable - This allows you to set retention rules on the bucket by using the <metadata_bucket> for temporary files that need to be deleted after the backup. When using immutable you must also have a temp_metadata_bucket
• temp_metadata_bucket - This is used with immutable to configure backups to go to 2 buckets, and this bucket will only contain a temporary object that gets deleted after the backup completes.

This command will create multiple attribute sets (between 1 and the number of streams) for the cloud_location that can be used for sending archival backups to the cloud with different numbers of channels.

The format of <copy_cloud_name> is a combination of  <bucket name> and <cloud_user>.

The format of the attributes used for the copy jobs is <Cloud_location_name>_<stream number>

Update          - racli alter cloud_location

Disable          - racli disable cloud_location  - This will pause all backups going to this location

Enable           - racli enable cloud_location  - This unpauses all backups going to this location

List                - racli list  cloud_location

Delete            - racli remove cloud_location

NOTE: There are quite a few items to note in this section.

• When configuring backups to go to ZFSSA use the documentation previously mentioned to ensure the parameters are correct.
• When executing this step with ZFSSA, make sure that the default OCI location on the ZFSSA is set to the share that you are currently configuring. If you are using multiple shares for buckets, then you will have to change the ZFSSA settings as you add cloud locations.
• When using OCI for archival ensure that you configure the archival rules using this command. This ensures that the metadata objects, which can't be archived are excluded as part of the lifecycle management rules created during this step.

Create the job template using the documentation.

ZFSSA replicating locked snaphots to OCI for offsite backup

ZFSSA replication can be used to create locked offsite backups. In this post I will show you how to take advantage of the new "Locked Snapshot" feature of ZFSSA and the ZFS Image in OCI to create an offsite backup strategy to OCI.

If you haven't heard of the locked snapshot feature of ZFSSA I blogged about here.  In this post I am going to take advantage of this feature and show you how you can leverage it to provide a locked backup in the Oracle Cloud using the ZFS image available in OCI.

In order to demonstrate this I will start by following the documentation to create a ZFS image in OCI as my destination.  Here is a great place to start with creating the virtual ZFS appliance in OCI.

Step 1 - Configure remote replication from source ZFSSA to ZFS appliance in OCI. 

By enabling the "Remote Replication" service with a named destination, "downstream_zfs" in my example, I can now replicate to my ZFS appliance in OCI.

Step 2 -  Ensure the source project/share has "Enable retention policy for Scheduled Snapshots" turned on

For my example I created a new project "Blogtest".  On the "snapshots" tab I put a checkmark next to 
"Enable retention policy for Scheduled Snapshots".  By checking this, the project will adhere to preventing the deletion of any locked snapshots.  This property is replicated to the downstream and will cause the replicated project shares to also adhere to locking snapshots.  This can also be set at the individual share level if you wish to control the configuration of locked snapshots for individual shares.

Below you can see where this is enabled for snapshots created within the project.

Step 3 -  Create a snapshot schedule with "locked" snapshots

The next step is to create locked snapshots. This can be done at the project level (affecting all shares) or at the share level. In my example below I gave the scheduled snapshots a label "daily_snaps".  Notice for my example I am only keeping only 1 snapshot and I am locking the snapshot at the source. In order for the snapshot to be locked at the destination

• Retention Policy MUST be enabled for the share (or inherited from the project).
• The source snapshot MUST be locked when it is created

Step 4 -  Add replication to downstream ZFS in OCI

The next step is to add replication to the project  configuration to replicate the shares to my ZFS in OCI. Below you can see the target is my "downstream_zfs" that I configured in the "Remote Replication" service.

You can also see that I am telling the replication to "include snapshots", which are my locked snapshots, and also to "Retain user snapshots on target".  Under "Disaster Recovery" you can see that I am telling the downstream to keep a 30 day recovery point.  Even though I am only keeping 1 locked snapshot on the source, I want to keep 30 days of recovery on the downstream in OCI.

Step 5 -  Configure snapshots to replicate

In this step I am updating the replication action to replicate the locked scheduled snapshot to the downstream.  Notice that I changed the number of snapshots from 1 (on the source) to 30 on the destination, and I am keeping the snapshot retention locked. This will ensure that the daily locked snapshot taken on the source will replicate to the destination as a locked snapshot, and 30 snapshots on the destination will remain locked.  The 31st snapshot is no longer needed.

Step 6 -  Configure the replication schedule

The last step is to configure the replication schedule. This ensures that on a daily basis the snapshots that are configured to be replicated will be replicated regularly to the downstream. You can make this more aggressive than daily if you wish the downstream to be more in sync in the primary.  In my example below I configured the replication to occur every 10 minutes. This means that the downstream should have all updates as of 10 minutes ago or less. If I need to go back in time, I will have daily snapshots for the last 30 days that are locked and cannot be removed.

Step 7 -  Validate the replication

Now that I have everything configured I am going to take a look at the replicated snapshots on my destination.  I navigate to "shares" and I look under "replicat" and find my share. By clicking on the pencil and looking at the "snapshots" tab I can see my snapshot replicated over.

And when I click on the pencil next to the snapshot I can see that the snapshot is locked and I can't unlock it.

From there I can clone the snap and create a local snapshot, back it up to object storage, or reverse the replication if needed.