Building a Cyber Vault ? Don't forget your keys

When building a cyber vault, one of the most important items to manage is encryption keys.  Encrypting your data is a fundamental pillar of ransomware protection, but encryption key management is often forgotten.  

Ensuring your Cyber Vault has a current copy of your encryption keys associated with your Oracle Databases is critically important for a successful restore and recover after an attack.

Starting with the Oracle Key Vault (OKV) 21.11 release, Oracle Key Vault includes a preview of the Key Transfer Across Oracle Key Vault Clusters Integration Accelerator. You can use this feature to transfer security objects from one OKV cluster to another.

You can find more detail on this feature here, and I will describe it's benefits in this post.

The diagram below shows a typical cyber vault architecture to protect Oracle Databases with the Zero Data Loss Recovery Appliance (ZDLRA) and OKV .

Transferring keys into a cyber vault with OKV

Encryption Key Architecture

Encryption key management is a critical piece of data protection, and it is important to properly manage your keys.  Good cyber protection begins with proper encryption key management.

Local Wallets

With Oracle databases, the default (and simplest) location for encryption keys is in a locally management wallet file.  The keys are often stored in an auto-login wallet, which is automatically opened by the database at startup making key management transparent and simple, but not very secure.

Why aren't wallets secure ?

• They are often auto-login (cwallet.sso) which allows the database to open the wallet without requiring a password. This wallet file gives full access to the encryption keys.
• The wallet file is stored with the database files.  A privileged administrator, such as a DBA has access to both the database and the keys to decrypt the data directly from the hosts.  They also have the ability to delete the wallet file.
• Often the wallet file is backed up with the database, which includes an auto-login wallet.  This allows anyone who has access to backups, to also be able to decrypt the data.
• Securely backing up the wallet file separate from the database is often forgotten, especially when ASM is used as the wallet location.  Not having the wallet file when restoring the database makes restoration and recovery impossible.

Steps you can take

• Create both a passworded wallet (ewallet.p12) and local auto-login wallet (cwallet.sso). With a local auto-login wallet, the wallet can only be opened on the host where the wallet was created.
• Backup the passworded wallet only (ewallet.p12).  The auto-login wallet can always be recreated from the passworded wallet.
• Properly store the password for you passworded encryption wallet. You will need the password to rotate your encryption keys, and create the auto-login wallet.

Oracle Key Vault (OKV)

 The best way to securely manage encryption keys is with OKV.

Why ?

• Keys are managed outside of the database and cannot be accessed locally outside of the database.
• Access to keys is granted to a specific database instance.
• OKV is clustered for High Availability, and the OKV cluster can be securely backed up.
• Key access is audited to allow for early detection of access.

Encryption Keys in a cyber vault architecture

Below is a diagram of a typical cyber vault architecture using ZDLRA.  Because the backups are encrypted, either because the databases are using TDE and/or they are creating RMAN encrypted backups sent to the ZDLRA, the keys need to be available in the vault also.

Not only is the cyber vault a separate, independent database and backup architecture, the vault also contains a separate, independent OKV cluster.

This isolates the cyber vault from any attack to the primary datacenter, including any attack that could compromise encryption key availability.

Encryption Keys in an advanced cyber vault architecture

Below is a diagram of an advanced cyber vault architecture using ZDLRA.  Not only are the backups replicated to a separate ZDLRA in the vault, they are internally replicated to an Isolated Recovery Environment (IRE).  In this architecture, the recover area is further isolated, and the OKV cluster is even further isolated from the primary datacenter. This provides the highest level of protection.

OKV Encryption Key Transfer

This blog post highlights the benefit of the newly released (21.11) OKV feature to allow for the secure transfer of encryption keys.

Periodic rotation of encryption keys is a required practice to protect encryption keys, and ensuring you have the current key available in a cyber vault is challenging.

OKV solves this challenge by providing the ability to transfer any new or changed keys between clusters.

Implementing OKV in a cyber Vault

When building a cyber vault, it is recommend to build an independent OKV cluster.  The OKV cluster in the vault is isolated from the primary datacenter and protected by an airgap.  The nodes in this cluster are  not be able to communicate with the OKV cluster outside of the vault.

The OKV cluster in the vault can be created using a full, secure, backup from the OKV cluster in the primary datacenter. The backup can be transferred into the vault, and then restored to the new, independent OKV cluster providing a current copy of the encryption keys.

Keeping OKV managed keys updated in a cyber Vault

The challenge, once creating an isolated OKV cluster has been keeping the encryption keys within the cluster current when new keys are created.  This was typically accomplished by transferring a full backup of OKV into the vault, and rebuilding the cluster using this backup.

OKV 21.11 provides the solution with secure Encryption Key Transfer.  Leveraging this feature you can securely transfer just the keys that have recently changed allowing you to manage independent OKV clusters that are synchronized on a regular basis.

The diagram below shows the flow of the secure Encryption Key Transfer package from the primary OKV cluster into the vault when the air-gap is opened.

This new OKV feature provides a much better way to securely manage encryption keys in a Cyber Vault.

Summary

As ransomware attacks increase, it is critical to protect a backup copy of your critical database in a cyber vault.  It is also critical to protect a copy of your encryption keys to ensure you can recovery those databases.  OKV provides the architecture for key management in both your primary datacenter and in a cyber vault.  The new secure key transfer feature within OKV allows you to synchronize keys across independent OKV clusters.

Listing Databases on an Oracle DB node

 In this blog post I am sharing a script that I wrote that will give you the list of databases running on a DB node.  The information  provided by the script is

• DB_UNIQUE_NAME
• ORACLE_SID
• DB_HOME

WHY

I have been working on a script to automatically configure OKV for all of the Oracle Databases running on a DB host.  In order to install OKV in a RAC cluster, I want to ensure the unique OKV software files are in the same location on every host when I set the WALLET_ROOT variable for my database.  The optimal location is to put the software under $ORACLE_BASE/admin/${DB_NAME} which should exist on single instance nodes, and RAC nodes.

Easy right?

I thought it would be easy to determine the name of all of the databases on a host so that I could make sure the install goes into $ORACLE_BASE/admin/{DB_NAME}/okv directory on each DB node.

The first item I realized is that the directory structure under $ORACLE_BASE/admin is actually the DB_UNIQUE_NAME rather than DB_NAME. This allows for 2 different instances of the same DB_NAME (primary and standby) to be running on the same DB node without any conflicts. 

Along with determining the DB_UNIQUE_NAME, I wanted to take the following items into account

• A RAC environment with, or without srvctl properly configured
• A non-RAC environment 
• Exclude directories that are under $ORACLE_BASE/admin that are not a DB_UNQUE_NAME running on the host.
• Don't match on ORACLE_SID.  The ORACLE_SID name on a DB node can be completely different from the DB_UNIQUE_NAME.

Answer:

After searching around Google and not finding a good answer I checked with my colleagues.  Still no good answer.. There were just suggestions like "srvctl config", which would only work on a RAC node where all databases are properly registered.  

The way I decided to this was to 

• Identify the possible DB_UNIQUE_NAME entries by looking in $ORACLE_BASE/admin
• Match the possible DB_UNIQUE_NAME with ORACLE_SIDs by looking in $ORACLE_BASE/diag/rdbms/${DB_UNIQUE_NAME} to find the ORACLE_SID name.  I would only include DB_UNIQUE_NAMEs that exist in this directory structure and have a subdirectory.
• Find the possible ORACLE_HOME by matching the ORACLE_SID to the /etc/oratab.  If there is no entry in /etc/oratab still include it.

Script:

Below is the script I came up with, and it displays a report of the database on the host.  This can be changed to store the output in a temporary file and read it into a script that loops through the databases.

#################################################################### ## Make sure that $ORACLE_BASE is set and exists before continuing #################################################################### if [ -z ${ORACLE_BASE+x} ] then echo "\$ORACLE_BASE must be set" exit 0 fi #################################################################### ## Make sure that $ORACLE_BASE/admin exists before continuing #################################################################### if [ -d ${ORACLE_BASE}/admin ] then echo "\$ORACLE_BASE is set to : ${ORACLE_BASE}" else echo "\$ORACLE_BASE is set to ${ORACLE_BASE} but ${ORACLE_BASE}/admin does not exist" exit 0 fi ######################################################################## ## Loop through $ORACLE_BASE/admin to find the list of DB_UNIQUE_NAMEs ######################################################################### for dbdir in $ORACLE_BASE/admin/* do DB_UNIQUE_NAME=${dbdir##*/} ## echo "possible DB_UNIQUE_NAME : ${DB_UNIQUE_NAME}" ######################################################################## ## Loop through $ORACLE_BASE/diag/rdbms/${DB_UNIQUE_NAME} to validate ######################################################################### for siddir in ${ORACLE_BASE}/diag/rdbms/${DB_UNIQUE_NAME}/* do db_sid=${siddir##*/} if [ -d ${siddir} ] then echo " " echo "DB_UNIQUE_NAME : ${DB_UNIQUE_NAME}" echo "ORACLE_SID : ${db_sid}" db_home=`cat /etc/oratab | grep ${db_sid} | cut -d ':' -f 2` if [ -z "${db_home}" ] then echo "ORACLE_HOME : ****** NOT IN /etc/oratab **** Cannot determine ORACLE_HOME *****" else echo "ORACLE_HOME : ${db_home}" fi echo " " fi done done

Output:

Below is the sample output from the script.. You can see that it doesn't require the DB to exist in the /etc/oratab file.

DB_UNIQUE_NAME : cdb1db1
ORACLE_SID     : cdb1db11
ORACLE_HOME    :  ******  NOT IN /etc/oratab **** Cannot determine ORACLE_HOME *****


DB_UNIQUE_NAME : daver
ORACLE_SID     : daver1
ORACLE_HOME    : /u01/app/oracle/product/19.0.0.0/dbhome_1


DB_UNIQUE_NAME : dbsgadat
ORACLE_SID     : dbsgadat1
ORACLE_HOME    : /u01/app/oracle/product/19.0.0.0/dbhome_1


DB_UNIQUE_NAME : dbsgprd
ORACLE_SID     : dbsgprd1
ORACLE_HOME    : /u01/app/oracle/product/19.0.0.0/dbhome_1

Finally:

If you are also trying to get a list of databases that are running on a DB node I hope this helps you.

ZFS storing encryption keys in Oracle Key Vault (OKV)

 ZFS can be configured to use Oracle Key Vault (OKV)  as a KMIPs cluster to store it's encryption keys. In this blog post I will go through how to configure my ZFS replication pair to utilize my OKV cluster and take advantage of the Raw Crypto Replication mode introduced in 8.8.57.

OKV Cluster Environment:

First I am going to describe the environment I am using for my OKV cluster.

I have 2 OKV servers, OKVEAST1 ( IP:10.0.4.230)  and OKVEAST2 (IP: 10.0.4.254). These OKV servers are both running 21.6 (the current release as of writing this post).

ZFS replication Pair:

For my ZFS pair, I am using a pair of ZFS hosts that I have been running for awhile.  My first ZFS host is "testcost-a" (IP: 10.0.4.45)  and my second ZFS host is "zfs_s3"( IP: 10.0.4.206).  Both of these servers are running the 8.8.60 release.

For my replication, I already have "testcost-a" configured as my upstream, and "zfs_s3" configured as the downstream.

Steps to configure encryption using OKV

Documentation:

The documentation I am using to configure ZFS can be found in the 8.8.x Storage Administrators guide.  I did look through the documentation for OKV, and I didn't find anything specific that needs to be done when using OKV as a KMIP server.

Step #1 - Configure endpoints/wallets in OKV

The first step is to create 2 endpoints in OKV and assign a shared wallet between these 2 endpoints. 

 I am starting by creating a single wallet that I am going to use share the encryption keys between my 2 ZFS replication pairs.  I

The next step after creating the wallet is to create the 2 endpoints. Each ZFS host is an endpoint. Below is the screenshot for adding the first node.

After creating both endpoints I see them in the OKV console.

Then I click on each endpoint and ensure that 

• The default wallet for each endpoint is the "ZFS_ENCRYPTION_KEYS" wallet
• The endpoint has the ability to manage this wallet.

Then I go back to endpoint list in the console and save the "enrollment token" for each node and logout.

Server                    Enrollment Token

ZFS_S3        FdqkaimSpCUBfVqV

TESTCOST-A         uy59ercFNjBisU12

I then go to the main screen for OKV and click on the enrollment token download

Enter the Enrollment Token and click on "Submit Token"

You see that the token is validated. Then click on Enroll and it will download the token "okvclient.jar" which I am renaming to okvclient_{zfs server}.zip.  This will allow me to extract the certificates.

When completed, I have enrolled the endpoints and I am ready to add them to the ZFS.

Step #2 - Add the Certificates 

When I look at the .jar files that were created for the endpoints I can see all the files that are included in the endpoint enrollment. I need to add the certificates to the ZFS servers.  I can find those in the "ssl" directory contained in .jar file.

I start by uploading the "key.pem" for my first ZFS "testcost-a" in the Configuration=>SETTINGS=>Certificates=>System section of the BUI.

After uploading it I then add the "cert.pem" certificate in system also.

After uploading, I clicked in the pencil to see the details for the certificate.  

NOTE: The IP Address is the primary node in my OKV cluster.

Under Certificates=>Trusted I uploaded the CA.pem certificate.

After uploading this certificate, I click on the pencil and select "kmip" identifying this certificate to be used for the KMIP service.

The certificate should now appear as a trusted KMIP services certificate.

I can now upload the certificates for my other ZFS server (zfs_s3) the same way.

Step #3 - Add the OKV/KMIP service

I now navigate to the Shares=>ENCRYPTION=>KMIP section of the BUI to add the KMIP servers to my first ZFS host.  Because I have 2 possible KMIP servers (I am using an OKV cluster), I am going to uncheck the "Match Hostname against certificate subject" button.  I left the default to destroy the key when removing it from the ZFS.

I added the 2 OKV servers (if I had a more than 2 nodes in my cluster I would add those nodes also).  I added the port used for KMIP services on OKV (5696), and I chose the "Client TLS Authentication Certificate" I uploaded in the previous step (FLxULFbeMO).

I perform the same process on my second ZFS so that the paired ZFS servers are all configured to communicate with my OKV cluster to provide KMIP services.

NOTE: If you want to get the list of OKV hosts in the cluster you can look in the .jar file within the conf=>install.cfg file to see the OKV servers details. Below is the contents of my file.

Once I add the KMIP configuration to both of my ZFS servers I can look at my endpoints in OKV and see that they are both ENROLLED, and that OKV knows the IP address of my ZFS servers.

Step #4 - Add one or more keys.

On my upstream ZFS, I click on the "+" to add a new key and save it.

After adding it, the key appears in this section.

Step #5 - Add the keys to the shared wallet

I noticed that even though the wallet is the default wallet for the endpoints, the key did not get added to the wallet. I can see that both nodes have access to manage the wallet.

I clicked on the wallet, and then the "Add Contents", from there I am adding my new key to the wallet.

And now I login into the second ZFS (zfs_s3) and add the same key.  Make sure you add the same named key on the second ZFS so that they can match.

Step #6 - Create a new encrypted project/share

On my first ZFS (upstream - testcost-a) I am creating a new project and share that is encrypted using the key from the KMIP service.

Then within the share, I configure replication to my paired ZFS.

And now I am creating a share within this project.

Step #7 - Configure replication

Finally I configured replication from my project in my upstream (testcost-a) to my downstream (zfs_s3).  Below are the settings for my replication processing to send a snapshot every 10 minutes.  Notice that I made sure that I did NOT disable raw Crypto Mode (which is what I am using for this replication).  You can follow this link to learn more about Raw Crypto Replication.

Result:

I now have replication on my encrypted share working between my upstream and downstream.  With this new feature, the blocks are sent in their original encrypted format, and are stored on the downstream encrypted.  Since both ZFS servers can access the encryption key, both servers are able to decrypt the blocks.

I did test shutting down one of my OKV hosts, and found that the ZFS severs were able to successfully connect to the surviving node.

I even mounted the share, stored some files, replicated it, mounted a snapshot copy, and ensured that both ZFS servers presented the shares readable.

Enrolling my ExaCC RAC database using REST APIs

 This post will continue the process of automating the enrollment of my RAC database using the OKV REST API, and some automation scripts. the steps to create the scripts are in my previous post.

NOTE: These steps are for ExaCC specific.  If you want to learn about configuring OKV with Autonomous Database (ADB) when using ExaCC, the product manager, Peter Wahl has a great blog post on this topic.  He also has videos as part of the "Ask Tom" series if you want to learn more about OKV 21c, or just OKV in general.

The first step is to download the zip file I created in the previous post. I downloaded it onto the first DB host in my RAC cluster.  I unzipped it into /home/oracle/okv.

Below is what I am starting with.

.
 |-lib
 | |-okvrestcli.jar
 |-bin
 |-conf
 | |-okvrestcli_logging.properties
 | |-okvrestcli.ini
 | |-ewallet.p12.lck
 | |-ewallet.p12
 | |-cwallet.sso.lck
 | |-cwallet.sso
 | |-okvclient.ora
 |-setenv.sh
 |-run-me.sh

STEP #1 - Set the environment

First I am going to set my environment to the database instance I want to configure (jckey1), and then I am going to source the environment for my OKV install.

[oracle@exacc1]$ cd /home/oracle/okv
[oracle@exacc1]$ . oraenv
ORACLE_SID = [jckey1] ? jckey1
The Oracle base remains unchanged with value /u02/app/oracle
[oracle@exacc1]$ . ./setenv.sh
 
 
create environment variables OKV_RESTCLI_HOME and OKC_RESTCLI_CONFIG  
 
$OKV_RESTCLI_HOME    :  /home/oracle/okv 
$OKV_RESTCLI_CONFIG  :  /home/oracle/okv/conf/okvrestcli.ini 
 
Adding $OKV_RESTCLI_BIN to the $PATH  

STEP #2 - Execute the enrollment creation script

The next step is to execute the run-me.sh that I created in the previous post. This will create the enrollment script. At the end of the output you will see the script it creates (okv-ep.sh).

NOTE: It will default to my DBNAME for the wallet name.

[oracle@exacc1]$ ./run-me.sh
 
executing script with $OKV_RESTCLI_HOME=/home/oracle/okv
 
 
DB Name is identified as jckey and ORACLE_SID is set to jckey1 setting
 
Press enter to keep this default [jckey], or enter the DB Name
DB Name [enter for Default]    : 
 
Using DB Name : jckey
 
#!/bin/bash
mkdir -pv /u02/app/oracle/admin/jckey/wallet
mkdir -pv /u02/app/oracle/admin/jckey/wallet/okv
 okv manage-access wallet create --wallet JCKEY --description "wallet for database JCKEY" --unique FALSE
okv admin endpoint create --endpoint JCKEY1_on_exacc1 --description "exacc11, 10.136.106.36" --type ORACLE_DB --platform L
INUX64 --unique FALSE
okv manage-access wallet set-default --wallet JCKEY --endpoint JCKEY1_on_exacc1
expect << _EOF
    set timeout 120
    spawn okv admin endpoint provision --endpoint JCKEY1_on_exacc1 --location /u02/app/oracle/admin/jckey/wallet/okv --auto
-login FALSE
    expect "Enter Oracle Key Vault endpoint password: "
    send "change-on-install\r"
    expect eof
_EOF

STEP #2 - Execute the enrollment script

[oracle@exacc1]$ ./okv-ep.sh
{
  "result" : "Success"
}
{
  "result" : "Success"
}
{
  "result" : "Success"
}
spawn okv admin endpoint provision --endpoint JCKEY1_on_exacc1 --location /u02/app/oracle/admin/jckey/wallet/okv --auto-login FALSE
Enter Oracle Key Vault endpoint password: 
{
  "result" : "Success",
  "value" : {
    "javaHome" : "/u02/app/oracle/product/19.0.0.0/dbhome_8/jdk"
  }
}

STEP #3 - We can verify what the enrollment script did

 

I am first going to look under $ORACLE_BASE/admin/$DBNAME/wallet where it placed the okv client.

[oracle@exacc1]$ pwd
/u02/app/oracle/admin/jckey/wallet
[oracle@exacc1]$ find . | sed -e "s/[^-][^\/]*\// |/g" -e "s/|\([^ ]\)/|-\1/"
.
  |-okv
 | |-bin
 | | |-okveps.x64
 | | |-okvutil
 | | |-root.sh
 | |-ssl
 | | |-ewallet.p12
 | |-csdk
 | | |-lib
 | | | |-liborasdk.so
 | |-jlib
 | | |-okvutil.jar
 | |-conf
 | | |-okvclient.ora
 | | |-logging.properties
 | | |-okvclient.lck
 | |-lib
 | | |-liborapkcs.so
 | |-log
 | | |-okvutil.deploy.log

Now I am going to verify in OKV and I can see the wallet got created for my database.

And I am going to look at the endpoint, and verify the default wallet is set.

STEP #4 Execute root.sh (only if this is the first install on this host).

I execute the root.sh script in the /bin directory as root.

[root@exacc1]# ./root.sh
Creating directory: /opt/oracle/extapi/64/hsm/oracle/1.0.0/
Copying PKCS library to /opt/oracle/extapi/64/hsm/oracle/1.0.0/
Setting PKCS library file permissions
Installation successful.

STEP #5 - Verify we can contact the OKV server

The next step is to execute the okvutil list command to verify we can contact the OKV host, and that the default wallet is configured.

[oracle@exacc1]$ ./okvutil list
Enter Oracle Key Vault endpoint password: 
Unique ID                               Type            Identifier
9E8BD892-D799-44B7-8289-94447E7ACC54    Template    Default template for JCKEY1_ON_ECC5C2N1

STEP #6 - change the OKV endpoint password 

[oracle@exacc1]$ /u02/app/oracle/admin/jckey/wallet/okv/bin/okvutil changepwd -t wallet -l /u02/app/oracle/admin/jckey/wallet/okv/ssl/
Enter wallet password: change-on-install
Enter new wallet password: {my new password}
Confirm new wallet password:  {my new password}
Wallet password changed successfully

STEP #7 Install the client and change the password on all nodes.

I followed the steps above on the other 3 nodes to install the client and change the password.

STEP #8 Upload the keys from the wallet file.

I uploaded the keys from the shared wallet files on ACFS.

[oracle@exacc1]$ /u02/app/oracle/admin/jckey/wallet/okv/bin/okvutil upload -t wallet -l /var/opt/oracle/dbaas_acfs/jckey/wallet_root/tde -v 2 -g JCKEY
okvutil version 21.1.0.0.0
Endpoint type: Oracle Database
Configuration file: /u02/app/oracle/admin/jckey/wallet/okv/conf/okvclient.ora
Server: 10.136.102.243:5696 
Standby Servers: 
Uploading from /acfs01/dbaas_acfs/jckey/wallet_root/tde
Enter source wallet password: 
Enter Oracle Key Vault endpoint password: 
ORACLE.SECURITY.DB.ENCRYPTION.Ab8Sv6Ezs08fv9Sy7/zZB8oAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.KM.ENCRYPTION.Ab8Sv6Ezs08fv9Sy7/zZB8oAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.KB.ENCRYPTION.
ORACLE.SECURITY.ID.ENCRYPTION.
ORACLE.SECURITY.KM.ENCRYPTION.ATQdCFHhVk9Yv7er6uZtDf8AAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.DB.ENCRYPTION.ATQdCFHhVk9Yv7er6uZtDf8AAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.DB.ENCRYPTION.MASTERKEY.BFF45EC14E46013BE053246A880A5564
ORACLE.SECURITY.DB.ENCRYPTION.MASTERKEY

Uploaded 2 TDE keys
Uploaded 0 SEPS entries
Uploaded 0 other secrets
Uploaded 4 opaque objects

Uploading private persona
Uploading certificate request
Uploading trust points

Uploaded 1 private keys
Uploaded 1 certificate requests
Uploaded 0 user certificates
Uploaded 0 trust points

Upload succeeded

STEP #9 Copy current wallet, and add OKV credentials.

Now you copy the current wallet file (from the ACFS location) to the tde directory (new OKV install)  next to the OKV install.

 In my case since my OKV client is installed in $ORACLE_BASE/admin/jckey/wallet (which will be the WALLET_ROOT),  the tde directory will be the file location for wallets.

I am also adding my password credentials to the local wallet.

NOTE: "OKV_PASSWORD" is used to open the wallet. "HSM_PASSWORD" is used to access the OKV server(s).

mkdir /u02/app/oracle/admin/jckey/wallet/tde_seps
mkdir /u02/app/oracle/admin/jckey/wallet/tde
cp /var/opt/oracle/dbaas_acfs/jckey/wallet_root/tde/* /u02/app/oracle/admin/jckey/wallet/tde/.
ADMINISTER KEY MANAGEMENT ADD SECRET 'Welcome1+' FOR CLIENT 'OKV_PASSWORD' TO LOCAL AUTO_LOGIN KEYSTORE '/u02/app/oracle/admin/jckey/wallet/tde_seps';
ADMINISTER KEY MANAGEMENT ADD SECRET 'Welcome1+' FOR CLIENT 'HSM_PASSWORD' TO AUTO_LOGIN KEYSTORE '/u02/app/oracle/admin/jckey/wallet/tde';

STEP # 10 Change the WALLET_ROOT

Since WALLET_ROOT can only be changed with a restart, I am going to shut down all instances in the cluster and perform the next few steps on the first node only.

SQL> alter system set WALLET_ROOT='/u02/app/oracle/admin/jckey/wallet' scope=spfile;

System altered.

SQL> shutdown immediate
startup mount;
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.
SQL> 
alter system set tde_configuration='KEYSTORE_CONFIGURATION=OKV|FILE' scope=both;

select b.name pdb_name,wrl_type,
wrl_parameter,
status,wallet_type,
keystore_mode,
fully_backed_up
from v$encryption_wallet a,v$containers b
where a.con_id = b.con_id(+);SQL> SQL> SQL> SQL> SQL> SQL> SQL>   2    3    4    5    6    7  

PDB Name   Type       WRL_PARAMETER					 Status 			WALLET_TYPE	     KEYSTORE Backed Up
---------- ---------- -------------------------------------------------- ------------------------------ -------------------- -------- ----------
CDB$ROOT   FILE       /u02/app/oracle/admin/jckey/wallet/tde/		 OPEN				AUTOLOGIN	     NONE     YES
CDB$ROOT   OKV								 OPEN_NO_MASTER_KEY		OKV		     NONE     UNDEFINED
PDB$SEED   FILE 							 OPEN				AUTOLOGIN	     UNITED   YES
PDB$SEED   OKV								 OPEN_NO_MASTER_KEY		OKV		     UNITED   UNDEFINED
JCKPDB	   FILE 							 OPEN				AUTOLOGIN	     UNITED   YES
JCKPDB	   OKV								 OPEN_NO_MASTER_KEY		OKV		     UNITED   UNDEFINED

SQL> shutdown immediate
startup ;

STEP # 11 Combine the local wallet File and OKV. 

  Next I need to migrate the keys using the local wallet. Note this will rekey the database.

ADMINISTER KEY MANAGEMENT SET ENCRYPTION KEY IDENTIFIED BY "-okv key" MIGRATE USING "-local wallet key-" WITH BACKUP;

STEP # 12 restart the instance and make sure the wallet open.

PDB Name   Type       WRL_PARAMETER                              Status              WALLET_TYPE     KEYSTORE Backed Up
---------- ---------- -------------------------------            ------------------- --------------- --------- ----------
CDB$ROOT   FILE       /u02/app/oracle/admin/jckey/wallet/tde/    OPEN                AUTOLOGIN       NONE     YES
CDB$ROOT   OKV                                                   OPEN                OKV             NONE     UNDEFINED
PDB$SEED   FILE                                                  OPEN                AUTOLOGIN       UNITED   YES
PDB$SEED   OKV                                                   OPEN                OKV             UNITED   UNDEFINED
JCKPDB     FILE                                                  OPEN                AUTOLOGIN       UNITED   YES
JCKPDB     OKV                                                   OPEN                OKV             UNITED   UNDEFINED

STEP # 13 rebuild the local wallet with the password

I deleted the original wallet files from the "tde" and "tde_seps" directories and recreated them using the exact same steps from step #9. The only addition is that I needed to create the wallet first

ADMINISTER KEY MANAGEMENT ADD SECRET 'Welcome1+' FOR CLIENT 'OKV_PASSWORD' TO LOCAL AUTO_LOGIN KEYSTORE '/u02/app/oracle/admin/jckey/wallet/tde_seps';
ADMINISTER KEY MANAGEMENT ADD SECRET 'Welcome1+' FOR CLIENT 'HSM_PASSWORD' TO AUTO_LOGIN KEYSTORE '/u02/app/oracle/admin/jckey/wallet/tde';

I then pushed executed the same commands to create the wallets on all the nodes in the clusters in the same location .

STEP # 14 - Bounce the database.

I bounced the database and made sure the wallet was open on all 4 nodes. Done.

INST_ID    PDB Name Type  WRL_PARAMETER                           Status               WALLET_TYPE   KEYSTORE Backed Up
-------- ---------- ----- ----------------------------------------  ------------------ -------------- -------- ---------
1        CDB$ROOT   OKV                                             OPEN               OKV            NONE     UNDEFINED
2        CDB$ROOT   OKV                                             OPEN               OKV            NONE     UNDEFINED
3        CDB$ROOT   OKV                                             OPEN               OKV            NONE     UNDEFINED
4        CDB$ROOT   OKV                                             OPEN               OKV            NONE     UNDEFINED
1        PDB$SEED   OKV                                             OPEN               OKV            UNITED   UNDEFINED
2        PDB$SEED   OKV                                             OPEN               OKV            UNITED   UNDEFINED
3        PDB$SEED   OKV                                             OPEN               OKV            UNITED   UNDEFINED
4        PDB$SEED   OKV                                             OPEN               OKV            UNITED   UNDEFINED
1        JCKPDB     OKV                                             OPEN               OKV            UNITED   UNDEFINED
2        JCKPDB     OKV                                             OPEN               OKV            UNITED   UNDEFINED
3        JCKPDB     OKV                                             OPEN               OKV            UNITED   UNDEFINED
4        JCKPDB     OKV                                             OPEN               OKV            UNITED   UNDEFINED
1        PDB$SEED   FILE                                            OPEN_NO_MASTER_KEY AUTOLOGIN      UNITED   UNDEFINED
1        CDB$ROOT   FILE  /u02/app/oracle/admin/jckey/wallet/tde/   OPEN_NO_MASTER_KEY AUTOLOGIN      NONE     UNDEFINED
2        CDB$ROOT   FILE  /u02/app/oracle/admin/jckey/wallet/tde/   OPEN_NO_MASTER_KEY AUTOLOGIN      NONE     UNDEFINED
3        CDB$ROOT   FILE  /u02/app/oracle/admin/jckey/wallet/tde/   OPEN_NO_MASTER_KEY AUTOLOGIN      NONE     UNDEFINED
4        CDB$ROOT   FILE  /u02/app/oracle/admin/jckey/wallet/tde/   OPEN_NO_MASTER_KEY AUTOLOGIN      NONE     UNDEFINED
1        PDB$SEED   FILE                                            OPEN_NO_MASTER_KEY AUTOLOGIN      UNITED   UNDEFINED
2        PDB$SEED   FILE                                            OPEN_NO_MASTER_KEY AUTOLOGIN      UNITED   UNDEFINED
3        PDB$SEED   FILE                                            OPEN_NO_MASTER_KEY AUTOLOGIN      UNITED   UNDEFINED
4        PDB$SEED   FILE                                            OPEN_NO_MASTER_KEY AUTOLOGIN      UNITED   UNDEFINED
1        JCKPDB     FILE                                            OPEN_NO_MASTER_KEY AUTOLOGIN      UNITED   UNDEFINED
2        JCKPDB     FILE                                            OPEN_NO_MASTER_KEY AUTOLOGIN      UNITED   UNDEFINED
3        JCKPDB     FILE                                            OPEN_NO_MASTER_KEY AUTOLOGIN      UNITED   UNDEFINED
4        JCKPDB     FILE                                            OPEN_NO_MASTER_KEY AUTOLOGIN      UNITED   UNDEFINED

That's all there is to it. I now have my ExaCC database configuring to use OKV as the key store, and autologin into the wallet on all instances !

Configuring OKV automation using REST APIs

 This post will go through the process of creating a few simple scripts to automate OKV installation using the REST API capability of OKV.

NOTE: This information was provided by the product manager Peter Wahl who has a great blog on the features of OKV and some great "Ask Tom" sessions on OKV (and other security pieces),you can watch and learn more.

Step #1 Configure RESTful Services and download client tool

First you need to configure the OKV server for RESTful Services. The instructions can be found here. This is done by navigating to the System tab and clicking on RESTful Services.

It is recommended that you only enable the RESTful services when you are actively onboarding new endpoints.

This bring up the window below.

 There are three things you want to do from this window.

1. Click on the "Enable" box to enable RESTful services
2. Download the okvrestcliepackage.zip which are the client utilities.
3. Save this setting to enable RESTful services.

Now that we have this file, we need to download it our client and start creating the scripts to automate this process.

I downloaded the zip file to my DB host to configure it. I unzipped it in /home/oracle/okv/rest

NOTE: you can also download it directly from the OKV hosts by using any of the commands below.

wget --no-check-certificate https://ip_address:5695/okvrestclipackage.zip curl -k https://ip_address:5695/okvrestclipackage.zip -o okvrestclipackage.zip curl -O -k https://ip_address:5695/okvrestclipackage.zip

Step #2 Create a user for the restapi steps

I create a new user in OKV called "restapi" and you can see the permissions below.  Before using the new user, you must login and change the password because the password is expired by default when creating a new user.

Step #3 unzip and configure the client tool 

I unzipped the client tool into my home directory on a DB server so I can put together the automation scripts. In my case I unzipped it into /home/oracle/okv/rest. This creates 3 sub directories. I am going to format the output using this command.

find . | sed -e "s/[^-][^\/]*\// |/g" -e "s/|\([^ ]\)/|-\1/"

Below is what the output looks like

.
 |-lib
 | |-okvrestcli.jar
 |-bin
 | |-okv.bat
 | |-okv
 |-conf
 | |-okvrestcli.ini
 | |-okvrestcli_logging.properties

Step #4 - Set the environment for the CLI

In order to configure OKV, I am going to need some variables set in my environment. I can do this manually, but in my case I decided to create a "setenv.sh" script that will set the variables and add the OKV script to my path to be executed. I also included the ability to pass a parameter (ORACLE_SID) so that you can use the script in a loop across multiple instances on the same host.

 The 3 main variables I will be using are

OKV_RESTCLI_HOME - Location of the scripts that I am going to be installing. If I source the setenv.sh script, it will set the home to this location.

OKV_RESTCLI_CONFIG - Name of the configuration file that contains the rest CLI configuration.

OKV_HOME - Location to install OKV for the current instance.  This location is $ORACLE_BASE/admin/${DB_UNIQUE_NAME}/okv_home.  This follows the standard for ExaCC.

NOTE: If this is a NEW database

If you want to use these steps to configure OKV on a new database, you need to perform the following steps prior to executing "/ ./setenv.sh".

1. Add the new $ORACLE_SIDE for the host in the "/etc/oratab" file.
2. Create the directory "mkdir $ORACLE_BASE/admin/{DB Unique Name}"
3. Create the directory "mkdir -p $ORACLE_BASE/diag/rdbms/${DB Unique Name}/${ORACLE_SID}
4. Use ". oraenv" to set the environment to this $ORACLE_SID

###Ensure that the setenv file is sourced from the current directory to properly set variables if [ "${BASH_SOURCE[0]}" != "${0}" ] then echo " " else echo " " echo "the setenv.sh file must be sourced with \". ./setenv.sh \"" return fi FILE=`pwd`/setenv.sh if [ -f "$FILE" ] then echo " " else echo " " echo "source this from RESTful install directory" return fi ############################################################################################# # # Check to see if a parameter is passed Or ORACLE_SID is set. Otherwise exit # ############################################################################################# if [ $# -eq 0 ] then echo " " if [ -z ${ORACLE_SID} ] then echo "No \$ORACLE_SID set and nothing passed. Exiting " echo " " return fi echo "No \$ORACLE_SID provided. Trying current \$ORACLE_SID value of : $ORACLE_SID" echo " " else echo "\$ORACLE_SID provided. Setting \$ORACLE_SID to : $1" export ORACLE_SID=$1 echo " " fi ############################################################################################# # # make sure SID is in the /etc/oratab so we can determine the # ############################################################################################# sid_found='false' for sids in `cat /etc/oratab | cut -d ":" -f 1` do if [ "$sids" = "${ORACLE_SID}" ] then sid_found='true' fi done if [ ${sid_found} = "false" ] then echo "\$ORACLE_SID : ${ORACLE_SID} not found in /etc/oratab file. Exiting" return fi ############################################################################################# # # make sure SID has exists under $ORACLE_BASE/admin to install software # ############################################################################################# db_dir=`find ${ORACLE_BASE}/diag/rdbms -maxdepth 2 -name ${ORACLE_SID}` delimeter_cut=`echo $db_dir | sed -e 's/\(.\)/\1\n/g' | grep \/ | wc -l` DB_UNIQUE_NAME=`echo $db_dir | cut -d "/" -f $delimeter_cut ` if [ -z ${DB_UNIQUE_NAME} ] then echo $db_dir echo "Invalid \$ORACLE_SID. Exiting script" echo " " echo " " return fi ############################################################################################# # # set the environment # ############################################################################################# export ORAENV_ASK=NO . oraenv export ORAENV_ASK=YES ############################################################################################# # # set the variables # ############################################################################################# echo "create environment variables OKV_HOME, OKV_RESTCLI_HOME and OKC_RESTCLI_CONFIG " ## set variables export OKV_HOME=$ORACLE_BASE/admin/$DB_UNIQUE_NAME/okv_home export OKV_RESTCLI_HOME=`pwd` export OKV_RESTCLI_BIN=$OKV_RESTCLI_HOME/bin export OKV_RESTCLI_CONFIG=$OKV_RESTCLI_HOME/conf/okvrestcli.ini ##add the OKV script to the path echo " " echo "\$OKV_HOME : $OKV_HOME " echo "\$OKV_RESTCLI_HOME : $OKV_RESTCLI_HOME " echo "\$OKV_RESTCLI_CONFIG : $OKV_RESTCLI_CONFIG " echo " " echo "Adding \$OKV_RESTCLI_BIN to the \$PATH " echo " " if [[ "$PATH" =~ (^|:)"${OKV_RESTCLI_BIN}"(:|$) ]] then echo " " else PATH=${OKV_RESTCLI_BIN}:$PATH fi string=`which dcli` echo $string if [[ $string == *"no dcli in"* ]] then export RAC_DATABASE="N" else export RAC_DATABASE="Y" fi ## Set JAVA_HOME to the location of Java if it isn't set already if [ -z ${JAVA_HOME+x} ] then export JAVA_HOME=$ORACLE_HOME/jdk echo "Setting \$JAVA_HOME to \$ORACLE_HOME/jdk : $JAVA_HOME" else echo "\$JAVA_HOME already set to : $JAVA_HOME" fi echo " "

Step #5 - Set initialization parameters in okvrestcli.ini file

Next, I am going to configure the initialization parameters. These are found in the okvrestcli.ini file.

You can see that the file contains a "[Default]" profile and a few other example profiles. We will start with the default profile. In this we are going to set a few of the properties.

LOG_PROPERTY - Location of the logging properties. Default location is ./conf directory.

SERVER - IP address (or DNS) of one or more OKV hosts 

 OKV_CLIENT_CONFIG - location of the config file. Default location is ./conf directory

USER - OKV user that has authority to administer endpoints an wallets. In this case it will be the restapi user that I created.

PASSWORD - Password for the user, or location of wallet containing the password. I am NOT going to use this as I am going to use a wallet file.

 CLIENT_WALLET - I am going to use a wallet to store the password, and this is the location of the wallet file. I will be creating the autologin wallet later.

 

Below is what my "[Default]" configuration file looks like after my changes which is located at $OKV_RESTCLI_HOME/conf/okvrestcli.ini . I am going to use the environmental variables I set in the setenv.sh script. 

NOTE: I am choosing to store my password in wallet rather than clear text in the .ini file.

          You need to change the server to either the server name or the IP address.

[Default] log_property=$OKV_RESTCLI_HOME/conf/okvrestcli_logging.properties server=10.0.0.150 okv_client_config=$OKV_RESTCLI_HOME/conf/okvclient.ora user=restapi client_wallet=$OKV_RESTCLI_HOME/conf

Step #6 Create the wallet to save the password encrypted

Since I chose to put my password in a wallet, I now need to create that wallet. Using the instructions in the document (linked to at the beginning of this blog), I execute the command from the directory I installed into (/home/oracle/okv/rest)

cd /home/oracle/okv/rest . ./setenv.sh ##### Output from script is below ######### # create environment variables OKV_RESTCLI_HOME and OKC_RESTCLI_CONFIG # # $OKV_RESTCLI_HOME : /home/oracle/okv/rest # $OKV_RESTCLI_CONFIG : /home/oracle/okv/rest/conf/okvrestcli.ini # # Adding $OKV_RESTCLI_BIN to the $PATH</i># okv admin client-wallet add --client-wallet $OKV_RESTCLI_HOME/conf --wallet-user restapi #Password: {my password} #{ # "result" : "Success" #}

Step #7 Create and execute the run-me.sh script

The last step is to create the script that will be executed  on the host to create the provision script.  In my script, I took the default and did some checking. This script will

• Ensure the variable OKV_RESTCLI_HOME is set before it can be executed.
• Determine the DB_UNIQUE_NAME from the $ORACLE_BASE/diag/rdbms/*/$ORACLE_SID directory. Solving for the  * should give us the DB_UNIQUE_NAME
• While executing, it tells you what it believes the DB_UNIQUE_NAME is, and gives you a chance to change it if incorrect.
• It will validate if the endpoint group exists by accessing OKV. If the endpoint group already exists, it does not try to create it again. If it doesn't exist it will create the endpoint group.
• It will validate if the wallet exists by accessing OKV. If the wallet already exists, it does not try to create it again. If it doesn't exist it will create the wallet .
• t will add the endpoint, add this endpoint to the endpoint group, and assign the wallet to this endpoint.
• It will install the client software in $ORACLE_BASE/admin/$DB_UNIQUE_NAME/okv_home/okv

Below is the script I am using.

#!/bin/bash ########################################################################################### # # Make sure the setenv script was executed to set the OKV_RESTCLI_HOME variable # ########################################################################################### if [ -v OKV_RESTCLI_HOME ] then echo " " echo "executing script with \$OKV_RESTCLI_HOME=$OKV_RESTCLI_HOME" echo " " echo " " else echo " " echo "\$OKV_RESTCLI_HOME not set" echo " " echo "source the setenv.sh script in the RESTful Services install directory and try again." echo " " exit 0 fi ########################################################################################### # # Determine the DBNAME Unique Name for this SID and have user validate # ########################################################################################### pushd $ORACLE_BASE/diag/rdbms/*/$ORACLE_SID > /dev/null cd .. export DB_UNIQUE_NAME=`printf '%s\n' "${PWD##*/}"` popd > /dev/null echo "DB Unique Name is identified as $DB_UNIQUE_NAME and ORACLE_SID is set to $ORACLE_SID setting" echo " " echo "****************************************************************************************** " echo " " echo "DB Name is identified as $DB_UNIQUE_NAME --- NOTE THIS MAY NOT be the correct DB_NAME --- " echo " --- NOTE THIS MAY NOT be the correct DB_NAME if there is a primary and standby --- " echo " " echo "****************************************************************************************** " echo " " echo " " echo "Press enter to keep this default [$DB_UNIQUE_NAME], or enter the DB Name" read -p 'DB Name [enter for Default] : ' newdbname if [ -z "$newdbname" ] then export DBNAME=`echo $DB_UNIQUE_NAME` else export DBNAME=`echo $newdbname` fi echo " " echo "Using DB Name : $DBNAME" echo "#!/bin/bash" > $OKV_RESTCLI_HOME/okv-install-${ORACLE_SID}.sh ########################################################################################### # # check to see if wallet already exists # create it if it doesn't exist # ########################################################################################### export wallet_exists=`okv manage-access wallet check-status --wallet ${DBNAME^^} | grep Success | wc -l` if [ $wallet_exists -ne 0 ] then echo " Wallet ${DBNAME^^} already exists skipping wallet creation. assuming this is NOT the first node for the database" else cat >> $OKV_RESTCLI_HOME/okv-install-${ORACLE_SID}.sh << EOF2 okv manage-access wallet create --wallet ${DBNAME^^} --description "wallet for database ${DBNAME^^}" EOF2 fi ########################################################################################### # # check to see if endpoint Group already exists # create endpoint group if it doesn't exist # ########################################################################################### export group_exists=`okv manage-access endpoint-group check-status --endpoint-group ${DBNAME^^} | grep Success | wc -l` if [ $group_exists -ne 0 ] then echo " endpoint group ${DBNAME^^} already exists skipping group creation. assuming this is NOT the first node for the Database" else cat >> $OKV_RESTCLI_HOME/okv-install-$ORACLE_SID.sh << EOF2 okv manage-access endpoint-group create --endpoint-group ${DBNAME^^} --description "endpoint group for database ${DBNAME^^}" EOF2 fi export EP_NAME=`echo ${ORACLE_SID^^}_on_${HOSTNAME/.*} | sed 's/\-/\_/g'` ########################################################################################### # # add commands to create/remove directories # ########################################################################################### cat >> $OKV_RESTCLI_HOME/okv-install-$ORACLE_SID.sh << EOF1 mkdir -pv ${OKV_HOME} mkdir -pv ${OKV_HOME}/okv mkdir -pv ${OKV_HOME}/tde mkdir -pv ${OKV_HOME}/tde_seps EOF1 cat > $OKV_RESTCLI_HOME/okv-remove-$ORACLE_SID.sh << EOF1 #!/bin/bash rm -r ${OKV_HOME} rm -r ${ORACLE_BASE}/okv/$ORACLE_SID EOF1 ########################################################################################### # # commands to provision instance on Node # ########################################################################################### cat >> $OKV_RESTCLI_HOME/okv-install-${ORACLE_SID}.sh << EOF3 okv admin endpoint create --endpoint ${EP_NAME} --description "$HOSTNAME, $(hostname -i)" --type ORACLE_DB --platform LINUX64 okv manage-access endpoint-group add-endpoint --endpoint-group ${DBNAME^^} --endpoint ${EP_NAME} okv manage-access wallet set-default --wallet ${DBNAME^^} --endpoint ${EP_NAME} okv manage-access wallet add-access --wallet ${DBNAME^^} --endpoint-group ${DBNAME^^} --access "RM_MW" expect << _EOF set timeout 120 spawn okv admin endpoint provision --endpoint ${EP_NAME} --location $OKV_HOME/okv --auto-login FALSE expect "Enter Oracle Key Vault endpoint password: " send "change-on-install\r" expect eof _EOF EOF3 chmod +x $OKV_RESTCLI_HOME/okv-install-$ORACLE_SID.sh more $OKV_RESTCLI_HOME/okv-install-$ORACLE_SID.sh

Step #8 Execute okv-install-${ORACLE_SID}.sh to validate that enrollment works

[oracle@pmdexa1dbadm01vm01 okv]$ ./okv-ep.sh
{
  "result" : "Success"
}
{
  "result" : "Success"
}
{
  "result" : "Success"
}
spawn okv admin endpoint provision --endpoint DEMO31_on_pmdexa1dbadm01vm01 --location /u01/app/oracle/admin/demo31/wallet/okv --auto-login FALSE
Enter Oracle Key Vault endpoint password:
{
  "result" : "Success"
}

Once I ran it I removed the files that were created for the install.

Step #9 Zip it all up and place it in a location to be downloaded

Below is the scripts that will be part of the zip file.

.
 |-wallet
 | |-ewallet.p12.lck
 | |-ewallet.p12
 | |-cwallet.sso.lck
 | |-cwallet.sso
 |-conf
 | |-okvrestcli_logging.properties
 | |-okvrestcli.ini
 |-setenv.sh
 |-runme.sh
 |-lib
 | |-okvrestcli.jar
 |-bin
 | |-okv
 | |-okv.bat

Now I am ready to download this zip file to my Database Host and enroll a database.

NOTE: To change the script to work on another OKV all host I only had to make 3 changes.

• Update the okvrestcli.ini file with OKV host IP
• Update the okvrestcli.ini file with the the user
• recreate the wallet file that contains the password for the OKV user

Step #10 Run root.sh if this is the first on a host

When a new endpoint is added, a script is downloaded into the $OKV_HOME/okv/bin/ directory called root.sh.. This script will copy the PKCS library (liborapkcs.so) in central location on the host as root. This file is needed for the database to access OKV. The location of this file is different on different OS systems.

NOTE: If you are already using a key management software on this host for Oracle databases, running root.sh could overwrite the library used.