Friday, August 29, 2025

Oracle Master Encryption Key Management: Wallets, OKV, and OCI KMS Explained

Data Encryption Keys (DEK) are used by encryption algorithms (typically AES256) to protect data at rest in tablespaces, backups and even datapump exports. Since DEKs are stored with the data, they need to be protected.  Master Encryption Keys (MEK) are stored outside the database in local wallets or in an external keystore, and MEKs are used to encrypt the DEK which provides a separation between the data that is encrypted and an external key.

DEKs are not only found on Tablespaces (TDE), but they are also found in RMAN encrypted backups, Space Efficient Encrypted backups to ZDLRA/Recovery Service and encrypted datapump exports.

The diagram below shows that relationship between MEKs and DEKs.




You can see in the diagram that MEKs are not only used to encrypt Tablespace Encryption Keys to support TDE, they are also used to encrypt RMAN backups to disk, Tape or ZDLRA/Recovery Service, along with Datapump encryption keys.  

NOTE: When implementing encryption you should make sure you meet ASO licensing restrictions
  • Creating encrypted tablespaces or encrypted columns in tables requires ASO, unless you are leveraging an OCI Cloud Database environment (including multicloud).
  • Creating RMAN encrypted disk backups requires licensing.
  • Creating RMAN encrypted tape backups is license included and only supported when writing to Oracle products.
    • OSB - Oracle Secure Backup is license included for creating encrypted backups.
    • OCI Object Storage - The OCI cloud library requires encryption and can only be leveraged for writing backups directly to OCI object storage in OCI, or writing backups from a ZDLRA to OCI object storage in cloud or on ZFSSA
    • ZDLRA - Creating Space Efficient Encrypted backups is license included on ZDLRA and will create encrypted backups on ZDLRA.
  • Creating Datapump Encrypted backups requires licensing. 


MEKs are commonly called "TDE keys", but they are much more, and I will refer to them as MEKs throughout this post.

MEK with a Wallet File

In this first section I am going to talk about how the Oracle Database manages an MEK when using a Wallet file.

Wallet settings

With wallet-based MEKs, the database stores and reads keys from a wallet file on disk (or +ASM or Exascale).

There are 2 parameters in the database (spfile) that help guide the database to the wallet file.

WALLET_ROOT - This replace the "ENCRYPTION_WALLET_LOCATION" parameter that was previous set in the sqlnet.ora file.  Having it set at the database level allows multiple databases to independently share the same $ORACLE_HOME. It even allows PDBs to have individual wallet files. When using a wallet file, the WALLET_ROOT points to top directory where the encryption keys will be kept.

TDE_CONFIGURATION - This tells the database what kind of MEK will be used. For wallets you would specify "TDE_CONFIGURATION=FILE".  This tells the database to use find the encryption keys in an encryption wallet.

Encryption key location

Within the WALLET_ROOT directory the encryption wallet containing the MEKs, the wallet must be stored in a subdirectory named "tde".

Within that directory you will find 2 wallet files

  • ewallet.p12 - The password-protected wallet (contains keys; required for updates).
  • ewallet.sso — the auto-login wallet (allowing no password prompt on open); The auto-login wallet can also be created as a “local” autologin wallet bound to a single host.

On startup (or when you open the keystore), the database retrieves the CDB/PDB MEKs from the wallet file and caches them in memory.

Database interaction with MEK

When using a wallet file, the auto-login wallet file is opened when the database is opened, and the database constantly interacts with this file.  If no auto-login wallet file is created, you must open the wallet manually providing the password.

Viewing V$ENCRYPTION_WALLET 

What you’ll typically see for FILE based wallets:

PDB_NAME        WALLET_TYPE  STATUS  WRL_PARAMETER                KEYSTORE
--------------  -----------  ------  ----------------------------  --------
CDB$ROOT        AUTOLOGIN    OPEN    {WALLET_ROOT}/tde             NONE
DB23AI_PDB      AUTOLOGIN    OPEN                                  UNITED

MEK with OKV/OCI KMS

When using an external keystore, the database communicates with the OKV/OCI KMS client libraries installed on the host in order to retrieve the MEKs.

OKV/OCI KMS settings

There are 2 parameters in the database (spfile) that help guide the database to the OKV/OCI KMS installation.

WALLET_ROOT -  When using OKV/OCI KMS, the WALLET_ROOT points to top directory of the client install of the OKV/OCI KMS software.

TDE_CONFIGURATION - For OKV/OCI KMS  specify "TDE_CONFIGURATION=OKV" or "TDE_CONFIGURATION=HSM".  It is possible to combine both wallets and  OKV/OCI KMS if there are older keys that are still stored a wallet file. In that case you would see "TDE_CONFIGURATION=OKV|FILE" for example, and the database would read older encryption keys from a wallet file along with reading the keys from OKV/OCI MKS.

Encryption key location

Within the WALLET_ROOT directory, the software to communicate with the HSM is typically installed in the "okv" or "hsm" subdirectory.  The keys themselves are stored externally, and could be securely cached locally depending on your configuration.

The credential used to access OKV/OCI KMS is typically stored under the /tde directory in an auto-login wallet. This allows the database to connect to OKV/OCI KMS at startup. The external password store entries (if used) live under /tde_seps which allows up to manage keys without entering a password.

Database interaction with MEK

The MEK is always permanently stored externally.  The database interacts with the external keystore through a library.  This library is named liborapkcs.so.  The library itself is vendor specific and is used by the database to interact with different vendors keystore,. Because there is a single library, you cannot use multiple external key store types on the same DB host.
When the database starts up, it reads the password for the external keystore out of the wallet, opens up a connection to the keystore (after authentication and encryption handshake) and caches the MEK.  The database periodically will communicate with the external keystore through that library.


OKV/OCI MKS in Practice: What You’ll See

When both FILE and OKV configurations exist (e.g., during migration), you may observe dual rows in V$ENCRYPTION_WALLET showing the FILE wallet (no active master key) and the active OKV keystore:

PDB_NAME        TYPE  STATUS             WALLET_TYPE  WRL_PARAMETER
--------------  ----  -----------------  -----------  ---------------------------
CDB$ROOT        FILE  OPEN_NO_MASTERKEY  AUTOLOGIN    {WALLET_ROOT}/tde
CDB$ROOT        OKV   OPEN               OKV
DB23AI_PDB      FILE  OPEN_NO_MASTERKEY  AUTOLOGIN
DB23AI_PDB      OKV   OPEN               OKV


Best Practices & Notes

  • Back up only the passworded wallet when using a wallet file. (ewallet.p12) and store it securely, separate from database backups.
  • Use local autologin wallets if you require host binding for additional security.
  • When migrating to OKV/OCI KMS, consider a staged approach (OKV|FILE) so you can fall back if needed.
  • Always test wallet/keystore open/close operations and RMAN/Data Pump encryption workflows in lower environments before production.
  • Keep client libraries (liborapkcs.so and OKV client) patched and consistent across RAC nodes.

Last updated: August 29, 2025

Monday, August 25, 2025

ZDLRA - How to manually register databases

Steps to Manually Register a Database and Backup to ZDLRA

Table of Contents


Section 1: User and Policy Setup

Step 1.1 – Create the VPC User Account

racli add db_user --user_name=USER_NAME --user_type=vpc [--insecure]

Notes:

  • If created with --insecure, the VPC password will expire, but can be reused. Password rollover is not supported in this mode.
  • Without --insecure, enable password rollover with the enable_stig option on alter db_user.

Step 1.2 – Create the DB Admin User Account

racli add db_user --user_name=USER_NAME --user_type=admin

Step 1.3 – Create a New Protection Policy (optional)

You can use the default protection policies provided with ZDLRA, but you may want to create custom policies for more control over the protection window and other attributes.  Creating a new policy also allows you the name the protection appropriately to ensure you chose the correct policy.

racli create protection_policy --protection_policy_name=VALUE --storage_location_name=VALUE \
 [--recovery_window_goal=VALUE] [--description=VALUE] [--polling_policy_name=VALUE] \
 [--max_retention_window=VALUE] [--recovery_window_sbt=VALUE] [--unprotected_window=VALUE] \
 [--guaranteed_copy=VALUE] [--allow_backup_deletion=VALUE] [--store_and_forward=VALUE] \
 [--log_compression_algorithm=VALUE] [--autotune_reserved_space=VALUE] \
 [--recovery_window_compliance=VALUE] [--keep_compliance=VALUE] [--comments=VALUE]

Example:

racli create protection_policy --protection_policy_name='test' --storage_location='delta' \
 --recovery_window_goal=10days --unprotected_window=30minutes --autotune_reserved_space=yes \
 --log_compression=low --recovery_window_compliance=1days --allow_backup_deletion=no



Section 2: Database Registration in ZDLRA

Step 2.1 – Add the Database to ZDLRA Metadata

racli add protected_db --db_unique_name={DB_NAME} --protection_policy_name={PROTECTION_POLICY} --reserved_space={VALUE}

Recommended reserved space = ~2x database size.

Step 2.2 – Grant VPC Users Access

racli grant db_access --db_unique_name={DB_NAME} --username={VPCUSER}



Section 3: Wallet and Network Configuration

Step 3.1 – Determine the Wallet Location

  • Check for existing wallets in $ORACLE_BASE/admin/<dbname>/wallet and sqlnet.ora.
  • Use WALLET_ROOT/server_seps (typically under $ORACLE_BASE/admin/{db_name}/wallet).
You should always store the SEPS wallet used by ZDLRA under the WALLET_ROOT location for your database. This will allow the wallet configuration for the ZDLRA to coincide with any other wallets.

NOTES:
  • Setting WALLET_ROOT requires a bounce of the database if it is not already set.  Setting WALLET_ROOT will better prepare the database for managing wallets for real-time redo, TLS, and any encryption keys in the future.
  • Having individual SEPS wallet files for each database, stored outside of $ORACLE_HOME will allow you to have different VPC users, and certificates, providing more flexibility. It also allows you to perform out-of-place upgrades without changing wallet locations.
  • Yes, you can store TDE wallets on ASM/ExaScale and customers sometimes set WALLET_ROOT to ASM/Exascale.  I don't recommend this because many of the additional wallets (SEPS, TLS, OKV) cannot be read from ASM/ExaScale.  I recommend ACFS as a shared file system.

Step 3.2 – Create or Update the Wallet

  • Create the wallet file (if It doesn't exist, or update the existing wallet file):
orapki wallet create -wallet $ORACLE_BASE/{DB_NAME}/wallet/server_seps
mkstore -wrl $ORACLE_BASE/{DB_NAME}/wallet/server_seps -createCredential {connect string} {VPC user} {password}

NOTE: Creating the wallet file with orapki allows you later add TLS certificates
  • Add any Certificates if TLS is configured on the ZDLRA (optional)
orapki wallet add -wallet $ORACLE_BASE/{DB_NAME}/wallet/server_seps -trusted_cert -cert {trusted cert PEM file}
  • Create the auto-login wallet from the passworded wallet:
orapki wallet create -wallet $ORACLE_BASE/{DB_NAME}/wallet/server_seps -auto_login


Copy wallet to all RAC nodes, or ensure it is on a shared mount point.

Step 3.3.A –Create a new sqlnet.ora for your database (Option A).

It is highly recommended to create a separate TNS_ADMIN directory for each database when multiple databases share the same host. This allows you to leverage different wallets for VPC users and TLS certificates.
This also ensures that setting WALLET_LOCATION does not affect any configurations that also use WALLET_LOCATION (i.e. TLS).

I would recommend storing the sqlnet.ora under $ORACLE_BASE/admin/{dbname}/tnsadmin and using TNS_ADMIN within your RMAN session to point to this location.
Within the sqlnet.ora make the following changes.  
SQLNET.WALLET_OVERRIDE=TRUE
WALLET_LOCATION=
 (SOURCE=
   (METHOD=FILE)
   (METHOD_DATA=
     (DIRECTORY={Wallet Location})
   )
 )

Create this directory on each RAC node (if it isn't shared) and copy the sqlnet.ora to all RAC nodes

Step 3.3.B –Update the general sqlnet.ora for your database (Option B).

If this is the only database on your host, you can update the general sqlnet.ora file within $ORACLE_HOME.  Be careful that other DB features, like TLS which may also use WALLET_LOCATION are not affected.

Within the sqlnet.ora make the following changes.  
SQLNET.WALLET_OVERRIDE=TRUE
WALLET_LOCATION=
 (SOURCE=
   (METHOD=FILE)
   (METHOD_DATA=
     (DIRECTORY={Wallet Location})
   )
 )

Copy this file to all nodes if implementing on a RAC cluster.

Step 3.4 – Test SEPS Connection

If using TNS_ADMIN, set it to appropriate location.

Execute the command below using the entry you created in the wallet with mkstore. This will ensure SEPS security is working correctly before continuing.
Check all nodes in a RAC environment.
sqlplus /@<db SCAN listener:PORT/Service:dedicated>




Section 4: Database Registration with RMAN

Step 4.1 – Register the Protected Database

Connect to the local database, and connect to the RMAN catalog using the entry in the SEPS wallet.
Once connect you need to register the database.
RMAN> CONNECT TARGET /  CATALOG /@<db_SCAN:PORT/Service:dedicated>
RMAN> REGISTER DATABASE;



Section 5: Real-Time Redo (Optional)


Step 5.1 – Update Database Parameters

ALTER SYSTEM SET REMOTE_LOGIN_PASSWORDFILE=shared;
ALTER SYSTEM SET LOG_ARCHIVE_FORMAT='log_%d_%t_%s_%r.arc';

Step 5.2 – Set LOG_ARCHIVE_CONFIG 

ALTER SYSTEM SET LOG_ARCHIVE_CONFIG='DG_CONFIG=(<ZDLRA DB>,<DB_UNIQUE_NAME>)' SCOPE=BOTH;

Step 5.3 – Update the WALLET_ROOT in the database (if not set)

ALTER SYSTEM SET WALLET_ROOT='{Oracle home}/admin/{DB name}/wallet' scope=spfile;

It is highly recommended to allow real-time redo use the default wallet located in WALLET_ROOT/server_seps.
If there is no wallet in WALLET_ROOT/server_seps, then the database will use the setting from WALLET_LOCATION in the sqlnet.ora file. The sqlnet.ora can only point to a single WALLET_LOCATION and this can cause confclits.

NOTE: This requires a bounce of the instance to take affect.

Step 5.5 – Create the Redo Transport user  (when using DG)

If you are using Dataguard, you need to ensure you add the VPC user in the database and grant it SYSOPER priveleges.
You also need to ensure that you standby database(s) have a current copy of the Password file.

For multi-tenant environments you need to execute "alter session set "_oracle_script"=true;" prior to creating the users.

Verify that the user exists in the wallet on all systems by querying against the v$password_users view. 

Step 5.6 – Change the Redo Transport User to the VPC user

Change the redo transport user on all environments (primary and standby).
ALTER SYSTEM SET redo_transport_user={VPCUSER};

Step 5.7 – Add ZDLRA as an Archive Log Destination

ALTER SYSTEM SET LOG_ARCHIVE_DEST_<n>=
 'SERVICE=<string from wallet>', 
 VALID_FOR=(ALL_LOGFILES,ALL_ROLES) ASYNC, 
 DB_UNIQUE_NAME='<ZDLRA DB>' SCOPE=BOTH;
ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_<n>=ENABLE SCOPE=BOTH;

NOTE: You can also configure DG broker to manage the ZDLRA using the "ADD RECOVERY_APPLIANCE " command.

Step 5.8 – Test Archive Log Configuration

SELECT status, error FROM v$archive_dest WHERE dest_id=<destination>;

Section 6: RMAN Channel and Backup Configuration

Step 6.1 – Add Default Channel Configuration

CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' FORMAT '%d_%U'
 PARMS 'SBT_LIBRARY=libra.so, ENV=(_RA_NO_SSL=TRUE,RA_WALLET="location=file:<ORACLE_BASE>/admin/<db_name>/wallet/server_seps credential_alias=<string from wallet>",RA_FORMAT=TRUE)';
CONFIGURE DEVICE TYPE 'SBT_TAPE' PARALLELISM <n> BACKUP TYPE TO BACKUPSET;
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE BACKUP OPTIMIZATION ON;
CONFIGURE ARCHIVELOG DELETION POLICY TO SHIPPED TO ALL STANDBY;
CONFIGURE DEFAULT DEVICE TYPE TO SBT_TAPE;
NOTE:
  1. Set _RA_NO_SSL to disable sending backups encrypted unless you are implementing TLS already.
  2. Set RA_FORMAT=TRUE to enable space efficient encrypted backups

Step 6.2 – Perform an Explicit Level 0 Backup

RMAN> CONNECT TARGET /
RMAN> CONNECT CATALOG <string from wallet>
RUN {
 BACKUP INCREMENTAL LEVEL 0 DATABASE FILESPERSET 1 SECTION SIZE 64g PLUS ARCHIVELOG NOT BACKED UP;
 FILESPERSET 32}

Step 6.3 – Schedule Regular Incremental Backups

RMAN> CONNECT TARGET /
RMAN> CONNECT CATALOG <string from wallet>
RUN {
 BACKUP INCREMENTAL LEVEL 1 DATABASE FILESPERSET 1 SECTION SIZE 64G PLUS ARCHIVELOG NOT BACKED UP;
 FILESPERSET 32}