ZDLRA best practices

 ZDLRA best practices (updated 2025)

I have seen a lot of changes with the ZDLRA over the last 10 years or so, and because of that I wanted to document the best practices to follow that evolved up to now (2025).

Because of these changes, I wanted to write a blog post on what customers should think about doing now.

1. Channel Configuration

There are a few items to talk about with channel configuration. First let's break down the pieces of the channel configuration

a) Library

Version

Up until version 19.26, you could use a shared version of the library (libra.so) based on the OS.  Updates to the library could be downloaded directly from MOS.

Starting with version 19.27 on Linux, you must use the version that is stored in $ORACLE_HOME and patched using OPatch.  More information can be found in the MOS note below.

ZDLRA: Where to download the SBT_LIBRARY (libra.so module) (Doc ID 2219812.1)

Location (channel setting)

In the previous section I mentioned that the library is tied to the version of Oracle starting with 19.26. Prior to this release, the default channel setting for the library with the ZDLRA  was

• "PARMS  'SBT_LIBRARY=$ORACLE_HOME/lib/libra.so"

In order to always utilize the current library for the $ORACLE_HOME, the recommendation is to set the channel setting to

• "SBT_LIBRARY=libra.so" - No path, and it will default to the current $ORACLE_HOME/lib"

    Or

• "SBT_LIBRARY=oracle.zldlra" - This will default to the current $ORACLE_HOME/lib/libra.so

b) Environment

In the environment section, you specify the SEPS wallet location. Because there could be conflicts with other oracle products that use a wallet (like OUD), the default location for the SEPS wallet should be set to the WALLET_ROOT location, rather than within the $ORACLE_HOME location.

In the past,  the channel setting for ZDLRA  was

• "RA_WALLET=location=file:${ORACLE_HOME}/dbs/ra_wallet credential_alias={SEPS entry in wallet}"

The recommendation is to set the WALLET_ROOT in the spfile, and store the SEPS wallet within the server_seps directory under this location. If WALLET_ROOT is set to $ORACLE_BASE/admin/{$DB_NAME}/wallet, the setting would be

• "RA_WALLET=location=file:${ORACLE_BASE}/admin/${DB_NAME}/wallet/server_seps/ credential_alias={SEPS entry in wallet}"

c) TLS or non-TLS

If a customer has the ZDLRA configured to encrypt backups with TLS/SSL, the library, by default, will attempt to encrypt the backup using SSL/HTTPS.  If TLS/SSL is configured as optional, and if the client does not have the certificate information, backups will fail. To avoid this, it is recommended to add the following setting to the channel configuration.  This setting will allow you to send backups even if optional TSL/SSL is configured.

• _RA_NO_SSL=TRUE

d) Space Efficient Encrypted Backups

In order to use Space Efficient Encrypted Backups, you must set the following in your channel configuration. This is only available on linux with DB version 19.18+. Keep in mind this setting will only compress datafile backup pieces.  

• RA_FORMAT=TRUE

2. RMAN encryption

One misconception with Space Efficient Encrypted Backups, is that by setting RA_FORMAT=TRUE you are both compressing and encrypting backups. This is not true. You are only compressing the backups. If the tablespace is encrypted with TDE, the backup will remain encrypted, but any non-TDE tablespace backups will be unencrypted.  In order to encrypt the backup pieces (Datafiles, controlfile,. archive logs and spfile) you need to both set an encryption key (if not already set), and set RMAN encryption on.  If you are backing up any tablespaces that are not TDE, you must also ensure you set RA_FORMAT=TRUE.

You should also set the default encryption algorithm to AES256, as most customers use this today rather than the default of AES128

NOTE: Use of RA_FORMAT=TRUE (RMAN Compression) and RMAN encryption is included in the license for ZDLRA and you do not need the ACO or ASO license for backups to the ZDLRA.

3. Real-time redo settings.

a) Default SEPS wallet location

Wallets are becoming more and more common with databases, and this increases the chances that there will be conflict with other Oracle features that also utilize a wallet. To avoid this there is a new default location for the SEPS wallet used by real-time redo. The database will first look in the {WALLET_ROOT}/server_seps location for the wallet.  It is recommended to set the WALLET_ROOT for all databases and use this default location for the SEPS wallet.  Of course you may use links to simplify management.

b) Encrypt Redo

If you have an encryption key set, and you wish to fully encrypt your backups, you also need to ensure that real-time redo is also encrypted. You need to set encryption on the destination for real-time redo

• ENCRYPTION=ON

4. Reserve Space

Reserve space has been difficult to explain, and it has become even more important when implementing a compliance window.  If there is no compliance set, then the reserve space is only interrogated when the ZDLRA runs out of space.  When a compliance window is set, the reserve space is interrogated when backing up to ensure that compliance locked backups fit within the reserved space. If they do not, backups are rejected.  Because of this, I always recommend setting the "auto-tune reserve space" for the policies, especially when setting a compliance window. This setting will automatically adjust the reserve space for you as databases grow.

5. When encrypting backups set "secure mode"

If you need to fully encrypt your backups with RMAN encryption, it is recommended to use "secure mode" on your protection policy. This setting checks all backup pieces (including real-time redo) to ensure that they are RMAN encrypted.  Any unencrypted backup pieces will be rejected.

6. Use racli commands

One of the biggest changes is the increase in the functionality available with racli command vs executing DBMS_RA PL/SQL packages. It is recommended to utilize RACLI commands when possible.  I know when trying to automate onboarding databases, it is much easier to utilize PL/SQL packages.

7. Pair ZDLRAs when using replication

Another change is the ability to pair ZDLRAs that have replication configured. This can be done with the "racli add ra_partner" command.  Using the ZDLRA pairing commands for replication greatly simplifies the configuration of replication. Existing configurations can be converting to using the pairing model.

8. Review types of DB users

There have been a few changes to DB users. There are now 4 types of DB users.

• ADMIN -Used to administer ZDLRA configuration settings
• MONITOR - Read-only account that can view metadata
• REPLICATION - Used to replicate backups between ZDLRAs
• VPC - Used by protected databases to send/received backups
• Insecure=TRUE - VPC user's password will expire, but it can be reset to the same password. This type of user does not support password rollover
• insecure=false - VPC user's password will expire, but this user can leverage password rollover (STIG) to manage password rotation without having backups fail.

NOTE: If you need to rotate password for VPC users you should leverage the STIG/password rollover function that allows you have two passwords associated with the same VPC user as you update wallets.

Summary:

1. Set WALLET_ROOT in the database and store the SEPS in the {WALLET_ROOT}/server_seps directory
2. Ensure the library location in the channel configuration defaults to the current $ORACLE_HOME
3. Set _RA_NO_SSL=TRUE to ensure that converting to TLS/SSL will not cause existing backups to fails
4. Set RA_FORMAT=TRUE to leverage space efficient encrypted backups --> Linux only
5. Enable RMAN encryption, and set algorithm to AES256 --> Linux only
6. Encrypt Real-time redo if applicable
7. If you require fully encrypted backups, set SECURE_MODE on the policy
8. Enable auto-tune of reserved space for all databases, especially those using a compliance window.
9. Use RACLI commands to manage the ZDLRA
10. Pair ZDLRAs when configuring replication
11. Leverage password rollover for VPC users if you require password rotation

Building a Cyber Vault ? Don't forget your keys

When building a cyber vault, one of the most important items to manage is encryption keys.  Encrypting your data is a fundamental pillar of ransomware protection, but encryption key management is often forgotten.  

Ensuring your Cyber Vault has a current copy of your encryption keys associated with your Oracle Databases is critically important for a successful restore and recover after an attack.

Starting with the Oracle Key Vault (OKV) 21.11 release, Oracle Key Vault includes a preview of the Key Transfer Across Oracle Key Vault Clusters Integration Accelerator. You can use this feature to transfer security objects from one OKV cluster to another.

You can find more detail on this feature here, and I will describe it's benefits in this post.

The diagram below shows a typical cyber vault architecture to protect Oracle Databases with the Zero Data Loss Recovery Appliance (ZDLRA) and OKV .

Transferring keys into a cyber vault with OKV

Encryption Key Architecture

Encryption key management is a critical piece of data protection, and it is important to properly manage your keys.  Good cyber protection begins with proper encryption key management.

Local Wallets

With Oracle databases, the default (and simplest) location for encryption keys is in a locally management wallet file.  The keys are often stored in an auto-login wallet, which is automatically opened by the database at startup making key management transparent and simple, but not very secure.

Why aren't wallets secure ?

• They are often auto-login (cwallet.sso) which allows the database to open the wallet without requiring a password. This wallet file gives full access to the encryption keys.
• The wallet file is stored with the database files.  A privileged administrator, such as a DBA has access to both the database and the keys to decrypt the data directly from the hosts.  They also have the ability to delete the wallet file.
• Often the wallet file is backed up with the database, which includes an auto-login wallet.  This allows anyone who has access to backups, to also be able to decrypt the data.
• Securely backing up the wallet file separate from the database is often forgotten, especially when ASM is used as the wallet location.  Not having the wallet file when restoring the database makes restoration and recovery impossible.

Steps you can take

• Create both a passworded wallet (ewallet.p12) and local auto-login wallet (cwallet.sso). With a local auto-login wallet, the wallet can only be opened on the host where the wallet was created.
• Backup the passworded wallet only (ewallet.p12).  The auto-login wallet can always be recreated from the passworded wallet.
• Properly store the password for you passworded encryption wallet. You will need the password to rotate your encryption keys, and create the auto-login wallet.

Oracle Key Vault (OKV)

 The best way to securely manage encryption keys is with OKV.

Why ?

• Keys are managed outside of the database and cannot be accessed locally outside of the database.
• Access to keys is granted to a specific database instance.
• OKV is clustered for High Availability, and the OKV cluster can be securely backed up.
• Key access is audited to allow for early detection of access.

Encryption Keys in a cyber vault architecture

Below is a diagram of a typical cyber vault architecture using ZDLRA.  Because the backups are encrypted, either because the databases are using TDE and/or they are creating RMAN encrypted backups sent to the ZDLRA, the keys need to be available in the vault also.

Not only is the cyber vault a separate, independent database and backup architecture, the vault also contains a separate, independent OKV cluster.

This isolates the cyber vault from any attack to the primary datacenter, including any attack that could compromise encryption key availability.

Encryption Keys in an advanced cyber vault architecture

Below is a diagram of an advanced cyber vault architecture using ZDLRA.  Not only are the backups replicated to a separate ZDLRA in the vault, they are internally replicated to an Isolated Recovery Environment (IRE).  In this architecture, the recover area is further isolated, and the OKV cluster is even further isolated from the primary datacenter. This provides the highest level of protection.

OKV Encryption Key Transfer

This blog post highlights the benefit of the newly released (21.11) OKV feature to allow for the secure transfer of encryption keys.

Periodic rotation of encryption keys is a required practice to protect encryption keys, and ensuring you have the current key available in a cyber vault is challenging.

OKV solves this challenge by providing the ability to transfer any new or changed keys between clusters.

Implementing OKV in a cyber Vault

When building a cyber vault, it is recommend to build an independent OKV cluster.  The OKV cluster in the vault is isolated from the primary datacenter and protected by an airgap.  The nodes in this cluster are  not be able to communicate with the OKV cluster outside of the vault.

The OKV cluster in the vault can be created using a full, secure, backup from the OKV cluster in the primary datacenter. The backup can be transferred into the vault, and then restored to the new, independent OKV cluster providing a current copy of the encryption keys.

Keeping OKV managed keys updated in a cyber Vault

The challenge, once creating an isolated OKV cluster has been keeping the encryption keys within the cluster current when new keys are created.  This was typically accomplished by transferring a full backup of OKV into the vault, and rebuilding the cluster using this backup.

OKV 21.11 provides the solution with secure Encryption Key Transfer.  Leveraging this feature you can securely transfer just the keys that have recently changed allowing you to manage independent OKV clusters that are synchronized on a regular basis.

The diagram below shows the flow of the secure Encryption Key Transfer package from the primary OKV cluster into the vault when the air-gap is opened.

This new OKV feature provides a much better way to securely manage encryption keys in a Cyber Vault.

Summary

As ransomware attacks increase, it is critical to protect a backup copy of your critical database in a cyber vault.  It is also critical to protect a copy of your encryption keys to ensure you can recovery those databases.  OKV provides the architecture for key management in both your primary datacenter and in a cyber vault.  The new secure key transfer feature within OKV allows you to synchronize keys across independent OKV clusters.

Oracle database wallets for TDE, ZDLRA and External Authentication

 One topic that I spend a lot of time on is "wallets" and the Oracle database. When working with multiple features in the database, there are multiple wallets that are used for different purposes. Along with multiple wallets, there are 2 ways to manage wallets (mkstore and orapki), and there a multiple types of wallets, passworded, auto-login and local.

Wallet Use Cases

Below is a subset of all the places where wallets are used.  

Encryption Wallet : This wallet contains the encryption keys used by DBMS_CRYPTO, TDE and/or RMAN encrypted backups

Strong authentication: Often when external authentication is configured in the database, each database has unique certificates that are stored in a wallet.  In this blog I will refer to this as Strong Authentication. This covers all of DB authentication terms. EUS, OUD, Kerberos, RADIUS, etc.

Certificate authorities and Self-signed certificates : These are used by the database to establish external calls to websites using SSL (HTTPS).  The database can validate the certificate with an external certificate authority, or the self-signed certificate can be stored directly in the wallet.

SEPS authentication : SEPS authentication is used by Oracle clients (including the ZDLRA) to allow scripts to authenticate with a username and password that is stored in an auto-login wallet.  The connection string to the DB is used as the key to retrieve the encrypted connection information.

Real-time redo and TLS certificates for ZDLRA : When the ZDLRA is configured to utilize HTTPS for send/receiving backups, a self-signed certificate is stored in a wallet. This is the same wallet that is used for SEPS authentication of the VPC user.

You can imagine the confusion when you try to combine multiple products that use a wallet, and you want to manage those wallets separately.

Encryption Wallet 

The encryption wallet is the easiest wallet to manage because it is typically isolated from the other wallets that are in use.

The hierarchy Oracle uses to find the location of an encryption wallet is below.  It follows this hierarchy and it will use the first wallet it finds.

WALLET_ROOT : This the recommended location for the encryption wallet as of 19c. The WALLET_ROOT is a spfile/pfile setting that allows you to specify a different location for each database.  It is recommended that the wallet is stored under $ORACLE_BASE/admin/{DB name}/wallet on each node to allow for out of place upgrades.

ENCRYPTION_WALLET_LOCATION in the sqlnet.ora : This was the recommended location prior to 19c.  When multiple databases were sharing the same $ORACLE_HOME (and thus the same sqlnet.ora file), this became confusing. The workaround was to set the location using a variable representation of the DB_NAME.  

$ORACLE_BASE/admin/{DB name}/wallet : This location is the recommended location, But you should set the WALLET_ROOT, or in on an old release (less than 19c)  set the ENCRYPTION_WALLET_LOCATION.  Depending on this location to be the "default" location can cause issues when you start using a wallet for other purposes.  This same location is the default location for any Strong authentication implementations.

Since you should be on 19c, you should be using WALLET_ROOT for encryption wallet location.

NOTE: If you running databases in OCI, it is mandatory to be using WALLET_ROOT in order to utilize the recovery service.

Recommendation :

My recommendation is to always use OKV to manage TDE encryption keys, but I understand that it is a licensable product and it isn't feasible to expect that all customers are using it.

When working in a RAC environment (non-OKV) it becomes critical to have a shared TDE wallet. You may be tempted to store the wallet on ASM, or Exascale. I recommend that you DO NOT.  This makes it much more difficult to backup the wallet, and it makes it more difficult to have a shared SEPS wallet if backing up to a ZDLRA.

Store the TDE encryption wallet on ACFS, and point the WALLET_ROOT to the ACFS location mounted on each node.  When backing up the encryption wallet, copy ONLY the passworded wallet ewallet.p12 to another location to be backed up outside of the DB backups.

Strong authentication wallets

This wallet typically causes the most headaches for users.  The hierarchy Oracle uses to find the location of a Strong authentication wallet is below.  Like the encryption wallet, it follows this hierarchy and it will use the first wallet it finds.

WALLET_LOCATION in the sqlnet.ora : When multiple databases are sharing the same $ORACLE_HOME (and thus the same sqlnet.ora file), this becomes confusing. The workaround was to set the location using a variable representation of the DB_NAME as part of location string.  

$ORACLE_BASE/admin/{DB name}/wallet : This is the location that most customers place their Strong authentication wallets in since it is isolated to the Database associated with the wallet

NOTE: The issue arises when customers use a product/feature that updates the WALLET_LOCATION in the sqlnet.ora, which breaks authentication since the WALLET_LOCATION is checked first.

Use separate wallets, and leverage the TNS_ADMIN variable to point to different sqlnet.ora files and sharing the same $ORACLE_HOME.

Certificate authorities and Self-signed certificates 

The most common use case for certificate authorities is when  utilizing the DBMS_CLOUD family of products.  Products such as DBMS_CLOUD call out object storage and require a secure (HTTPS) connection. In order to open a secure connection the client needs to authenticate the certificate as  valid certificate, or use a self-signed certificate that is stored in the wallet.

This same issue is true when using DBMS_CLOUD_AI and DBMS_VECTOR_CHAIN which makes calls to external LLMs that often require a secure connection.

This wallet is controlled by setting the database property "SSL_WALLET". 

For simplicity I would recommend creating a central wallet that can be used by ALL databases on the host and is stored within $ORACLE_BASE. My favorite location is $ORACLE_BASE/cert_wallet which identifies it as containing certificate authorities.

I do not recommend adding certificates to the Strong authentication wallet, or the SEPS wallet (discussed next) as it becomes more difficult to mange multiple wallets to make updates.

SEPS authentication 

The next wallet I want to discuss is the SEPS authentication wallet. This wallet is used by Oracle clients (sqlplus, RMAN, and ZDLRA) to store the credentials for a database.

The connection string (either an ezconnect string or a tnsnames.ora entry) is added to the wallet, along with the username and password that will be used when connecting using this entry.  

The location of the wallet is stored in the sqlnet.ora file, and there are 2 parameters associated with this setting.

SQLNET.WALLET_OVERRIDE=true

WALLET_LOCATION={location on disk}

NOTE: Setting the WALLET_OVERRIDE to true disables any OPS$ usage and allows the usage of SEPS wallets for authentication. 

Setting the WALLET_LOCATION on a host that supports databases utilizing Strong authentication often causes issues if it does not specify a separate location each database using variable.  The sqlnet.ora file is only read at startup, so changes to the WALLET_LOCATION might not become apparent to after a database bounce.

Recommendation :

If you are using multiple products that use a wallet AND share the same Oracle Home, I recommend using the TNS_ADMIN variable to mange which wallet to use in scripts. 

As wallets become more common for security, separating out the use cases, if possible, will make it easier to manage and rotate authentication information.  With TNS_ADMIN you can point to a directory containing a sqlnet.ora file specific to the database, and leave the original sqlnet.ora file without a WALLET_LOCATION entry. 

Real-time redo and TLS certificates for ZDLRA 

Prior to the 19.18 DB release, configuring real-time redo for databases sending backups to the ZDLRA required a bounce of the database (to refresh the DBs copy of the sqlnet.ora), and it required the WALLET_LOCATION to be set in the sqlnet.ora.

This changed with 19.18, and I recommend you use the new location.

   The hierarchy Oracle uses to find the location of the wallet real-time wallet is below.  Like the encryption wallet, it follows this hierarchy and it will use the first wallet it finds.

WALLET_ROOT/server_seps : If the variable WALLET_ROOT is set, and a wallet exists in the server_seps subdirectory, that wallet is used by the real-time redo.  This is a HUGE improvement as it doesn't require a bounce, and it makes it much easier to avoid issues with Strong authentication, and databases that share the same $ORACLE_HOME.

NOTE: WALLET_ROOT was added in 18c. If you are still using 12.x, you need to use the sqlnet.ora.

WALLET_LOCATION in the sqlnet.ora : When multiple databases are sharing the same $ORACLE_HOME (and thus the same sqlnet.ora file), this becomes confusing. The workaround was to set the location using a variable representation of the DB_NAME.  This is what I mentioned for Strong authentication.

Recommendation :

When backing up to a ZDLRA, especially with real-time redo you should be using a SEPS wallet that is stored under WALLET_ROOT.  

Since the ZDLRA supports encrypted backups, even if you don't own ASO, I recommend creating an encryption wallet with keys to encrypt your backups.  This is much more secure, and this ability is included in the ZDLRA license.

The steps I would recommend for any customer using the ZDLRA are

• If you don't have an encryption wallet (because you don't own ASO), create one and set the  encryption keys for both the CDB and PDB (if it is multi-tenant). This does require a DB bounce to set the WALLET_ROOT, but this will allow you to have RMAN encrypted backups.
• In a RAC environment store the encryption wallet on ACFS and point WALLET_ROOT to the ACFS location.
• Store the SEPS wallet containing the VPC user credentials for the ZDLRA in the WALLET_ROOT/server_seps directory.  This will automatically be used by real-time redo starting with 19.18.
• Ensure your channel configuration for RMAN points to the WALLET_ROOT/server_seps directory on ACFS for the wallet.
• In your RMAN scripts ensure that you are pointing to a TNS_ADMIN location that has a sqlnet.ora file pointing to the WALLET_ROOT/server_seps location for WALLET_LOCATION or ensure that OEM has the correct SEPS wallet location set. 

MKSTORE vs ORAPKI

orapki 

The orapki utility manages public key infrastructure (PKI) elements, such as wallets and certificate revocation lists, from the command line.  This is the recommended method of managing wallet files.

You can use the orapki command-line utility to perform the following tasks:

• Creating and viewing signed certificates for testing purposes

• Manage Oracle wallets (except for Transparent Data Encryption keystores):

  • Create and display Oracle wallets

  • Add and remove certificate requests

  • Add and remove certificates

  • Add and remove trusted certificates

• Manage certificate revocation lists (CRLs):

  • Renaming CRLs with a hash value for certificate validation

  • Uploading, listing, viewing, and deleting CRLs in Oracle Internet Directory

NOTE: The above is directly from the 19c documentation.  You can see that orapki is used to manage certificates with no mention of managing SEPS credentials.

mkstore

The first thing you will notice with mkstore, is that the mkstore command should be considered deprecated.  Upon digging into this some more, I found a comment from Russ Lowenthal (VP of Database Security products) who mentions that the SEPS credential wallet management will not be added to orapki until AFTER 23c.

NOTE: Even though it is considered deprecated, mkstore is the only way to manage SEPS credentials from the command line, and should only be used to manage SEPS credentials.

Administer key management

I added the "Administer Key Management" command to this section because it can also be used to manage both secrets and SEPS credentials.

The following options are available and can be found in the documentation.

• add/update/delete Secret '{secret name}' for client '{client identifier}' --> secret
• add/update/delete secret '{secret name}' for client '{client identifier}' to {local optionally} auto_login keystore {keystore location}  --> SEPS

How to manage wallets

Wallet Type	How to manage contents
Encryption Keys	Utilize the "ADMINISTER KEY MANAGEMENT" statement from the database
External user authentication	Use orapki to manage certificates, or the OWM tool which uses orapki
Certificate authorities and Self-signed certificate	Use orapki to manage certificates
SEPS authentication	Use mkstore for now, as orapki does not support SEPS
Real-time redo for ZDLRA	Use mkstore for now, as orapki does not support SEPS
TLS certificates for ZDLRA	Use orapki to manage the certificates

Wallet names and type

When you look in the wallet directory you would see one, or both of these wallets.

cwallet.sso - This is an auto-login wallet.  With an auto-login wallet you can access the contents without having to provide a password. In almost all cases, you will have this type of wallet entry.

ewallet.p12 - This is the passworded wallet. In order add/change/delete entries you need to specify a password when making those changes.  

NOTE:

• If only the cwallet.sso exist, you can assume it is an auto-login only wallet.
• If both wallets exist, you can access the contents without a password, but any add/change/deletion commands will require a password and update both the passworded wallet and the auto-login wallet.
• If only the ewallet.p12 exists, to access the contents of the wallet  you must provide a password.

Standard Password Protected wallet

This is the least common wallet type (at least alone without an auto-login wallet), since it requires a password to access the contents. This is most commonly used to protect encryption keys for databases since it will require entering password to open the wallet when the database is started.   In this configuration you create a new wallet using orapki or Administore key store and provide a password.  In this case there will only be a single wallet file, ewallet.p12.

NOTE: You cannot create a non auto-login wallet with mkstore 

• orapki wallet create -wallet {wallet location}
• administer key management create keystore {wallet location}

Auto-login only wallets

You can create an auto-login wallet using e mkstore,  orapki, or the administer key manage command.  The idea of an auto-login wallet, is that you can add entries to this wallet without needing a password. You can also list the entries in the wallet using either CLI tool. In this configuration there is only a cwallet.sso file in the wallet directory

Auto-login wallets

This is the most common configuration that you will see.  There is both a passworded wallet, and an auto-login wallet. With both wallets, it requires a password to make changes, but no password is required to open the wallet and use it.  The two wallets are synchronized when you make changes.

There are two ways to create auto-login wallets.

    1. Create a non auto-login wallet using orapki or within the database, then create an auto-login wallet from the non auto-login wallet.

• orapki wallet create -wallet {wallet location}
• orapki wallet create -wallet {wallet location} -auto_login  OR
• mkstore -wrl {wallet location} -createSSO
• administer keystore create keystore {wallet location}
• administer keystore create auto_login keystore from keystore {wallet location}

     2. Create an auto-login wallet  and non auto-login wallet together

•     orapki wallet create -wallet {wallet location} -auto_login

Local Auto-login wallets

Local auto-login wallets work the same way as the auto-login wallet, EXCEPT, the wallet is encrypted in a way that makes it only usable on the host it was created.  This limits any security risks if the wallet is copied (or restored) onto a different host.

When creating a local auto-login wallet you would use 

• mkstore -wrl {wallet location} -createLSSO
• orapki wallet create -wallet {wallet location} auto_login_local
• administer keystore create local auto_login keystore from keystore {wallet location}

NOTE:

• Local auto-login wallets are much more secure as they can only be used on the host  where the wallet was created. 
• When backing up wallets, this includes Encryption wallets, only backup the ewallet.p12 file.  This ensures that a password is required to utilize the wallet.

NOTE: When only backing up the ewallet.p12, be sure you know the password so that you can recreate the auto-login wallet.

• ALWAYS review the permissions on your wallet files, especially the auto-login wallet files containing credentials.  Any user that can access the auto-login wallet file can utilize the credentials contained within the wallet.

ASM/Exascale for Encryption wallets

You probably noticed that I am not a fan of ASM/Exascale as an encryption wallet location, even though ASM in mentioned in the documentation. 

I will add more to this section, but this is my reasoning for not preferring ASM.

1. It's easy to forget backing up the wallet file.  Having it on ASM requires copying it back to the file system to get backed up.  It is very easy to forget about this, rotate the keys, and not have a wallet backup.
2. WALLET_ROOT is becoming the starting point for different wallet files, not just encryption wallets.  ZDLRA is the first example. When WALLET_ROOT points to ASM or Exascale, then the same wallet cannot be used by many tools because they only expect wallets on the file system.

Shared wallets make sense, that's why I prefer ACFS, or a mounted filesystem for WALLET_ROOT.

Summary 

Starting with DB 19.18, you have the ability to store individual credential wallets for real-time redo transportation when leveraging ZDLRA for backups.  You can also use the TNS_ADMIN variable to set a different location when using SEPS authentication.  It is now possible to manage multiple wallets separately without having conflicts between products and features.

MY RECOMMENDATIONS (summary):

• Use Oracle Key Vault (OKV) for encryption keys.  OKV is an Oracle product specifically designed to securely store and manage encryption keys, and much more.  OKV has tight integration with the Oracle Database.  If you are not using OKV, at least store Encryption Keys on ACFS as the shared location (not ASM or Exascale).

• Use WALLET_ROOT if you are on 18c+.  This will continue to be used products to help separate wallet locations for different uses cases.  The ZDLRA is the first of many products to use the hierarchy for wallet files. 

• Backup only the ewallet.p12.  This is the passworded wallet and with the password it can be used to recreate the auto-login wallet. This is especially critical for Encryption keys.

BUT - Make sure you know the password. Without the password, you can't recreate the auto-login wallet.

• Lock down permissions on wallet files to only the account that needs access, especially the cwallet.sso file (auto-login).

• Whenever possible create local auto-login wallets that can only be used on the source host where the wallet was created. This wallet, however,  cannot be shared across nodes.

• Keep your SEPS wallets separate by utilizing the TNS_ADMIN variable and having a custom sqlnet.ora file.

• If you are backing up to a ZDLRA create an encryption wallet with keys, and set the WALLET_ROOT location.  Put the SEPS wallet for ZDLRA under WALLET_ROOT/server_seps.  This wallet can also be used for the TPCS certificate if you configure HTTPS.   Keep this configuration separate to avoid conflicts with other products.

ZDLRA backups -- How do I know if they are Encrypted

 The ZDLRA introduced a new feature with release 23.1 that can both encrypt backups (if they are not already encrypted from TDE) and  compress the backups .  The combing of both encryption and compression with this feature is unique to the ZDLRA.

I talked about this new exciting feature in a blog post on Oracle.com you can find here.

What I am am going to cover in this blog post is how to audit the RMAN catalog on the ZDLRA to validate that your backups are completely RMAN encrypted.

There are two big advantages of ensuring your backups are fully encrypted

1) With the prevalence of data exfiltration, and the advent of new regulations in many industries,  full encryption of backups is mandatory

2) When sending a backup to the Oracle cloud (either in OCI or to object storage on ZFS) full encryption is required to protect the backup data.

The question I often get asked with this feature is..

 "How do you tell  if your backups are encrypted ?"

You can can determine that your backups are encrypted by looking at the RMAN catalog.

The RC_BACKUP_PIECE view contains a column identifying if the backup is encrypted.  This column is set to "YES" only when the backup piece is encrypted.

Keep in mind that there multiple types of backups pieces contained in the catalog

• Controlfile backups
• Spfile backups
• Archive log sweeps
• Archive log backups from real-time redo
• Datafile backups
• Virtual Full backups created from incremental backups.

All of these backups except for two are sent from RMAN with "encryption on" and the backup set will marked as encrypted based on the RMAN encryption setting.

The two that are not set by RMAN directly are

• Real-time redo backups. Real-time redo backups are identified in the RMAN catalog as encrypted when the destination setting on the protected database has ENCRYPTION=ENABLE set.
• Virtual Full backups.  Virtual full backups are identified, for each datafile backup set, as encrypted ONLY after a new L0 is taken with RMAN encryption on, and all subsequent L1 backups are encrypted.  I know that is a lot of stipulations on identifying the virtual full backup as encrypted.  Only when a new FULL encrypted backup is taken, and all future incremental backups are encrypted can the ZDLRA be sure the backup has remained completely encrypted.

Checking the catalog

  The script below takes 2 parameters (&db_name, and &days_to_compare) and it will check the RMAN catalog and display the status of the backups, by backup type making it easier to identify any backup pieces that may not be encrypted.

set linesize 170; set pagesize 100 ALTER SESSION SET NLS_DATE_FORMAT = 'YYYY-MM-DD hh24:mi:ss'; define db_name =MYDB define days_to_compare = 7 col backup_count format 99999 heading 'Backup|pieces' col encrypted format a10 heading 'Encrypted | Yes or No' col compressed format a10 heading 'Compressed | Yes or No' col backup_type_known format a40 heading 'Backup piece type' col start_time format a25 heading 'Start Time | by Day' col backup_size format 999,999.99 heading 'Backup Size | by Day (GB)' set pagesize 150 set linesize 150 ttitle skip 2 CENTER 'Database backup summary for last &days_to_compare days database: &db_name' SKIP 2 set feedback off set echo off set underline = set verify off set termout off spool database_report_&db_name.out select Encrypted, compressed, count(1) backup_count, backup_type_known from ( select case when (backup_type = 'L' and handle like '$RSCN%') then 'Archive Log - real-time redo' when (backup_type = 'L' and handle not like '$RSCN%') then 'Archive Log - log sweep' when (backup_type = 'D' and handle like 'c-%') then 'Controlfile/SPFILE backup' when (backup_type = 'D' and incremental_level=0 and handle like 'VB$%') then 'Virtual Full backup' when (backup_type = 'I' and incremental_level=1 and handle like 'VB$%') then 'Incremental L1 backup' when (backup_type = 'I' and incremental_level=1 and handle not like 'VB$%') then 'Incr. L1 --> Not yet virtualized' else backup_type || ' ' || incremental_level || ' ' || media end as backup_type_known, to_char(START_TIME,'mm/dd/yy hh24:mi:ss') Start_time, Encrypted, compressed from rc_backup_piece_details where start_time > sysdate - &days_to_compare and db_name = '&db_name' and (media is null or media like 'Recovery Appliance%') order by backup_type_known,encrypted,compressed ) group by encrypted,backup_type_known,compressed;

This provides a nicely formatted output as you can see below.

                                             Database backup summary for last 15 days database: DBSG23AI

Encrypted  Compressed Backup
 Yes or No  Yes or No pieces Backup piece type
========== ========== ====== ========================================
YES        YES            69  Full backup
YES        NO             39 Archive Log - log sweep
NO         YES             1 Incremental L1  backup
YES        NO           3958 Archive Log - real-time redo
YES        YES            67 Incremental L1  backup
NO         YES             3  Full backup
NO         NO              1 Controlfile/SPFILE backup
YES        NO             26 Controlfile/SPFILE backup
YES        NO            221 Incremental L1  backup

In the report you can see that there a  few backups that not encrypted, along with some controlfile/spfile backups.

NOTE: In order to run this report, I created a REPORT user in the database on the ZDLRA as an "monitor" user.. A report has enough permissions to create this report.

OKV and ZDLRA 

Previously when sending backups to Cloud (which included OCI object storage on ZFSSA), OKV was required. When using Space Efficient Encrypted backups, you can ensure that EVERY backup piece is fully encrypted and RMAN recognizes them as encrypted.

If you follow the information in the blog, and what I have posted in the past, you will no longer need to configure OKV when sending backups to the cloud. 

If all backup pieces are encrypted, and the RMAN catalog reports that all backup are encrypted, you can create backups using DBMS_RA.CREATE_ARCHIVAL_BACKUP setting the "encryption_algorithm" to "CLIENT" or "ENC_CLIENT". This will tell the ZDLRA not utilize OKV to encrypt backups, but if any backup pieces are NOT encrypted, the archival backups will fail.

Oracle Backup Compression and Encryption layers explained

 When working with customers who are applying compression and/or encryption to their Oracle DB backups, I found that it isn't always clear if backups are compressed or encrypted, or both. In this blog post I will break compression and encryption of Oracle backups down into the levels where these operations could occur.  Below is a high level view of these 3 levels.

Database

Compression

Data in the database can be compressed in any one of the following formats or all of the formats

HCC - Available only on Exadata, or ZFS storage, this compression is a columnar compression format with different options that allow you to choose the appropriate access speed and compression ratio for your data

Advanced Compression - A licensable option that will automatically compress data in the background to optimize storage without compromising performance.

Basic Compression - Requires a lock on object during insert and is typically used within a data warehouse.

 External Compression - In some cases the data stored in the database may already be compressed externally. An example of this is image files which are already stored in a compressed format.

 

Encryption

Data in the database can be encrypted in any one of the following formats or all of the formats

TDE - All data in the tablespace is encrypted by database.

Column Encryption - Specific data within a column is encrypted, SSN for example.  This is less widely used and most customers use TDE instead.

 External Encryption - In some cases the data stored in the database may already be encrypted by the application.

 

 NOTE: 

1. If the data is compressed and/or encrypted in these manors it will continue in that format when backed up.  

• Any data that encrypted in the database will remain encrypted in the backups
• Any data that is compressed in the database will remain compressed in the backups
• Backups of data that is compressed and/encrypted will get little to no compression when backed up

2. RMAN does not know that the data  is either compressed or encrypted, and querying the RMAN views will not tell you that either has occurred.

3. Having data encrypted and/or compressed in the database may not stop you from further compressing and/or encrypted the backups.

ZDLRA

Compression

Datafile Compression - With Datafile compression you have 2 choices to compress the backups

• RA_FORMAT = TRUE - This  compresses all datafile backups in the new ZDLRA 23.1 format.  If the datafile is part of a TDE tablespace, the blocks will be decrypted prior to compression to ensure the best compression ratio.  
• RA_FORMAT not set or  FALSE - Backups of datafiles will be sent as uncompressed (unless you create a RMAN compressed backupset which the ZDLRA will uncompress before ingesting).  Once they are received on the ZDLRA they will be compressed in storage on ZDLRA.  When replicated to another ZDLRA, or restored, they are uncompressed.

Real-time Redo Compression - When sending real-time redo to the ZDLRA you can have the ZDLRA create an RMAN compressed backupset for the archive logs.  The level of compression can be set on the policy.  Once stored in an RMAN compressed backupset format, it is replicated and restored as a compressed backupset.  

          NOTE: If the Redo stream contains changes to a TDE tablespace, or you are                                            configuring encryption on the RA as destination, you may get little to no actual compression 

SPFILE, Controfile, archivelog backups - The ZDLRA will NOT attempt to compress these backupsets internally.  Only datafile backups are compressed on the ZDLRA.

 

Encryption

Datafile Encryption - Whether a datafile is encrypted by the ZDLRA in the new ZDLRA 23.1 format depends on these 2 conditions.

• RA_FORMAT = TRUE and "RMAN Encryption ON" - If the datafile is NOT part of a TDE tablespace, this will force BOTH compression and encryption of that datafile backup.
• RA_FORMAT = TRUE and "RMAN Encryption OFF" - If the datafile is part of a TDE tablespace, the backup of this datafile will remain encrypted.  If the datafile is NOT part of a TDE tablespace, the backup will NOT be encrypted.

Real-time Redo Encryption - If real-time redo is utilized and your database has implemented TDE, the change data in the archive log backups will be encrypted.  However, this backup is not considered RMAN encrypted, and ENCRYPTION=ENABLED must be set on the destination definition to ensure that the real-time redo backupsets are considered fully encrypted by RMAN.

SPFILE, Controlfile, archivelog backup Encryption - These are not encrypted by the ZDLRA.

 

 NOTE: 

1. The new Space Efficient Encrypted backup feature of the ZDLRA only affects datafile backups.

2. Real-time redo backups can be compressed and/or encrypted by the ZDLRA.

3. If you are using the new RA_FORMAT=TRUE for non-TDE datafile backup, you will only get a compressed a backupset.  You have set RMAN Encryption on with RA_FORMAT=TRUE in order to encrypt the backupset.

4. If you are backing up a non-TDE  datafile, and wish to encrypt it with the library, it will also be compressed.  You cannot encrypt without compression, but you can compress without encryption.

5. If datafile backups are sent to the ZDLRA  without RA_FORMAT=TRUE, they will appear as compressed in the RMAN catalog.  With RA_FORMAT=TRUE they will not appear as compressed.

6.If real-time redo is sent to the ZDLRA, and the profile for the database is set to compress the archivelogs, they will appear as compressed in the RMAN catalog. 

 

RMAN

Compression

Datafile Compression - With Datafile compression you have 2 choices to compress the backups

• RA_FORMAT = TRUE - RMAN compression is ignored when this option is set.  
• RA_FORMAT not set or  FALSE - RMAN can create a compressed backupset for datafiles.  If the datafile is part of a TDE tablespace, this datafile will not be able to create a virtual full.  If the Datafile is NOT part of a TDE tablespace, the backset will be decompressed on the ZDLRA and will not be stored as a Compressed backupset.

SPFILE, Controfile, archivelog backups - The ZDLRA will NOT attempt to compress these backupsets internally.  They remain compressed.

 

Encryption

Datafile Encryption - RMAN Encrypt ON  creates an Encrypted backupset which cannot be virtualized by the ZDLRA.  This should only be set when using RA_FORMAT=TRUE which bypasses RMAN encryption

SPFILE, Controlfile, archivelog backup Encryption - These can be encrypted by setting RMAN Encryption on.

 NOTE: 

1. The new Space Efficient Encrypted backup feature of the ZDLRA only affects datafile backups.

2. Real-time redo backups can be compressed and/or encrypted by the ZDLRA.

3. If you are using the new RA_FORMAT=TRUE for non-TDE datafile backup, you will only get a compressed a backupset.  You have set RMAN Encryption on with RA_FORMAT=TRUE in order to encrypt the backupset.

4. If you are backing up a non-TDE  datafile, and wish to encrypt it with the library, it will also be compressed.  You cannot separate encryption from compression, but you can compress only.

ZDLRA's space efficient encrypted backups with TDE explained

 In this post I will explain what typically happens  when RMAN either compresses, or encrypts backups and how the new space efficient encrypted backup feature of the ZDLRA solves these issues.

TDE - What does a TDE encrypted block look like ?

In the image above you can see that only the data is encrypted with TDE.  The header information (metadata) remains unencrypted.  The metadata is used by the database to determine the information about the block, and is used by the ZDLRA to create virtual full backups.

Normal backup of TDE encrypted datafiles

First let's go through what happens when TDE is utilized, and you perform a RMAN backup of the database.

In the image below, you can see that the blocks are written and are not changed in any way. 

NOTE: Because the blocks are encrypted, they cannot be compressed outside of the database.  

Compressed backup of TDE encrypted datafiles

Next let's go through what happens if you perform an RMAN backup of the database AND tell RMAN to create compressed backupsets.  As I said previously, the encrypted data will not compress., and because the data is TDE the backup must remain encrypted.

Below you can see that RMAN handles this with series of steps.  

RMAN will

1. Decrypt the data in the block using the tablespace encryption key.
2. Compress the data in block (it is unencrypted in memory).
3. Re-encrypt the whole block (including the headers) using a new encryption key generated by the RMAN job

You can see in the image below, after executing two RMAN backup jobs the blocks are encrypted with two different encryption keys. Each subsequent backup job will also have new encryption keys.

Compression or Deduplication

This leaves you with having to chose one or the other when performing RMAN backup jobs to a deduplication appliance.  If you execute a normal RMAN backup, there is no compression available, and if you utilize RMAN compression, it is not possible to dedupe the data. The ZDLRA, since it needs to read the header data, didn't support using RMAN compression.

How space efficient encrypted backups work with TDE

So how does the ZDLRA solve this problem to be able provide both compression and the creation of virtual full backups?

The flow is similar to using RMAN compression, BUT instead of using RMAN encryption, the ZDLRA library encrypts the blocks in a special format that leaves the header data unencrypted.  The ZDLRA library only encrypts the data contents of blocks.

1. Decrypt the data in the block using the tablespace encryption
2. Compress the data in block (it is unencrypted in memory).
3. Re-encrypt the data portion of the block (not the headers) using a new encryption key generated by the RMAN job

In the image below you can see the flow as the backup is migrating to utilizing this feature.  The newly backed up blocks are encrypted with a new encryption key with each RMAN backup, and the header is left clear for the ZDLRA to still create a virtual full backup.

This allows the ZDLRA to both compress the blocks AND provide space efficient virtual full backups

How space efficient encrypted backups work with non-TDE blocks

So how does the ZDLRA feature work with non-TDE data ?

The flow is similar to that of TDE data, but the data does not have to be unencrypted first.  The blocks are compressed using RMAN compression, and are then encrypted using the new ZDLRA library.

In the image below you can the flow as the backup is migrating to utilizing this feature.  The newly backed up blocks are encrypted with a new encryption key with each RMAN backup, and the header is left clear for the ZDLRA to still create a virtual full.

I hope this helps to show you how space efficient encrypted backups work, and how it is a much more efficient way to both protect you backups with encryption, and utilize compression.

NOTE: using space efficient encrypted backups does not require with the ACO or the ASO options.

Oracle Recovery Service now offers retention lock

 Oracle DB Recovery Service recently added a new feature to protect backups from being prematurely deleted, even by a tenancy administrator.  This new feature adds a retention lock to the Backup Retention Period at the policy level. The image below shows the new settings that you see within the protection policy.

Enabling retention lock

The recovery service comes with some default policies that appear as "oracle defined" policy types

Name            Backup retention period

Platinum            46 days

Gold                   65 days

Silver                 35 days

Bronze               14 days

These policies can't' be changed, and they do not enable retention lock.

In order to implement a retention lock you need to create a new protection policy or  update an existing user defined protection policy.

Step #1 Set/Adjust "Backup retention period"

If you are creating a new "user defined" protection policy, you need to set the backup retention to a number of days between 14 and 95.  You should also take this opportunity to adjust the backup retention of an existing policy, if appropriate, before it is locked.

NOTE: Once a retention lock on the protection policy is activated (discussed in step #3), the backup retention period cannot be decreased, it can only be increased.

Step #2 Click on "enable retention lock"

This step is pretty straightforward. But the most important item to know is that the retention lock is not immediately in effect.  Much like the "retention lock" that is set on object storage, there is a minimum period of at least 14 days before the lock is "active".

 Note: Once the grace period has expired for the policy (explained later in this blog post) the  "retention lock"  is permanent and cannot be removed.

Step #3 Set "Scheduled lock time"

As I said in the previous step, the lock isn't immediately active. In this step you set the future date/time  that the lock time becomes active, and this Date/Time must be at least 14 days in the future.  This provides a grace period that delays when the lock on the policy becomes active. You have up until the lock activation date/time to adjust the scheduled lock time further into the future if it becomes necessary to further day lock activation.

Grace Period 

I wanted to make sure I explain what happens with this grace period so that you can plan accordingly.

• If you change an existing "user defined" policy to enable the retention lock, any databases that are a member of this policy will not have locked backups until the scheduled lock date/time activates the lock.  

• If you add databases to a protection policy that has a retention lock enabled, the backups will not be locked until whichever time is farther in the future.
• Scheduled lock time for the policy if the retention lock has not yet activated.
• 14 days after the database is added to the protection policy.

• Databases can be removed from a retention locked protection policy during this grace period.

• If the policy itself is still within it's grace period from activating, the backup retention period can be adjusted down for the protection policy.

NOTE: This 14 day grace period allows you to review the estimated space needed.  On the protected database summary page, for each database, you can see the "projected space for policy"  in the Space Usage section.  This value can be used to estimate the "locked backup" utilization.

What happens with a retention lock ?

Once the grace period expires the backups for the protected database are time locked and can't be prematurely deleted.  

The backups are protected by the following rules.

1. The database cannot be moved to another policy. No user within the tenancy, including an administrator can remove a database from it's retention enabled policy.  If it becomes necessary to move a database to another policy , an SR needs to raised, and security policies are followed to ensure that this is an approved change.

2.  There is always a 14 day grace period in which changes can be made before the backups become locked. This is your window to verify the backup storage usage required before the lock activates.

3. Even if you check the "72 hour termination option" on the database, backups are locked throughout the retention window.

Comments:

This is a great new feature that protects backups from being deleted by anyone in the tenancy, including tenancy administrators.  This provides an extra layer of security from an attack with compromised credentials.  Because the lock is permanent, always use the 14 day grace period to ensure the usage and duration is appropriate for you database.