Oracle database wallets for TDE, ZDLRA and External Authentication

 One topic that I spend a lot of time on is "wallets" and the Oracle database. When working with multiple features in the database, there are multiple wallets that are used for different purposes. Along with multiple wallets, there are 2 ways to manage wallets (mkstore and orapki), and there a multiple types of wallets, passworded, autologin and local.

Wallet types

Encryption Wallet : This wallet contains the encryption keys used by DBMS_CRYPTO, TDE and/or RMAN encrypted backups

External user authentication: Often when external authentication is configured in the database, each database has unique certificates that are stored in a wallet.  In this blog I will refer to this as Enterprise User Security (EUS).

Certificate authorities and Self-signed certificates : These are used by the database to establish external calls to websites using SSL (HTTPS).  The database can validate the certificate with an external certificate authority, or the self-signed certificate can be stored directly in the wallet.

SEPS authentication : SEPS authentication is used by Oracle clients (including the ZDLRA) to allow scripts to authenticate with a username and password that is stored in an autologin wallet.  The connection string to the DB is used as the key to retrieve the encrypted connection information.

Real-time redo and TLS certificates for ZDLRA : When the ZDLRA is configured to utilize HTTPS for send/receiving backups, a self-signed certificate is stored in a wallet. This is the same wallet that is used for SEPS authentication of the VPC user.

You can imagine the confusion when you try to combine multiple products that use a wallet, and you want to manage those wallets separately.

Encryption Wallet 

The encryption wallet is the easiest wallet to manage because it is typically isolated from the other wallets that are in use.

The hierarchy Oracle uses to find the location of an encryption wallet is below.  It follows this hierarchy and it will use the first wallet it finds.

WALLET_ROOT : This the recommended location for the encryption wallet as of 19c. The WALLET_ROOT is a spfile/pfile setting that allows you to specify a different location for each database.  It is recommended that the wallet is stored under $ORACLE_BASE/admin/{DB name}/wallet on each node to allow for out of place upgrades.

ENCRYPTION_WALLET_LOCATION in the sqlnet.ora : This was the recommended location prior to 19c.  When multiple databases were sharing the same $ORACLE_HOME (and thus the same sqlnet.ora file), this became confusing. The workaround was to set the location using a variable representation of the DB_NAME.  

$ORACLE_BASE/admin/{DB name}/wallet : This location is the recommended location, But you should set the WALLET_ROOT, or in on an old release (less than 19c)  set the ENCRYPTION_WALLET_LOCATION.  Depending on this location to be the "default" location can cause issues when you start using a wallet for other purposes.  This same location is the default location for any EUS implementations.

Since you should be on 19c, you should be using WALLET_ROOT for encryption wallet location.

NOTE: If you running databases in OCI, it is mandatory to be using WALLET_ROOT in order to utilize the recovery service.

EUS wallet

This wallet typically causes the most headaches for users.  The hierarchy Oracle uses to find the location of an EUS wallet is below.  Like the encryption wallet, it follows this hierarchy and it will use the first wallet it finds.

WALLET_LOCATION in the sqlnet.ora : When multiple databases are sharing the same $ORACLE_HOME (and thus the same sqlnet.ora file), this becomes confusing. The workaround was to set the location using a variable representation of the DB_NAME as part of location string.  

$ORACLE_BASE/admin/{DB name}/wallet : This is the location that most customers place their EUS wallets in since it is isolated to the Database associated with the wallet

NOTE: The issue arises when customers use a product/feature that updates the WALLET_LOCATION in the sqlnet.ora, which breaks authentication since the WALLET_LOCATION is checked first.

Certificate authorities and Self-signed certificates 

The most common use case for certificate authorities is when  utilizing the DBMS_CLOUD family of products.  Products such as DBMS_CLOUD call out object storage and require a secure (HTTPS) connection. In order to open a secure connection the client needs to authenticate the certificate as  valid certificate, or use a self-signed certificate that is stored in the wallet.

This same issue is true when using DBMS_CLOUD_AI and DBMS_VECTOR_CHAIN which makes calls to external LLMs that often require a secure connection.

This wallet is controlled by setting the database property "SSL_WALLET". 

For simplicity I would recommend creating a central wallet that can be used by ALL databases on the host and is stored within $ORACLE_BASE. My favorite location is $ORACLE_BASE/cert_wallet which identifies it as containing certificate authorities.

I do not recommend adding certificates to the EUS wallet, or the SEPS wallet (discussed next) as it becomes more difficult to mange multiple wallets to make updates.

SEPS authentication 

The next wallet I want to discuss is the SEPS authentication wallet. This wallet is used by Oracle clients (sqlplus, RMAN, and ZDLRA) to store the credentials for a database.

The connection string (either an ezconnect string or a tnsnames.ora entry) is added to the wallet, along with the username and password that will be used when connecting using this entry.  

The location of the wallet is stored in the sqlnet.ora file, and there are 2 parameters associated with this setting.

SQLNET.WALLET_OVERRIDE=true

WALLET_LOCATION={location on disk}

NOTE: Setting the WALLET_OVERRIDE to true disables any OPS$ usage and allows the usage of SEPS wallets for authentication. 

Setting the WALLET_LOCATION on a host that supports databases utilizing EUS often causes issues if it does not specify a separate location each database using variable.  The sqlnet.ora file is only read at startup, so changes to the WALLET_LOCATION might not become apparent to after a database bounce.

Recommendation :

If you are using multiple products like EUS, I recommend using the TNS_ADMIN variable to mange which wallet to use in scripts.  With TNS_ADMIN you can point to a directory containing a sqlnet.ora file specific to the database, and leave the original sqlnet.ora file without a WALLET_LOCATION entry.

Real-time redo and TLS certificates for ZDLRA 

Prior to the 19.18 DB release, configuring real-time redo for databases sending backups to the ZDLRA required a bounce of the database (to refresh the DBs copy of the sqlnet.ora), and it required the WALLET_LOCATION to be set in the sqlnet.ora.

This changed with 19.18, and I recommend you use the new location.

   The hierarchy Oracle uses to find the location of the wallet real-time wallet is below.  Like the encryption wallet, it follows this hierarchy and it will use the first wallet it finds.

WALLET_ROOT/server_seps : If the variable WALLET_ROOT is set, and a wallet exists in the server_seps subdirectory, that wallet is used by the real-time redo.  This is a HUGE improvement as it doesn't require a bounce, and it makes it much easier to avoid issues with EUS, and databases that share the same $ORACLE_HOME.

WALLET_LOCATION in the sqlnet.ora : When multiple databases are sharing the same $ORACLE_HOME (and thus the same sqlnet.ora file), this becomes confusing. The workaround was to set the location using a variable representation of the DB_NAME.  This is what I mentioned for EUS.

MKSTORE vs ORAPKI

orapki 

The orapki utility manages public key infrastructure (PKI) elements, such as wallets and certificate revocation lists, from the command line.  This is the recommended method of managing wallet files.

You can use the orapki command-line utility to perform the following tasks:

• Creating and viewing signed certificates for testing purposes

• Manage Oracle wallets (except for Transparent Data Encryption keystores):

  • Create and display Oracle wallets

  • Add and remove certificate requests

  • Add and remove certificates

  • Add and remove trusted certificates

• Manage certificate revocation lists (CRLs):

  • Renaming CRLs with a hash value for certificate validation

  • Uploading, listing, viewing, and deleting CRLs in Oracle Internet Directory

NOTE: The above is directly from the 19c documentation.  You can see that orapki is used to manage certificates with no mention of managing SEPS credentials.

mkstore

The first thing you will notice with mkstore, is that the mkstore command should be considered deprecated.  Upon digging into this some more, I found a comment from Russ Lowenthal (VP of Database Security products) who mentions that the SEPS credential wallet management will not be added to orapki until AFTER 23c.

NOTE: Even though it is considered deprecated, mkstore is the only way to manage SEPS credentials, and should only be used to manage SEPS credentials.

How to manage the types of wallets

Wallet Type	How to manage contents
Encryption Keys	Utilize the "ADMINISTER KEY MANAGEMENT" statement from the database
External user authentication	Use orapki to manage certificates, or the OWM tool which uses orapki
Certificate authorities and Self-signed certificate	Use orapki to manage certificates
SEPS authentication	Use mkstore for now, as orapki does not support SEPS
Real-time redo for ZDLRA	Use mkstore for now, as orapki does not support SEPS
TLS certificates for ZDLRA	Use orapki to manage the certificates

Wallet names and type

When you look in the wallet directory you would see one, or both of these wallets.

cwallet.sso - This is an auto login wallet.  With an autologin wallet you can access the contents without having to provide a password. In almost all cases, you will have this type of wallet entry.

ewallet.p12 - This is the passworded wallet. In order add/change/delete entries you need to specify a password when making those changes.  

NOTE:

• If only the cwallet.sso exist, you can assume it is an autologin only wallet.
• If both wallets exist, you can access the contents without a password, but any add/change/deletion commands will require a password and update both the passworded wallet and the autologin wallet.
• If only the ewallet.p12 exists, to access the contents of the wallet  you must provide a password.

Non Auto Login wallet

This is the least common wallet type, since it requires a password to access the contents.  In this configuration you create a new wallet using orapki and provide a password.  In this case there will only be a single wallet file, ewallet.p12.

NOTE: You cannot create a non autologin wallet with mkstore.

• orapki wallet create -wallet {wallet location}

Auto login only wallets

You can create an autologin wallet using either mkstore, or orapki command.  The idea of an autologin wallet, is that you can add entries to this wallet without needing a password. You can also list the entries in the wallet using either CLI tool. In this configuration there is only a cwallet.sso file in the wallet directory

Auto login wallets

This is the most common configuration that you will see.  There is both a passworded wallet, and an autologin wallet. With both wallets, it requires a password to make changes, but no password is required to open the wallet and use it.  The two wallets are synchronized when you make changes.

There are two ways to create autologin wallets.

    1. Create a non autologin wallet using orapki, then create an autologin wallet from the non autologin wallet.

• orapki wallet create -wallet {wallet location}
• orapki wallet create -wallet {wallet location} -auto_login  OR
• mkstore -wrl {wallet location} -createSSO

     2. Create an autologin wallet  and non autologin wallet together

•     orapki wallet create -wallet {wallet location} -auto_login

Local Autologin wallets

Local Autologin wallets work the same way as the autologin wallet, EXCEPT, the wallet is encrypted in a way that makes it only usable on the host it was created.  You cannot copy the autologin wallet to another host and use it.

When creating a local autologin wallet you would use either

• mkstore -wrl {wallet location} -createLSSO
• orapki wallet create -wallet {wallet location} auto_login_local

NOTE:

• Local autologin wallets are much more secure as they can only be used on the host  where the wallet was created. 
• When backing up wallets, this includes Encryption wallets, only backup the ewallet.p12 file.  This ensures that a password is required to utilize the wallet.

Summary 

Starting with DB 19.18, you have the ability to store individual credential wallets for real-time redo transportation when leveraging ZDLRA for backups.  You can also use the TNS_ADMIN variable to set a different location when using SEPS authentication.  It is now possible to manage multiple wallets separately without having conflicts between products and features.