TLS security is becoming more and more necessary to protect data, and the ZDLRA supports configuring TLS security for both communicating with the RMAN catalog, and for send/receiving backup pieces. In this blog post I will go through the steps to configure TLS security.
These steps are well documented and you can find the documentation below
Configuring Certificate Management
As a DBA, some of the concepts were new to me, and I wanted to go through the steps in detail so even a DBA (like me) could understand the process.
- Determine the DNS entries
In order to create the certificate request I first need to gather DNS network information about the ZDLRA.
This can be done by utilizing the "racli list san" command.
racli list san
Below is the example output from the documentation showing the DNS entries.
Created log /opt/oracle.RecoveryAppliance/log/racli_list_san.log
Thu May 6 16:18:33 2021: Start: List SAN
CN = zdlra09ingest-scan1.yourdomain.com
DNS.1 = zdlra09adm01.yourdomain.com
DNS.2 = zdlra09adm02.yourdomain.com
DNS.3 = zdlra09ingest-scan1.yourdomain.com
DNS.4 = zdlra09ingest01-vip.yourdomain.com
DNS.5 = zdlra09ingest01.yourdomain.com
DNS.6 = zdlra09ingest02-vip.yourdomain.com
DNS.7 = zdlra09ingest02.yourdomain.com
Thu May 6 16:18:39 2021: End: List SAN
This returns the two pieces of information necessary to create the request.
CN (Common Name)
The common name is the name associated with the certificate. For the ZDLRA this will be the scan listener's fully qualified DNS name. You will request a certificate registered to this name.
Alternate Names
As you probably know the Protected Database client connects to the ZDLRA using DNS entries other than the Scan Listener. The scan listener is used to balance traffic and the connection itself is handed to a DNS/IP address assigned to one of the hosts.
Because of this you can see in the example that there are 7 additional DNS entries that are associated with this certificate as alternate names.
2. Prepare Certificate Details
The next step is to create the signed certificate and add it to the ZDLRA
There are 2 methods you can use to create the signed certificate, and they are outlined in the documentation
NOTE: You need a signed certificate in order to implement TLS on the ZDLRA. If you are not familiar with the process of requesting a signed certificate you should contact someone from your security team, or talk to someone in your organization who has gone through the process of requesting a server certificate.
A) Certificate Creation Using Third Party Software.
Request Certificate
With this method you request a signed certificate from your company's certificate authority.
Along with the signed certificate the private key is generated by the certificate authority and provided to you.
I used this method because my companies certificate authority process pre-filled in most of the fields required for the request.
In order acquire the certificate I created a request (following my companies process) and entered the two items determined above in the request details along with the company specific information provided to me by the security team.
- CN name
- Alternate DNS names
From this, my company's certificate request process (once approved) returned to me a .ZIP file containing two files
- {CN name}.crt
- {CN name}.key
I was also given a link to download the Trusted Certificate associated with the certificate authority used by my company.
I stored the Trusted Certificate in a .pem file.
- {CN name}.pem
NOTE: The PKCS12 command (used in the next step) requires a "Trusted Certificate" as input. When signing my certificate, I was also given a link to the CA Chain certificates, including ROOT and intermediate CA. As a DBA, I did not immediately understand this concept. The "Trusted Certificate" contains the ROOT certificate and any intermediate certificates for the 3rd part Certificate Authority that will validate the signed certificate and ensure the certificate is valid and belongs to DNS names provided with the certificate request.
Examples of Trusted Certificate Authorities are DigiCert, Entrust, GlobalSign, Symantec, etc.
You can find more detail on this concept here.
Once completed you can continue to Step #3 with the 3 files
- {CN Name}.crt - Signed certificate request
- {CN Name}.key - Private key
- {CN name}.pem - Trusted Certificate (may be named differently)
B) Create the request and send the request to be signed
With this method, you create the request and generate the key on the ZDLRA. The request is sent to the Certificate Authority, and signed. You are responsible for creating and storing the private key.
Manually create the certificate request
You begin by creating a ".crt" file with your company's certificate information.
You can see an example in the documentation.
You need to replace the information in the request file with your companies detail that was provided to you.
With your company specific detail, you also need to ensure that the request contains the proper
- CN name from the first step
- Alternate names from the first step
Upload your certificate request using your company's process
Upload your certificate request to get it signed.
Once signed you will be returned your certificate and you will be given a copy of (or a link to) your company's Trusted Certificate. (see explanation from the first method if this is a new concept).
You should have 2 files
- {CN name}.crt
- {CN name}.pem
NOTE: The .pem file for the trusted certificate might be named differently as it is associated with Certificate Authority rather than the Common Name.
Generate a private key
You need to generate a private key to be associated with your signed certificate. Using the first method, the key was provided to me. With this method you will generate the key yourself.
The command to generate this key is below, and you may assign a password to the private key.
openssl genrsa -passout pass:<your password> --out /dir/{CN name>.key 2048
Once completed you will have 3 files
- {CN Name}.crt - Signed certificate request
- {CN Name}.key - Private key
- {CN name}.pem - Trusted Certificate
3. Add Certificate to ZDLRA
Sign the user Certificate with the Trusted Certificate
The next step is create a signed user certificate. This step will create a {CN name}.p12 file that can be added to the wallet on the ZDLRA.
NOTE: A PKCS12 file, often also called a PFDX file, is a standard format for storing digital certificates and private keys within a single, encrypted file. It's commonly used to bundle all the necessary cryptographic components for secure communicate, especially for SSL/TLS certificates and code signing. This file combines the signed certificate, the private encryption key, and the Certificate Authority (Trusted Certificate) information into a single file.
The PKCS12 file is created by executing "openssl pkcs12 -export" and passing in all three files from above.
openssl pkcs12 -export --in /<DIR>/<NAME>.crt
--inkey /<DIR>/<NAME2>.key --certfile /<DIR>/<NAME3>.pem
--passin pass:<YOURPASSWORD> --passout pass:<YOURPASSWORD>
--out /<DIR>/<NAME4>.p12
NOTE: The command includes a password. The --passin password is the password for the .key file and the --passout password will be used adding to the .p12 file when created.
openssl pkcs12 -export --in /<DIR>/<NAME>.crt
--inkey /<DIR>/<NAME2>.key --certfile /<DIR>/<NAME3>.pem
--passin pass:<YOURPASSWORD> --passout pass:<YOURPASSWORD>
--out /<DIR>/<NAME4>.p12
Import the user Certificate with the Trusted Certificate
The next step is to take the .PKCS12 file (contained the signed certificate, the private key, and the Certificate Authority) along with the trusted certificate and add them to the wallet on the ZDLRA.
In this step you execute the command below to add both files to the wallet. Ensure you give a fully qualified location for the files.
racli add certificate --signed_cert /dir/{CN name}.p12 --trusted_cert=/dir/{CN name}.pem
NOTE: If this is the first time executing this command you will be asked to provide a password for the wallet. The password will be stored on the ZDLRA and you will not be asked for it again. You will also be asked for the password used to create the PKCS12 files in the previous step
racli add certificate --signed_cert /dir/{CN name}.p12 --trusted_cert=/dir/{CN name}.pem
Validate both the user Certificate and Trusted Certificate were added
Execute the "racli list certificate command".
This command should return BOTH the "trusted_cert" and "signed_cert" similar to the example in the documentation.
4. Enable the certificate
Once both the trusted_cert and signed_cert have been added to the wallet on the ZDLRA we can configure the ZDLRA to utilize TLS encryption for BOTH TNS (TCPS SQL*net) and backup (HTTPS) traffic.
NOTE:
- The ZDLRA will bounce when implementing this step
- TLS will be the default for sending and receiving backups. ALL channel configurations should contain "_RA_NO_SSL=TRUE" in order to ensure they continue sending backups until they are configured with a client wallet.
- Configuring TLS with "ONLY" will disable non-TLS connections and traffic. Ensure all of your clients are configured for TLS prior to enabling ONLY.
The command to enable TLS is specified in the documentation with the default settings.
When enabling TLS you need to decide if you want TLS to be mandatory, or optional.
You can use the simple command (without any defaults), or you can change the default ports.
If you want to use all of the defaults you can execute the command below
racli alter network --service=ra_server --encrypt=enable | only
4. Validate the TLS configuration
You can validate the configuration in a couple of different ways.
- Check the health of TLS with "racli run check --check_name=tls_health". This should give you a PASS status
- As Oracle execute a listener status "lsnrctl status". Below is the output and I can see that TPCS traffic is configured to go over port 2484 and HTTP will go over port 8005 has a certificate configured to ensure it is encrypted.
5. Configuring the Protected DB Client wallet
When using TLS there are 2 wallet entries that need to be referenced when communicating with the ZDLRA
- Trusted Certificate - This is a new wallet entry and needs to be present so that the client can validate the certificate.
- VPC credentials (SEPS) - These are the login credentials for the VPC user. If you are using an EZ connect entry, then you would need to change the entry format to use EZConnect Plus (see below) or a TNS alias. If you are currently using a TNS alias you would need to update the tnsnames.ora entry.
These two entries are independent and do not have to be in the same wallet file.
Update current wallet file to include the Trusted Certificate
In order to add certificates to the wallet file, the wallet file must have been created using the orapki command (rather than mkstore).
If the your wallet file was created with mkstore, you will need to recreate it with the orapki command
You will be loading the Trusted Cert into the wallet file used by the ZDLRA to authenticate the VPC user.
More discussion on where the wallet should be located, and how to set the wallet location is covered in my last blog post you can find here.
Assuming the Trusted Certificate is stored in a file names {CN name).pem you would execute the command below also providing the location of the wallet you want to add the certificate to.
Remember it needs to be added to all nodes in a RAC cluster.
orapki wallet add --wallet <WRL> --trusted_cert --cert <dir>/{CN name}.pem
Update the SEPS wallet
If you have been using the EZConnect string in your SEPS wallet vs a tnsnames alias you are at a decision point.
EZConnect - {host}:{port}/{service}
When using TCPS to encrypt communication, you need to indicate that you are using TCPS, and you need to tell EZConnect where to find the wallet file containing the Trusted Certificate for the communication.
Because EZConnect does not support TCPS you need to use EZConnect Plus. EZConnect Plus was introduced with 18g, and is supported with release 19g and above.
EZConnect Plus [Protocol]://{host}:{port}/{service}?wallet_location={wallet location on disk}
When adding the SEPS credential for the VPC user you need to use the EZConnect Plus format in your wallet for the connection to succeed.
TNSNAMES Alias - As an alternate to using EZConnect, you can use a tnsnames.ora alias. If using an alias, you would add the entry to the tnsnames.ora file that points to the correct port and uses TPCS, and you would ensure the WALLET_LOCATION set in the associated sqlnet.ora file contains the Trusted Certificate.
EZConnect Plus example :
TCPS://ra1-scan1.mycorp.com:2484/radb1?/u1/app/oracle/mydb/wallet/server_seps
6. Configuring the Client to utilize TLS
There are three places where you would change the settings to use TLS
- RMAN catalog connections (/@)
- RMAN channel configuration
- Protected DB real-time redo destination
RMAN catalog connections
In order to connect to the RMAN securely with TLS you need to ensure that your connection is using the correct encrypted port using the TPCS protocol, and you need to ensure that the WALLET_LOCATION in the sqlnet.ora file points to a wallet containing the Trusted Certificate. You can connect to the catalog using a few different methods
- Connect directly using the password and using a tnsnames.ora alias.
- Connect directly using the password and using the EZConnect Plus format specifying TCPS and the location of the Trusted Certificate wallet.
- Connect using a SEPS entry pointing to a tnsnames alias or the EZConnect Plus formatted string.
RMAN channel configuration
There are a couple of changes to the channel configuration. The channel configuration will use the two different wallet entries.
- ra_wallet - This points to the SEPS wallet location and you specify the credential you want to utilize which is in this SEPS wallet.
- Trusted Certificate location - If you are using the EZConnect Plus format, you would specify the wallet_location which points to a wallet containing the Trusted Certificate. If you are using a tnsnames alias, the WALLET_LOCATION in the sqlnet.ora points to a wallet containing the Trusted Certificate.
NOTE: If your entry is in the EZConnect Plus format, you might have to enclose the credential on double quotes.
Real-time redo configuration
The real-time redo connection will use the two different wallet entries.
- SEPS wallet - Real-time redo will determine if the parameter WALLET_ROOT is set and if there is a server_seps directory within the WALLET_ROOT. If so, that wallet is used to look up the SEPS connection. If WALLET_ROOT is not set, the connection will use the WALLET_LOCATION set in the sqlnet.ora file pointed to during DB startup.
- Trusted Certificate location - Real-time redo will always use the WALLET_LOCATION from the sqlnet.ora file to locate the Trusted Certificate.
NOTE: If your entry is in the EZConnect Plus format, you might have to enclose the credential on double quotes.
