Showing posts with label data guard. Show all posts
Showing posts with label data guard. Show all posts

Tuesday, January 9, 2024

RMAN create standby database - Restore or Duplicate ?

RMAN create standby database - Are you like me and use "restore database" for large databases, or like most people (based on my Linkedin poll) and use "duplicate for standby"? 

The table below shows you the 3 main differences between the 2 methods.


This post started with a discussion within my team around which method you use. I, being of the "restore database" camp, didn't realize how commonly used "duplicate for standby" is. 
I have also dug through the documentation, and there is no common method that is mentioned. Even the 21c documentation for creating a standby database doesn't mention using the duplicate command.
I also was pointed to a MOS note that goes through creating a standby directly from the primary across multiple nodes, and with encryption.  Creating a Physical Standby database using RMAN restore database from service (Doc ID 2283978.1)

Well in this post, I will explain why "restore database" has been my preference. 

NOTE : If you are creating a standby database that is encrypted and the source database is not (creating a standby database in OCI for example) then I have instructions at the end of this post for how to use "Restore Database" to create a hybrid standby database.

Duplicate database for standby


From the poll I ran, this is the most common way to create a standby database.  It is probably the simplest way also because a lot of the configuration of the standby database is done automatically as part of the automated process.
Below is the simplified steps to perform this process.

PRE work

  1. Create simple initfile on the standby host.  The real SPFILE will be brought over as part of the duplication process.  This may contain location parameters for datafiles and redo logs if different from the primary.
  2. Create directories on the standby host.  This includes the audit directory, and possibly the database file directories if they are different from the host.
  3. Startup nomount.

Duplicate 

The duplicate process automatically performs these major steps using the standby as an auxiliary instance.

  1.  Create an SPFILE. The process creates an SPFILE for the standby and sets parameters for the standby.
  2. Shutdown/Startup standby database. This will use the newly created SPFILE during the rest of the processing
  3. Restore backup controlfile for standby database. The controlfile for the standby database is put in place, and the spfile is updated to it's location
  4. Mount controlfile . Mount the controlfile that was restored
  5. Restore database . Restore the datafiles files for the CDB and PDBs to their new location on the standby
  6. Switch datafile . Uses the new location of the datafiles that were restored.
  7. Create standby redo logs.
  8. Set parameters for standby database. The parameters necessary to communicate with the primary database are set.
  9. Put standby in recover mode . By this time, you should have set the primary database to communicate with the standby database.

NOTES

If you noticed above, I highlighted the second step which forces a shutdown/startup of the standby database. Because of this step, it is not possible to use this method and restore across nodes in a RAC database.  This can cause the duplicate operation to take much longer for larger databases.
Then in step #5 you can see that the "Restore Database" is automatic in the processing and it is not possible to perform a "restore as encrypted" if you are migrating to OCI from a non-TDE database.  The duplicate process does support "restore as encrypted", but only for creating a new Database, not a standby database.

Restore Database


This is the method that I've always used.  There is no automation, but it gives you much more control over the steps.  

PRE work

  1. Restore copy of prod SPFILE to standby host.  For this process, it doesn't matter if it is an intifile or spfile.  In this file you set all the parameters that are needed for the standby database to communicate with the primary and store datafiles/logfiles in the correct location.
  2. Create directories on the standby host.  This includes the audit directory, and possibly the database file directories if they are different from the host.
  3. Startup nomount.
  4. Create copy of primary controlfile for standby. This will be used for the standby database, and should contain the backup catalog  of the primary database, and the RMAN settings including the  channel definitions.
  5. Copy standby controlfile to standby host. The controlfile is copied to the standby host, and may be put in ASM at this point. Ensure the spfile points to the controlfile (and/or srvctl).
  6. Alter database mount.  Mount the controlfile. 
  7. Start up ALL nodes in the RAC cluster in mount mode.  This will allow you to restore the database across ALL nodes in the RAC cluster, and include all the networking from these nodes.  For a large database hosted on multiple DB nodes this can make a HUGE difference when restoring the database.
  8. Create (or copy) TDE wallet.  If the standby database is going to be TDE, then include the wallet if the primary is TDE, or create a new wallet and key if the standby database is going to be TDE.

Restore Database 

The restore process is a manual process

  1.  RMAN Connect to database (and possibly RMAN catalog). Connect to the database and make sure you have access to the backups. For ZDLRA this may mean connecting to the RMAN catalog.
  2. Restore Database (as encrypted). This will restore the database to the new location.  With Restore Database, the database can be encrypted during the restore operation.  With 19c it is supported to have the standby database be encrypted without the primary database being encrypted (Hybrid dataguard).
  3. Switch datafile . Uses the new location of the datafiles that were restored.
  4. Recover database. This will use the archive logs that are cataloged to bring the standby database forward
  5. Create standby redo logs.
  6. Set parameters for standby database. The parameters necessary to communicate with the primary database are set.
  7. Put standby in recover mode . By this time, you should have set the primary database to communicate with the standby database.


NOTES

With the restore database, there are 2 sections I highlighted and these are the advantages that I love about using this method.
  • RMAN is restoring across multiple nodes in a RAC cluster which can make the restore operation much faster.
  • Restore as encrypted allows you take a database that may have TDE partially implemented, or not implemented and create a new standby database that is encrypted. With the duplicate method, TDE would have to be implemented separately.
If you are restoring a VERY large database (200 TB for example) that was not TDE from object storage to the Exadata Cloud Service, both of these advantages can make a HUGE difference when creating a standby database.

Comparison

The chart below compares the the differences between "Duplicate Database" and "Restore Database".

WARNING: When using a ZDLRA for backups, it is NOT recommended to use the "Restore Database" to clone a database as a new copy. Registering the restored copy can cause issues with the RMAN catalog because the "restore database" leaves entries in the RC_SITE table.



Data Guard Hybrid Cloud Configuration

The ability to create a hybrid cloud configuration was introduced in Version 19.16 and there is a great blog post from Glen Hawkins explaining this feature.
This feature allows you have your Primary database remain unencrypted (no ASO license), but still have the standby database in OCI be TDE encrypted.

In this section I want to talk about how you can use "Restore Database as Encrypted" to implement this configuration quickly.

If you want to implement this feature using "Duplicate for standby" you have to separately encrypt the datafiles once they are restored in OCI.  This can be done online, or offline, but it is still a time consuming task.

Prepare the primary and future standby databases

The first step is prepare the primary database and future standby database by creating a wallet file and setting encryption keys.  There is a great video put together by Peter Wahl (PM for TDE and OKV) that goes through a lot of the steps.

Below is a summary of the steps you need to perform.  You can follow along the steps in Peter's video and I will point out where in the video you will find each step.

  • Create the directories on the primary (3:40) -  Directories are specified in the video and need to be created on all nodes in a RAC cluster.
  • Create the directories on the standby database (4:18) -Directories are specified in the video and need to be created on all nodes in a RAC cluster.
  • Set the wallet_root in the primary (4:25) - This is set in the SPFILE only
  • Set tablespace_encryption to decrypt_only on primary (4:40) -  This is set in the SPFILE only
  • Set the default algorithm to AES256 on primary (4:50) - This is set in the SPFILE only
  • Set wallet_root on standby, tablespace_encryption to auto_enable, and default algorithm on standby --  This is set in the initfile that you create prior to performing the restore.  This step is different from the video because there is no standby at this point.
  • Bounce the primary database (5:50) - This can be in a rolling manner.
  • Create a password protected wallet on the primary (7:25) - This gets created in the default location specified from WALLET_ROOT
  • Create an auto open wallet on the primary (7:30) - This will make it automatically open for the database.
  • Set the encryption keys in the primary (7:40) - The keys will be created so that they can be used when restoring the datafiles on the standby database.
  • Copy the wallets from the primary to the standby (7:55) - This provides the standby database with the keys to encrypt.




Friday, September 29, 2023

DG per PDB with 23c on DBCS (Oracle Base Database)

At Oracle Cloud World (a few weeks ago as of writing this) 23c was announced as being available with DBCS.
 I've been wanting to test out DGPDB, which is Data Guard per PDB. This new feature was introduced with 21c and is covered in detail in this blog here .


Since I cover a lot of topics around backup, recovery, and availability this feature intrigued me.

NOTE: DGPDB is NOT supported with the tooling on DBCS and by testing it in your environment as I am going to do here your databases will not be supported.

First I am going to start with the documentation which can be found here.

My Environment :

I created 2 database services in ExaCS, both of which are 23c databases that are TDE encrypted (since they are in DBCS), and I am using a locally managed key (local wallet file).


CDB Name DB Unique Name PDB name
cdb1 cdb1_c83_iad pdb1
cdb2 cdb2_cdx_iad cdb2_pdb2


Step # 1 - Prepare the 2 CDBs

The first step is to prepare the 2 CDBs and follow the steps in the documentation

On both databases execute
  • ALTER DATABASE FORCE LOGGING;
  • ALTER DATABASE FLASHBACK ON;

On the source database execute
  • alter system set dg_broker_start=true scope=both;
  • alter system set log_archive_dest_1="LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=cdb1_c83_iad";
  • alter system set standby_file_management=auto scope=both; ;
On the destination database execute
  • alter system set dg_broker_start=true scope=both;
  • alter system set log_archive_dest_1="LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=cdb2_cdx_iad";
  • alter system set standby_file_management=auto scope=both;

Step # 2 - Configure tnsname.oras and SEPS wallets

First I am going to ensure that both tnsnames.ora files contains entries for both CDBs.
  • CDB1_C83_IAD
  • CDB2_CDX_IAD
Next I create a SEPS wallet on both nodes, and add entries for both databases
mkdir -p $ORACLE_HOME/dbs/wallets/dgpdb
mkstore -wrl $ORACLE_HOME/dbs/wallets/dgpdb -createALO
mkstore -wrl $ORACLE_HOME/dbs/wallets/dgpdb -createCredential cdb1_c83_iad sys {password}
mkstore -wrl $ORACLE_HOME/dbs/wallets/dgpdb -createCredential cdb2_cdx_iad sys {password}
mkstore -wrl $ORACLE_HOME/dbs/wallets/dgpdb -listCredential


Then I update the sqlnet.ora to use this wallet file.

WALLET_LOCATION =
    (SOURCE =
      (METHOD = FILE)
      (METHOD_DATA =
        (DIRECTORY = /u01/app/oracle/product/23.0.0.0/dbhome_1/dbs/wallets/dgpdb)
    )
)
SQLNET.WALLET_OVERRIDE = TRUE

And finally I test the connection
  • sqlplus /@cdb1_c83_iad as sysdba
  • sqlplus /@cdb2_cdx_iad as sysdba

Step # 3 - Create source and target configuration

Configure the source


dgmgrl /@cdb1_c83_iad <<EOF

CREATE CONFIGURATION 'cdb1_c83_iad' AS CONNECT IDENTIFIER IS cdb1_c83_iad; 
SHOW CONFIGURATION; 

EOF

Configure the target
dgmgrl /@cdb2_cdx_iad <<EOF

CREATE CONFIGURATION 'cdb2_cdx_iad' AS CONNECT IDENTIFIER IS cdb2_cdx_iad; 
SHOW CONFIGURATION; 

EOF

Step # 4 - Establish the connection between configurations

I ran the step in the documentation to configure the connection.


dgmgrl /@cdb1_c83_iad <<EOF

ADD CONFIGURATION 'cdb2_cdx_iad' CONNECT IDENTIFIER IS cdb2_cdx_iad; 
SHOW CONFIGURATION; 
ENABLE CONFIGURATION ALL; 
SHOW CONFIGURATION; 
EOF


And I validated that the output looked correct.

Configuration - cdb1_c83_iad

  Protection Mode: MaxPerformance
  Members:
  cdb1_c83_iad - Primary database
  cdb2_cdx_iad - Primary database in cdb2_cdx_iad configuration

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 1 second ago)

Step # 5 - Prepare the databases for DGPDB

The first step was to ensure that the PDBS are open in both database, which they are.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           READ WRITE NO
SQL>

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 CDB2_PDB1                      READ WRITE NO
SQL>


Then prepare the databases for DGPDB

 dgmgrl /@cdb1_c83_iad
DGMGRL for Linux: Release 23.0.0.0.0 - Production on Thu Sep 28 17:40:38 2023
Version 23.3.0.23.09

Copyright (c) 1982, 2023, Oracle and/or its affiliates.  All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected to "cdb1_c83_iad"
Connected as SYSDBA.
DGMGRL> EDIT CONFIGURATION PREPARE DGPDB;
Enter password for DGPDB_INT account at cdb1_c83_iad:
Enter password for DGPDB_INT account at cdb2_cdx_iad:

Prepared Data Guard for Pluggable Database at cdb2_cdx_iad.

Prepared Data Guard for Pluggable Database at cdb1_c83_iad.
DGMGRL>


Step # 6 - Restart the listeners and databases with the wallet location


On both the source and destination, I performed an "su - grid" and restarted the listeners.
I also restarted both databases. Until I restarted the databases I kept getting a 

 "Error: ORA-12578: A wallet file was not found or failed to open."

error message.


Step # 7 - Add the pluggable database to the destination

The next step is to add the pluggable database to the destination in dgmgrl.

DGMGRL> add pluggable database pdb1_stby at cdb2_cdx_iad source is pdb1 at cdb1_c83_iad  pdbfilenameconvert is "'+DATA/CDB1_C83_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/','+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/'"  ;
ORA-46697: Keystore password required.
Help: https://docs.oracle.com/error-help/db/ora-46697/


At this point by following the documentation, adding the pluggable database failed with  "ORA-46697: Keystore password required."

After some digging I found that I need to pass the wallet password (which is my sys password), and that the phrase needs to be in single quotes.


DGMGRL> add pluggable database pdb1_stby at cdb2_cdx_iad source is pdb1 at cdb1_c83_iad  pdbfilenameconvert is "'+DATA/CDB1_C83_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/','+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/'"  'keystore  IDENTIFIED BY "W3lCom3#123#123"';

Pluggable Database "PDB1_STBY" added


Step # 8 - List datafiles in the source PDB


     FILE#  NAME
----------  ------------------------------------------------------------------------------------------------------------------------
         8  +DATA/CDB1_C83_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/system.273.1148746431
         9  +DATA/CDB1_C83_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/sysaux.270.1148746437
        10  +DATA/CDB1_C83_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/undotbs1.271.1148746445
        12  +DATA/CDB1_C83_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/users.274.1148746543

Step # 9 - Copy the datafiles to the destination


First I logged onto the primary database and closed it.

 alter pluggable database pdb1 close;

I decided to use DBMS_FILE_TRANSFER.GET_FILE

Step 9a - create directory on source pointing to datafiles


SQL*Plus: Release 23.0.0.0.0 - Production on Fri Sep 29 11:33:11 2023
Version 23.3.0.23.09

Copyright (c) 1982, 2023, Oracle.  All rights reserved.


Connected to:
Oracle Database 23c EE High Perf Release 23.0.0.0.0 - Production
Version 23.3.0.23.09

SQL> create directory SOURCE_DUMP as '+DATA/CDB1_C83_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/';

Directory created.

SQL> grant read,write on directory SOURCE_DUMP to public;

Grant succeeded.

SQL>

Step 9b - Log onto ASM and precreate directory

ASMCMD> mkdir 
+data/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/datafile


Step 9c - create directory on target pointing to datafiles

[oracle@cdb1 admin]$ sqlplus /@cdb2_cdx_iad as sysdba

SQL*Plus: Release 23.0.0.0.0 - Production on Fri Sep 29 11:33:44 2023
Version 23.3.0.23.09

Copyright (c) 1982, 2023, Oracle.  All rights reserved.


Connected to:
Oracle Database 23c EE High Perf Release 23.0.0.0.0 - Production
Version 23.3.0.23.09

SQL> create directory TARGET_DUMP as '+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/';

grant read,write on directory TARGET_DUMP to public;



Directory created.

SQL> SQL>
Grant succeeded.

Step 9d - create database link on target  pointing to source database


SQL> SQL> SQL> create public database link SOURCEDB connect to system identified by {password} using 'CDB1_C83_IAD';

Database link created.


SQL> select sysdate from dual@SOURCEDB;
select sysdate from dual@SOURCEDB
                         *
ERROR at line 1:
ORA-02085: database link SOURCEDB.ZFSADMIN.VCN.ORACLEVCN.COM connects to
CDB1.ZFSADMIN.VCN.ORACLEVCN.COM
Help: https://docs.oracle.com/error-help/db/ora-02085/


SQL> show parameter global

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
allow_global_dblinks                 boolean     FALSE
global_names                         boolean     TRUE
global_txn_processes                 integer     1
SQL> alter system set global_names=false;

System altered.

SQL> select sysdate from dual@SOURCEDB;

SYSDATE
---------
29-SEP-23

SQL>

Step 9e - Get the datafiles from the source


SQL> set timing on
BEGIN
dbms_file_transfer.get_file('SOURCE_DUMP',
'system.273.1148746431',
'SOURCEDB',
'TARGET_DUMP',
'SYSTEM.DBF');
END;
/SQL>   2    3    4    5    6    7    8

PL/SQL procedure successfully completed.

Elapsed: 00:00:01.21

Step # 10 - Rename datafiles in the destination.


List the placeholder datafiles in  destination database
SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 CDB2_PDB1                      READ WRITE NO
         4 PDB1_STBY                      MOUNTED
SQL> alter session set container=pdb1_stby;

Session altered.

Elapsed: 00:00:00.02
SQL> select name from v$datafile;

NAME
--------------------------------------------------------------------------------
+DATA/MUST_RENAME_THIS_DATAFILE_8.4294967295.4294967295
+DATA/MUST_RENAME_THIS_DATAFILE_9.4294967295.4294967295
+DATA/MUST_RENAME_THIS_DATAFILE_10.4294967295.4294967295
+DATA/MUST_RENAME_THIS_DATAFILE_12.4294967295.4294967295

Elapsed: 00:00:00.02
SQL>

Match the file number in the destination files to the source datafiles and rename them.

SQL> alter database rename file '+DATA/MUST_RENAME_THIS_DATAFILE_8.4294967295.4294967295' to '+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/SYSTEM.dbf';

Database altered.

Elapsed: 00:00:00.04
SQL> alter database rename file '+DATA/MUST_RENAME_THIS_DATAFILE_9.4294967295.4294967295' to '+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/SYSAUX.dbf';

Database altered.

Elapsed: 00:00:00.02

SQL> alter database rename file '+DATA/MUST_RENAME_THIS_DATAFILE_10.4294967295.4294967295' to '+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/UNDOTBS1.dbf';

Database altered.

Elapsed: 00:00:00.05
SQL> alter database rename file '+DATA/MUST_RENAME_THIS_DATAFILE_12.4294967295.4294967295' to '+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/USERS.dbf';

Database altered.

Elapsed: 00:00:00.03
SQL> select name from v$datafile;

NAME
--------------------------------------------------------------------------------
+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/system.dbf
+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/sysaux.dbf
+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/undotbs1.dbf
+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/users.dbf

Elapsed: 00:00:00.02
SQL>


Step # 11 - Bring over the TDE keys

Export the encryption keys from the source database

SQL> ADMINISTER KEY MANAGEMENT
  EXPORT ENCRYPTION KEYS WITH SECRET "export_secret"
  TO '/tmp/mykey'
  FORCE KEYSTORE
  IDENTIFIED BY "{password}";  2    3    4    5

keystore altered.



Import them into the target database
ADMINISTER KEY MANAGEMENT
  IMPORT KEYS WITH SECRET "export_secret"
  FROM '/tmp/mykeypdb1'
  force keystore
  IDENTIFIED BY "W3lCom3#123#123"
  WITH BACKUP;

SQL> SQL>   2    3    4    5    6
keystore altered.

SQL> SQL>

List the encryption keys on the  source database
PDB Name        Key ID                              Master Key ID
--------------- ----------------------------------- -------------------------
CDB$ROOT        35B424EFDB6D4F19BFF31B5D3DE0BFFA    ATW0JO/bbU8Zv/MbXT3gv/o=
PDB$SEED        00000000000000000000000000000000    AQAAAAAAAAAAAAAAAAAAAAA=
PDB1            1AAD19EB6F364F3BBF6FE226A25AF1E5    ARqtGetvNk87v2/iJqJa8eU=


List the encryption keys on the destination database
Master Key ID                                           Tag                  PDB Name        KEYSTORE_TYPE     Origin     Key Creation Time  Key Act. Time
------------------------------------------------------- -------------------- --------------- ----------------- ---------- ------------------ ------------------
AZ5CKuJLrU/Nv5M5pXQCFcEAAAAAAAAAAAAAAAAAAAAAAAAAAAAA                         CDB$ROOT        SOFTWARE KEYSTORE LOCAL      09/28/2023 09:09   09/28/2023 09:09
ATW0JO/bbU8Zv/MbXT3gv/oAAAAAAAAAAAAAAAAAAAAAAAAAAAAA                         CDB$ROOT        SOFTWARE KEYSTORE LOCAL      09/28/2023 09:10   09/28/2023 09:10
AVAymCcev0/Pv94Tiqw/P50AAAAAAAAAAAAAAAAAAAAAAAAAAAAA                         CDB2_PDB1       SOFTWARE KEYSTORE LOCAL      09/28/2023 09:12   09/28/2023 09:12
ARqtGetvNk87v2/iJqJa8eUAAAAAAAAAAAAAAAAAAAAAAAAAAAAA                         CDB2_PDB1       SOFTWARE KEYSTORE LOCAL      09/28/2023 09:13   09/28/2023 09:13




Step # 12 - Add the standby redo logs to the PDB and validate

I created standby redo logs in the PDB on Target database

select 'ALTER DATABASE ADD STANDBY LOGFILE thread ' ||a.thread# || ' group 1' ||a.group# || ' (''' ||substr(b.member,1,35) || 'standby_pdb1_1' || a.group# ||  ''') size ' || a.bytes ||';'
  from v$log a, v$logfile b
  where a.group#=b.group# ;SQL>   2    3

'ALTERDATABASEADDSTANDBYLOGFILETHREAD'||A.THREAD#||'GROUP1'||A.GROUP#||'('''||SUBSTR(B.MEMBER,1,35)||'STANDBY_PDB1_1'||A.GROUP#||'
----------------------------------------------------------------------------------------------------------------------------------
ALTER DATABASE ADD STANDBY LOGFILE thread 1 group 13 ('+RECO/CDB2_CDX_IAD/ONLINELOG/group_standby_pdb1_13') size 1073741824;
ALTER DATABASE ADD STANDBY LOGFILE thread 1 group 12 ('+RECO/CDB2_CDX_IAD/ONLINELOG/group_standby_pdb1_12') size 1073741824;
ALTER DATABASE ADD STANDBY LOGFILE thread 1 group 11 ('+RECO/CDB2_CDX_IAD/ONLINELOG/group_standby_pdb1_11') size 1073741824;

SQL> alter session set container=PDB1_STBY;

Session altered.

SQL> ALTER DATABASE ADD STANDBY LOGFILE thread 1 group 4 ('+RECO/CDB2_CDX_IAD/ONLINELOG/group_standby_pdb1_13') size 1073741824;

Database altered.


SQL> ALTER DATABASE ADD STANDBY LOGFILE thread 1 group 5 ('+RECO/CDB2_CDX_IAD/ONLINELOG/group_standby_pdb1_12') size 1073741824;

Database altered.

SQL>  ALTER DATABASE ADD STANDBY LOGFILE thread 1 group 6  ('+RECO/CDB2_CDX_IAD/ONLINELOG/group_standby_pdb1_11') size 1073741824;

Database altered.

SQL>

And then I validated the configuration

DGMGRL> VALIDATE PLUGGABLE DATABASE PDB1_STBY AT cdb2_cdx_iad;

  Ready for Switchover:    NO

  Data Guard Role:         Physical Standby
  Apply State:             Not Running
  Standby Redo Log Files:  3
  Source:                  PDB1 (con_id 3) at cdb1_c83_iad



Step # 13 - change the state of standby to APPLY-ON

I changed the state of the standby and checked the configuration.


DGMGRL> VALIDATE PLUGGABLE DATABASE PDB1_STBY AT cdb2_cdx_iad;

  Ready for Switchover:    NO

  Data Guard Role:         Physical Standby
  Apply State:             Not Running
  Standby Redo Log Files:  3
  Source:                  PDB1 (con_id 3) at cdb1_c83_iad


DGMGRL>
EDIT PLUGGABLE DATABASE PDB1_STBY AT cdb2_cdx_iad SET STATE='APPLY-ON';DGMGRL>
Succeeded.
DGMGRL> SHOW CONFIGURATION;

Configuration - cdb2_cdx_iad

  Protection Mode: MaxPerformance
  Members:
  cdb2_cdx_iad - Primary database
    Warning: ORA-16910: inconsistency detected for one or more pluggable databases

  cdb1_c83_iad - Primary database in cdb1_c83_iad configuration

Data Guard for PDB:   Enabled in TARGET role

Configuration Status:
SUCCESS   (status updated 37 seconds ago)

DGMGRL> SHOW PLUGGABLE DATABASE pdb1 AT cdb1_c83_iad;

Pluggable database - PDB1 at cdb1_c83_iad

  Data Guard Role:     Primary
  Con_ID:              3
  Active Target:       con_id 4 at cdb2_cdx_iad

Pluggable Database Status:
SUCCESS

DGMGRL> SHOW PLUGGABLE DATABASE pdb1_stby AT cdb2_cdx_iad ;

Pluggable database - PDB1_STBY at cdb2_cdx_iad

  Data Guard Role:     Physical Standby
  Con_ID:              4
  Source:              con_id 3 at cdb1_c83_iad
  Transport Lag:       20 hours 26 minutes 24 seconds (computed 9 seconds ago)
  Apply Lag:           (unknown)
  Intended State:      APPLY-ON
  Apply State:         Running
  Apply Instance:      cdb2
  Average Apply Rate:  (unknown)
  Real Time Query:     OFF

Pluggable Database Status:
SUCCESS

DGMGRL> VALIDATE PLUGGABLE DATABASE PDB1_STBY AT cdb2_cdx_iad;

  Ready for Switchover:    NO

  Data Guard Role:         Physical Standby
  Apply State:             Waiting for Redo Data
  Standby Redo Log Files:  3
  Source:                  PDB1 (con_id 3) at cdb1_c83_iad

DGMGRL>

Step # 14 - Perform a log switch at both databases



 sqlplus /@cdb1_c83_iad as sysdba

SQL*Plus: Release 23.0.0.0.0 - Production on Fri Sep 29 15:57:52 2023
Version 23.3.0.23.09

Copyright (c) 1982, 2023, Oracle.  All rights reserved.


Connected to:
Oracle Database 23c EE High Perf Release 23.0.0.0.0 - Production
Version 23.3.0.23.09

SQL> ALTER SYSTEM ARCHIVE LOG CURRENT;

System altered.

SQL> ALTER SYSTEM ARCHIVE LOG CURRENT;

System altered.

SQL> exit
Disconnected from Oracle Database 23c EE High Perf Release 23.0.0.0.0 - Production
Version 23.3.0.23.09
[oracle@cdb2 admin]$ sqlplus /@cdb2_cdx_iad as sysdba

SQL*Plus: Release 23.0.0.0.0 - Production on Fri Sep 29 15:58:11 2023
Version 23.3.0.23.09

Copyright (c) 1982, 2023, Oracle.  All rights reserved.


Connected to:
Oracle Database 23c EE High Perf Release 23.0.0.0.0 - Production
Version 23.3.0.23.09

SQL> ALTER SYSTEM ARCHIVE LOG CURRENT;

System altered.

SQL> ALTER SYSTEM ARCHIVE LOG CURRENT;

System altered.

SQL>


Step # 15 - Validate standby is applying redo.


DGMGRL> SHOW PLUGGABLE DATABASE pdb1_stby AT cdb2_cdx_iad ;

Pluggable database - PDB1_STBY at cdb2_cdx_iad

  Data Guard Role:     Physical Standby
  Con_ID:              4
  Source:              con_id 3 at cdb1_c83_iad
  Transport Lag:       0 seconds (computed 1 second ago)
  Apply Lag:           4 seconds (computed 1 second ago)
  Intended State:      APPLY-ON
  Apply State:         Running
  Apply Instance:      cdb2
  Average Apply Rate:  489 KByte/s
  Real Time Query:     OFF

Pluggable Database Status:
SUCCESS




SUMMARY :

This is an exciting new feature, and I am looking forward to see it expand.

Since a target database, can only have one source, I see this being useful to "pair" CDBs.
Instead of having 2 Primary CDBs in DC1 (for example) and 2 standby CDBs in DC2 you would have 2 Primary CDBs in in each Data Center, and you would be able to switch over any PDBs individually.


Since this is a paradigm change for RMAN, I am going to be following how this affects RMAN, the RMAN catalogs, and the ZDLRA/Recovery Service in it's support.


Finally :

It is all working, and for fun I am going to look at the destination settings on the source Database.

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
log_archive_dest_1                   string      LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=cdb1_c83_iad
log_archive_dest_2                   string      service="cdb2_cdx_iad", ASYNC NOAFFIRM delay=0 optional compression=disable max_failure=0 reopen=300 db_unique_name="cdb2_cdx_iad" net_timeout=30, valid_for=(online_logs)


Today I did have time to perform a switchover and a second switchover to bring it back.
Because my database is encrypted, I did import the keys from CDB2 into CDB1.  I am thinking when you implement this with TDE, you do need to keep the keys from both CDBs syncronized.
In OKV, I'm thinking you would want to possibly even use a single wallet. At the very least both primary databases would need access to both wallets.

OBSERVATIONS:


I will continue to update this post as I find useful observations.

Observation 1 :

I looked at the archive log using v$archived_logs view in the target (CDB2).
In it I only 6 archive logs were created.
SQL> select thread#,sequence#,dest_id,blocks,status from v$ARCHIVED_LOG;

   THREAD#  SEQUENCE#    DEST_ID     BLOCKS S
---------- ---------- ---------- ---------- -
         1          2          1     230208 A
         1          3          1     896391 A
         1          4          1          7 A
         1          5          1      35474 A
         1          6          1        387 A

I added a couple more PDBS and made changes to a second PDB (that didn't have a standby) and the original PDB (that had a standby).
I then created a workload in both PDBS and looked at the archive logs on the source.

   THREAD#  SEQUENCE#    DEST_ID     BLOCKS S
---------- ---------- ---------- ---------- -
         1          3          2       4894 A
         1          4          2     567681 A
         1          5          2         14 A
         1          6          2     871136 A
         1          7          2    1756544 A
         1          8          2    1847012 A
         1          9          2    1931501 A
         1         10          2    1901263 A
         1         11          2    1931491 A
         1         12          2    1931837 A
         1         13          2    1465002 A
         1         14          2    1955017 A
         1         15          2    1936946 A
         1         16          2    1685507 A
         1         17          2    1931499 A
         1         18          2    1808038 A
         1         19          2    1874455 A
         1         20          2    1910501 A
         1         21          2    2038138 A
         1         22          2    1812968 A
         1         23          2    1935941 A
         1         24          2    1939459 A
         1         25          2    2030413 A
         1         26          2    1934927 A

24 rows selected.

Then I looked at ASM on the target and I could ALL the archive logs on the target.


ASMCMD> ls
thread_1_seq_10.282.1148837541
thread_1_seq_11.283.1148837541
thread_1_seq_12.284.1148837557
thread_1_seq_13.285.1148837667
thread_1_seq_14.287.1148837943
thread_1_seq_15.288.1148837963
thread_1_seq_16.289.1148837979
thread_1_seq_17.290.1148837979
thread_1_seq_18.291.1148838023
thread_1_seq_19.292.1148838023
thread_1_seq_20.293.1148838101
thread_1_seq_21.294.1148838119
thread_1_seq_22.295.1148838137
thread_1_seq_23.296.1148838137
thread_1_seq_24.297.1148838183
thread_1_seq_25.298.1148838213
thread_1_seq_26.299.1148838237
thread_1_seq_3.275.1148831895
thread_1_seq_4.276.1148831899
thread_1_seq_5.274.1148831883
thread_1_seq_5.278.1148837115
thread_1_seq_6.277.1148836711
thread_1_seq_6.286.1148837681
thread_1_seq_7.279.1148837481
thread_1_seq_8.280.1148837499
thread_1_seq_9.281.1148837517


Observation 2 :

RMAN does not support this feature at this point.

 What I have discovered is that

  • Within the CDB, V$ARCHIVED_LOG does not recognize archive logs from the second primary needed to recovery the standby
  • Within the CDB, V$DATAFILE does not recognize the datafiles for the “standby database”
  • Within the standby PDB, V$DATAFILE recognizes the datafiles for that standby database, and only the standby datafiles
  • Within the standby PDB V$ARCHIVED_LOG only sees this archive logs for the standby database, and not the local archive logs produced
  • When executing RMAN for the CDB, only the locally produced archive logs are backed up
  • When executing RMAN for the CDB, only the primary datafiles are backed up.
  • When executing RMAN for the standby PDB, no datafiles are backed up.  “pluggable database PDB1_STBY does not have any data files”
  • When executing RMAN for the standby PDB, only the primary database’s locally produced archive logs are backed up.

 

SUMMARY – with DGPDB

 Within the PDB, the PDB can see the datafiles, and archive logs needed to recover with V$DATAFILE and V$ARCHIVED_LOG

 RMAN does not currently see the standby datafiles, or it’s archive logs to back them up.