Showing posts with label ORA-46697. Show all posts
Showing posts with label ORA-46697. Show all posts

Friday, September 29, 2023

DG per PDB with 23c on DBCS (Oracle Base Database)

At Oracle Cloud World (a few weeks ago as of writing this) 23c was announced as being available with DBCS.
 I've been wanting to test out DGPDB, which is Data Guard per PDB. This new feature was introduced with 21c and is covered in detail in this blog here .


Since I cover a lot of topics around backup, recovery, and availability this feature intrigued me.

NOTE: DGPDB is NOT supported with the tooling on DBCS and by testing it in your environment as I am going to do here your databases will not be supported.

First I am going to start with the documentation which can be found here.

My Environment :

I created 2 database services in ExaCS, both of which are 23c databases that are TDE encrypted (since they are in DBCS), and I am using a locally managed key (local wallet file).


CDB Name DB Unique Name PDB name
cdb1 cdb1_c83_iad pdb1
cdb2 cdb2_cdx_iad cdb2_pdb2


Step # 1 - Prepare the 2 CDBs

The first step is to prepare the 2 CDBs and follow the steps in the documentation

On both databases execute
  • ALTER DATABASE FORCE LOGGING;
  • ALTER DATABASE FLASHBACK ON;

On the source database execute
  • alter system set dg_broker_start=true scope=both;
  • alter system set log_archive_dest_1="LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=cdb1_c83_iad";
  • alter system set standby_file_management=auto scope=both; ;
On the destination database execute
  • alter system set dg_broker_start=true scope=both;
  • alter system set log_archive_dest_1="LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=cdb2_cdx_iad";
  • alter system set standby_file_management=auto scope=both;

Step # 2 - Configure tnsname.oras and SEPS wallets

First I am going to ensure that both tnsnames.ora files contains entries for both CDBs.
  • CDB1_C83_IAD
  • CDB2_CDX_IAD
Next I create a SEPS wallet on both nodes, and add entries for both databases
mkdir -p $ORACLE_HOME/dbs/wallets/dgpdb
mkstore -wrl $ORACLE_HOME/dbs/wallets/dgpdb -createALO
mkstore -wrl $ORACLE_HOME/dbs/wallets/dgpdb -createCredential cdb1_c83_iad sys {password}
mkstore -wrl $ORACLE_HOME/dbs/wallets/dgpdb -createCredential cdb2_cdx_iad sys {password}
mkstore -wrl $ORACLE_HOME/dbs/wallets/dgpdb -listCredential


Then I update the sqlnet.ora to use this wallet file.

WALLET_LOCATION =
    (SOURCE =
      (METHOD = FILE)
      (METHOD_DATA =
        (DIRECTORY = /u01/app/oracle/product/23.0.0.0/dbhome_1/dbs/wallets/dgpdb)
    )
)
SQLNET.WALLET_OVERRIDE = TRUE

And finally I test the connection
  • sqlplus /@cdb1_c83_iad as sysdba
  • sqlplus /@cdb2_cdx_iad as sysdba

Step # 3 - Create source and target configuration

Configure the source


dgmgrl /@cdb1_c83_iad <<EOF

CREATE CONFIGURATION 'cdb1_c83_iad' AS CONNECT IDENTIFIER IS cdb1_c83_iad; 
SHOW CONFIGURATION; 

EOF

Configure the target
dgmgrl /@cdb2_cdx_iad <<EOF

CREATE CONFIGURATION 'cdb2_cdx_iad' AS CONNECT IDENTIFIER IS cdb2_cdx_iad; 
SHOW CONFIGURATION; 

EOF

Step # 4 - Establish the connection between configurations

I ran the step in the documentation to configure the connection.


dgmgrl /@cdb1_c83_iad <<EOF

ADD CONFIGURATION 'cdb2_cdx_iad' CONNECT IDENTIFIER IS cdb2_cdx_iad; 
SHOW CONFIGURATION; 
ENABLE CONFIGURATION ALL; 
SHOW CONFIGURATION; 
EOF


And I validated that the output looked correct.

Configuration - cdb1_c83_iad

  Protection Mode: MaxPerformance
  Members:
  cdb1_c83_iad - Primary database
  cdb2_cdx_iad - Primary database in cdb2_cdx_iad configuration

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 1 second ago)

Step # 5 - Prepare the databases for DGPDB

The first step was to ensure that the PDBS are open in both database, which they are.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           READ WRITE NO
SQL>

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 CDB2_PDB1                      READ WRITE NO
SQL>


Then prepare the databases for DGPDB

 dgmgrl /@cdb1_c83_iad
DGMGRL for Linux: Release 23.0.0.0.0 - Production on Thu Sep 28 17:40:38 2023
Version 23.3.0.23.09

Copyright (c) 1982, 2023, Oracle and/or its affiliates.  All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected to "cdb1_c83_iad"
Connected as SYSDBA.
DGMGRL> EDIT CONFIGURATION PREPARE DGPDB;
Enter password for DGPDB_INT account at cdb1_c83_iad:
Enter password for DGPDB_INT account at cdb2_cdx_iad:

Prepared Data Guard for Pluggable Database at cdb2_cdx_iad.

Prepared Data Guard for Pluggable Database at cdb1_c83_iad.
DGMGRL>


Step # 6 - Restart the listeners and databases with the wallet location


On both the source and destination, I performed an "su - grid" and restarted the listeners.
I also restarted both databases. Until I restarted the databases I kept getting a 

 "Error: ORA-12578: A wallet file was not found or failed to open."

error message.


Step # 7 - Add the pluggable database to the destination

The next step is to add the pluggable database to the destination in dgmgrl.

DGMGRL> add pluggable database pdb1_stby at cdb2_cdx_iad source is pdb1 at cdb1_c83_iad  pdbfilenameconvert is "'+DATA/CDB1_C83_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/','+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/'"  ;
ORA-46697: Keystore password required.
Help: https://docs.oracle.com/error-help/db/ora-46697/


At this point by following the documentation, adding the pluggable database failed with  "ORA-46697: Keystore password required."

After some digging I found that I need to pass the wallet password (which is my sys password), and that the phrase needs to be in single quotes.


DGMGRL> add pluggable database pdb1_stby at cdb2_cdx_iad source is pdb1 at cdb1_c83_iad  pdbfilenameconvert is "'+DATA/CDB1_C83_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/','+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/'"  'keystore  IDENTIFIED BY "W3lCom3#123#123"';

Pluggable Database "PDB1_STBY" added


Step # 8 - List datafiles in the source PDB


     FILE#  NAME
----------  ------------------------------------------------------------------------------------------------------------------------
         8  +DATA/CDB1_C83_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/system.273.1148746431
         9  +DATA/CDB1_C83_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/sysaux.270.1148746437
        10  +DATA/CDB1_C83_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/undotbs1.271.1148746445
        12  +DATA/CDB1_C83_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/users.274.1148746543

Step # 9 - Copy the datafiles to the destination


First I logged onto the primary database and closed it.

 alter pluggable database pdb1 close;

I decided to use DBMS_FILE_TRANSFER.GET_FILE

Step 9a - create directory on source pointing to datafiles


SQL*Plus: Release 23.0.0.0.0 - Production on Fri Sep 29 11:33:11 2023
Version 23.3.0.23.09

Copyright (c) 1982, 2023, Oracle.  All rights reserved.


Connected to:
Oracle Database 23c EE High Perf Release 23.0.0.0.0 - Production
Version 23.3.0.23.09

SQL> create directory SOURCE_DUMP as '+DATA/CDB1_C83_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/';

Directory created.

SQL> grant read,write on directory SOURCE_DUMP to public;

Grant succeeded.

SQL>

Step 9b - Log onto ASM and precreate directory

ASMCMD> mkdir 
+data/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/datafile


Step 9c - create directory on target pointing to datafiles

[oracle@cdb1 admin]$ sqlplus /@cdb2_cdx_iad as sysdba

SQL*Plus: Release 23.0.0.0.0 - Production on Fri Sep 29 11:33:44 2023
Version 23.3.0.23.09

Copyright (c) 1982, 2023, Oracle.  All rights reserved.


Connected to:
Oracle Database 23c EE High Perf Release 23.0.0.0.0 - Production
Version 23.3.0.23.09

SQL> create directory TARGET_DUMP as '+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/';

grant read,write on directory TARGET_DUMP to public;



Directory created.

SQL> SQL>
Grant succeeded.

Step 9d - create database link on target  pointing to source database


SQL> SQL> SQL> create public database link SOURCEDB connect to system identified by {password} using 'CDB1_C83_IAD';

Database link created.


SQL> select sysdate from dual@SOURCEDB;
select sysdate from dual@SOURCEDB
                         *
ERROR at line 1:
ORA-02085: database link SOURCEDB.ZFSADMIN.VCN.ORACLEVCN.COM connects to
CDB1.ZFSADMIN.VCN.ORACLEVCN.COM
Help: https://docs.oracle.com/error-help/db/ora-02085/


SQL> show parameter global

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
allow_global_dblinks                 boolean     FALSE
global_names                         boolean     TRUE
global_txn_processes                 integer     1
SQL> alter system set global_names=false;

System altered.

SQL> select sysdate from dual@SOURCEDB;

SYSDATE
---------
29-SEP-23

SQL>

Step 9e - Get the datafiles from the source


SQL> set timing on
BEGIN
dbms_file_transfer.get_file('SOURCE_DUMP',
'system.273.1148746431',
'SOURCEDB',
'TARGET_DUMP',
'SYSTEM.DBF');
END;
/SQL>   2    3    4    5    6    7    8

PL/SQL procedure successfully completed.

Elapsed: 00:00:01.21

Step # 10 - Rename datafiles in the destination.


List the placeholder datafiles in  destination database
SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 CDB2_PDB1                      READ WRITE NO
         4 PDB1_STBY                      MOUNTED
SQL> alter session set container=pdb1_stby;

Session altered.

Elapsed: 00:00:00.02
SQL> select name from v$datafile;

NAME
--------------------------------------------------------------------------------
+DATA/MUST_RENAME_THIS_DATAFILE_8.4294967295.4294967295
+DATA/MUST_RENAME_THIS_DATAFILE_9.4294967295.4294967295
+DATA/MUST_RENAME_THIS_DATAFILE_10.4294967295.4294967295
+DATA/MUST_RENAME_THIS_DATAFILE_12.4294967295.4294967295

Elapsed: 00:00:00.02
SQL>

Match the file number in the destination files to the source datafiles and rename them.

SQL> alter database rename file '+DATA/MUST_RENAME_THIS_DATAFILE_8.4294967295.4294967295' to '+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/SYSTEM.dbf';

Database altered.

Elapsed: 00:00:00.04
SQL> alter database rename file '+DATA/MUST_RENAME_THIS_DATAFILE_9.4294967295.4294967295' to '+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/SYSAUX.dbf';

Database altered.

Elapsed: 00:00:00.02

SQL> alter database rename file '+DATA/MUST_RENAME_THIS_DATAFILE_10.4294967295.4294967295' to '+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/UNDOTBS1.dbf';

Database altered.

Elapsed: 00:00:00.05
SQL> alter database rename file '+DATA/MUST_RENAME_THIS_DATAFILE_12.4294967295.4294967295' to '+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/USERS.dbf';

Database altered.

Elapsed: 00:00:00.03
SQL> select name from v$datafile;

NAME
--------------------------------------------------------------------------------
+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/system.dbf
+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/sysaux.dbf
+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/undotbs1.dbf
+DATA/CDB2_CDX_IAD/066E9679DE2850DCE063A104000ABBA7/DATAFILE/users.dbf

Elapsed: 00:00:00.02
SQL>


Step # 11 - Bring over the TDE keys

Export the encryption keys from the source database

SQL> ADMINISTER KEY MANAGEMENT
  EXPORT ENCRYPTION KEYS WITH SECRET "export_secret"
  TO '/tmp/mykey'
  FORCE KEYSTORE
  IDENTIFIED BY "{password}";  2    3    4    5

keystore altered.



Import them into the target database
ADMINISTER KEY MANAGEMENT
  IMPORT KEYS WITH SECRET "export_secret"
  FROM '/tmp/mykeypdb1'
  force keystore
  IDENTIFIED BY "W3lCom3#123#123"
  WITH BACKUP;

SQL> SQL>   2    3    4    5    6
keystore altered.

SQL> SQL>

List the encryption keys on the  source database
PDB Name        Key ID                              Master Key ID
--------------- ----------------------------------- -------------------------
CDB$ROOT        35B424EFDB6D4F19BFF31B5D3DE0BFFA    ATW0JO/bbU8Zv/MbXT3gv/o=
PDB$SEED        00000000000000000000000000000000    AQAAAAAAAAAAAAAAAAAAAAA=
PDB1            1AAD19EB6F364F3BBF6FE226A25AF1E5    ARqtGetvNk87v2/iJqJa8eU=


List the encryption keys on the destination database
Master Key ID                                           Tag                  PDB Name        KEYSTORE_TYPE     Origin     Key Creation Time  Key Act. Time
------------------------------------------------------- -------------------- --------------- ----------------- ---------- ------------------ ------------------
AZ5CKuJLrU/Nv5M5pXQCFcEAAAAAAAAAAAAAAAAAAAAAAAAAAAAA                         CDB$ROOT        SOFTWARE KEYSTORE LOCAL      09/28/2023 09:09   09/28/2023 09:09
ATW0JO/bbU8Zv/MbXT3gv/oAAAAAAAAAAAAAAAAAAAAAAAAAAAAA                         CDB$ROOT        SOFTWARE KEYSTORE LOCAL      09/28/2023 09:10   09/28/2023 09:10
AVAymCcev0/Pv94Tiqw/P50AAAAAAAAAAAAAAAAAAAAAAAAAAAAA                         CDB2_PDB1       SOFTWARE KEYSTORE LOCAL      09/28/2023 09:12   09/28/2023 09:12
ARqtGetvNk87v2/iJqJa8eUAAAAAAAAAAAAAAAAAAAAAAAAAAAAA                         CDB2_PDB1       SOFTWARE KEYSTORE LOCAL      09/28/2023 09:13   09/28/2023 09:13




Step # 12 - Add the standby redo logs to the PDB and validate

I created standby redo logs in the PDB on Target database

select 'ALTER DATABASE ADD STANDBY LOGFILE thread ' ||a.thread# || ' group 1' ||a.group# || ' (''' ||substr(b.member,1,35) || 'standby_pdb1_1' || a.group# ||  ''') size ' || a.bytes ||';'
  from v$log a, v$logfile b
  where a.group#=b.group# ;SQL>   2    3

'ALTERDATABASEADDSTANDBYLOGFILETHREAD'||A.THREAD#||'GROUP1'||A.GROUP#||'('''||SUBSTR(B.MEMBER,1,35)||'STANDBY_PDB1_1'||A.GROUP#||'
----------------------------------------------------------------------------------------------------------------------------------
ALTER DATABASE ADD STANDBY LOGFILE thread 1 group 13 ('+RECO/CDB2_CDX_IAD/ONLINELOG/group_standby_pdb1_13') size 1073741824;
ALTER DATABASE ADD STANDBY LOGFILE thread 1 group 12 ('+RECO/CDB2_CDX_IAD/ONLINELOG/group_standby_pdb1_12') size 1073741824;
ALTER DATABASE ADD STANDBY LOGFILE thread 1 group 11 ('+RECO/CDB2_CDX_IAD/ONLINELOG/group_standby_pdb1_11') size 1073741824;

SQL> alter session set container=PDB1_STBY;

Session altered.

SQL> ALTER DATABASE ADD STANDBY LOGFILE thread 1 group 4 ('+RECO/CDB2_CDX_IAD/ONLINELOG/group_standby_pdb1_13') size 1073741824;

Database altered.


SQL> ALTER DATABASE ADD STANDBY LOGFILE thread 1 group 5 ('+RECO/CDB2_CDX_IAD/ONLINELOG/group_standby_pdb1_12') size 1073741824;

Database altered.

SQL>  ALTER DATABASE ADD STANDBY LOGFILE thread 1 group 6  ('+RECO/CDB2_CDX_IAD/ONLINELOG/group_standby_pdb1_11') size 1073741824;

Database altered.

SQL>

And then I validated the configuration

DGMGRL> VALIDATE PLUGGABLE DATABASE PDB1_STBY AT cdb2_cdx_iad;

  Ready for Switchover:    NO

  Data Guard Role:         Physical Standby
  Apply State:             Not Running
  Standby Redo Log Files:  3
  Source:                  PDB1 (con_id 3) at cdb1_c83_iad



Step # 13 - change the state of standby to APPLY-ON

I changed the state of the standby and checked the configuration.


DGMGRL> VALIDATE PLUGGABLE DATABASE PDB1_STBY AT cdb2_cdx_iad;

  Ready for Switchover:    NO

  Data Guard Role:         Physical Standby
  Apply State:             Not Running
  Standby Redo Log Files:  3
  Source:                  PDB1 (con_id 3) at cdb1_c83_iad


DGMGRL>
EDIT PLUGGABLE DATABASE PDB1_STBY AT cdb2_cdx_iad SET STATE='APPLY-ON';DGMGRL>
Succeeded.
DGMGRL> SHOW CONFIGURATION;

Configuration - cdb2_cdx_iad

  Protection Mode: MaxPerformance
  Members:
  cdb2_cdx_iad - Primary database
    Warning: ORA-16910: inconsistency detected for one or more pluggable databases

  cdb1_c83_iad - Primary database in cdb1_c83_iad configuration

Data Guard for PDB:   Enabled in TARGET role

Configuration Status:
SUCCESS   (status updated 37 seconds ago)

DGMGRL> SHOW PLUGGABLE DATABASE pdb1 AT cdb1_c83_iad;

Pluggable database - PDB1 at cdb1_c83_iad

  Data Guard Role:     Primary
  Con_ID:              3
  Active Target:       con_id 4 at cdb2_cdx_iad

Pluggable Database Status:
SUCCESS

DGMGRL> SHOW PLUGGABLE DATABASE pdb1_stby AT cdb2_cdx_iad ;

Pluggable database - PDB1_STBY at cdb2_cdx_iad

  Data Guard Role:     Physical Standby
  Con_ID:              4
  Source:              con_id 3 at cdb1_c83_iad
  Transport Lag:       20 hours 26 minutes 24 seconds (computed 9 seconds ago)
  Apply Lag:           (unknown)
  Intended State:      APPLY-ON
  Apply State:         Running
  Apply Instance:      cdb2
  Average Apply Rate:  (unknown)
  Real Time Query:     OFF

Pluggable Database Status:
SUCCESS

DGMGRL> VALIDATE PLUGGABLE DATABASE PDB1_STBY AT cdb2_cdx_iad;

  Ready for Switchover:    NO

  Data Guard Role:         Physical Standby
  Apply State:             Waiting for Redo Data
  Standby Redo Log Files:  3
  Source:                  PDB1 (con_id 3) at cdb1_c83_iad

DGMGRL>

Step # 14 - Perform a log switch at both databases



 sqlplus /@cdb1_c83_iad as sysdba

SQL*Plus: Release 23.0.0.0.0 - Production on Fri Sep 29 15:57:52 2023
Version 23.3.0.23.09

Copyright (c) 1982, 2023, Oracle.  All rights reserved.


Connected to:
Oracle Database 23c EE High Perf Release 23.0.0.0.0 - Production
Version 23.3.0.23.09

SQL> ALTER SYSTEM ARCHIVE LOG CURRENT;

System altered.

SQL> ALTER SYSTEM ARCHIVE LOG CURRENT;

System altered.

SQL> exit
Disconnected from Oracle Database 23c EE High Perf Release 23.0.0.0.0 - Production
Version 23.3.0.23.09
[oracle@cdb2 admin]$ sqlplus /@cdb2_cdx_iad as sysdba

SQL*Plus: Release 23.0.0.0.0 - Production on Fri Sep 29 15:58:11 2023
Version 23.3.0.23.09

Copyright (c) 1982, 2023, Oracle.  All rights reserved.


Connected to:
Oracle Database 23c EE High Perf Release 23.0.0.0.0 - Production
Version 23.3.0.23.09

SQL> ALTER SYSTEM ARCHIVE LOG CURRENT;

System altered.

SQL> ALTER SYSTEM ARCHIVE LOG CURRENT;

System altered.

SQL>


Step # 15 - Validate standby is applying redo.


DGMGRL> SHOW PLUGGABLE DATABASE pdb1_stby AT cdb2_cdx_iad ;

Pluggable database - PDB1_STBY at cdb2_cdx_iad

  Data Guard Role:     Physical Standby
  Con_ID:              4
  Source:              con_id 3 at cdb1_c83_iad
  Transport Lag:       0 seconds (computed 1 second ago)
  Apply Lag:           4 seconds (computed 1 second ago)
  Intended State:      APPLY-ON
  Apply State:         Running
  Apply Instance:      cdb2
  Average Apply Rate:  489 KByte/s
  Real Time Query:     OFF

Pluggable Database Status:
SUCCESS




SUMMARY :

This is an exciting new feature, and I am looking forward to see it expand.

Since a target database, can only have one source, I see this being useful to "pair" CDBs.
Instead of having 2 Primary CDBs in DC1 (for example) and 2 standby CDBs in DC2 you would have 2 Primary CDBs in in each Data Center, and you would be able to switch over any PDBs individually.


Since this is a paradigm change for RMAN, I am going to be following how this affects RMAN, the RMAN catalogs, and the ZDLRA/Recovery Service in it's support.


Finally :

It is all working, and for fun I am going to look at the destination settings on the source Database.

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
log_archive_dest_1                   string      LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=cdb1_c83_iad
log_archive_dest_2                   string      service="cdb2_cdx_iad", ASYNC NOAFFIRM delay=0 optional compression=disable max_failure=0 reopen=300 db_unique_name="cdb2_cdx_iad" net_timeout=30, valid_for=(online_logs)


Today I did have time to perform a switchover and a second switchover to bring it back.
Because my database is encrypted, I did import the keys from CDB2 into CDB1.  I am thinking when you implement this with TDE, you do need to keep the keys from both CDBs syncronized.
In OKV, I'm thinking you would want to possibly even use a single wallet. At the very least both primary databases would need access to both wallets.

OBSERVATIONS:


I will continue to update this post as I find useful observations.

Observation 1 :

I looked at the archive log using v$archived_logs view in the target (CDB2).
In it I only 6 archive logs were created.
SQL> select thread#,sequence#,dest_id,blocks,status from v$ARCHIVED_LOG;

   THREAD#  SEQUENCE#    DEST_ID     BLOCKS S
---------- ---------- ---------- ---------- -
         1          2          1     230208 A
         1          3          1     896391 A
         1          4          1          7 A
         1          5          1      35474 A
         1          6          1        387 A

I added a couple more PDBS and made changes to a second PDB (that didn't have a standby) and the original PDB (that had a standby).
I then created a workload in both PDBS and looked at the archive logs on the source.

   THREAD#  SEQUENCE#    DEST_ID     BLOCKS S
---------- ---------- ---------- ---------- -
         1          3          2       4894 A
         1          4          2     567681 A
         1          5          2         14 A
         1          6          2     871136 A
         1          7          2    1756544 A
         1          8          2    1847012 A
         1          9          2    1931501 A
         1         10          2    1901263 A
         1         11          2    1931491 A
         1         12          2    1931837 A
         1         13          2    1465002 A
         1         14          2    1955017 A
         1         15          2    1936946 A
         1         16          2    1685507 A
         1         17          2    1931499 A
         1         18          2    1808038 A
         1         19          2    1874455 A
         1         20          2    1910501 A
         1         21          2    2038138 A
         1         22          2    1812968 A
         1         23          2    1935941 A
         1         24          2    1939459 A
         1         25          2    2030413 A
         1         26          2    1934927 A

24 rows selected.

Then I looked at ASM on the target and I could ALL the archive logs on the target.


ASMCMD> ls
thread_1_seq_10.282.1148837541
thread_1_seq_11.283.1148837541
thread_1_seq_12.284.1148837557
thread_1_seq_13.285.1148837667
thread_1_seq_14.287.1148837943
thread_1_seq_15.288.1148837963
thread_1_seq_16.289.1148837979
thread_1_seq_17.290.1148837979
thread_1_seq_18.291.1148838023
thread_1_seq_19.292.1148838023
thread_1_seq_20.293.1148838101
thread_1_seq_21.294.1148838119
thread_1_seq_22.295.1148838137
thread_1_seq_23.296.1148838137
thread_1_seq_24.297.1148838183
thread_1_seq_25.298.1148838213
thread_1_seq_26.299.1148838237
thread_1_seq_3.275.1148831895
thread_1_seq_4.276.1148831899
thread_1_seq_5.274.1148831883
thread_1_seq_5.278.1148837115
thread_1_seq_6.277.1148836711
thread_1_seq_6.286.1148837681
thread_1_seq_7.279.1148837481
thread_1_seq_8.280.1148837499
thread_1_seq_9.281.1148837517


Observation 2 :

RMAN does not support this feature at this point.

 What I have discovered is that

  • Within the CDB, V$ARCHIVED_LOG does not recognize archive logs from the second primary needed to recovery the standby
  • Within the CDB, V$DATAFILE does not recognize the datafiles for the “standby database”
  • Within the standby PDB, V$DATAFILE recognizes the datafiles for that standby database, and only the standby datafiles
  • Within the standby PDB V$ARCHIVED_LOG only sees this archive logs for the standby database, and not the local archive logs produced
  • When executing RMAN for the CDB, only the locally produced archive logs are backed up
  • When executing RMAN for the CDB, only the primary datafiles are backed up.
  • When executing RMAN for the standby PDB, no datafiles are backed up.  “pluggable database PDB1_STBY does not have any data files”
  • When executing RMAN for the standby PDB, only the primary database’s locally produced archive logs are backed up.

 

SUMMARY – with DGPDB

 Within the PDB, the PDB can see the datafiles, and archive logs needed to recover with V$DATAFILE and V$ARCHIVED_LOG

 RMAN does not currently see the standby datafiles, or it’s archive logs to back them up.