Sunday, February 25, 2018

Is corruption always true corruption ?


Well the short answer is that there are different types of corruption.  I am writing this blog because I recently ran across “nologging” corruption which is considered softl corruption, and the handling of this type of corruption has changed across versions (which I will cover at the end of this article).

First, how does this happen ?  It can be seen in a physical standby, but you can also get this type of corruption in a Primary database.  I will go through how to reproduce this type of corruption shortly. 

Next, what is nologging ?  Nologging processes are direct path updates to the database that do not contain detailed log information.  Notice I used the word “detailed” .  This is because some logging is captured as to what blocks are updated, but the individual updates are not captured.
Nologging is most common in datawarehouse load processes (ELT)  that are part of workflow that can be restarted.  Often tables are created during this processing that are only kept as part of the processing.  Nologging can also be used for performing large inserts into existing tables.  Because this type of processing tends to be “logging intensive”, and steps can be re-run, nologging is utilized for these objects.  Nologging can speed up processing by performing limited logging.  The downside of nologging is that for objects updated with nologging, there is no point in time recovery capability.  The object can only be restore/recovered to the point where a backup is taken (full or incremental).  More on that later.

I will start by showing a nologging workflow.

Below are the steps on how to reproduce a nologging test.


  1)      Ensure that force_logging=false   ---  If FORCE_LOGGING is turned on, any nologging processing is handled as logging



SQL> select force_logging from v$database;


FOR
---
NO


2)   Create a nologging table


SQL>  create table bgrenn.test nologging as select * from dba_objects where 0=1;

Table created.
 

3)      Ensure the new table is nologging


SQL> select owner,table_name,logging from dba_tables where owner='BGRENN';

OWNER                          TABLE_NAME                     LOG
------------------------------ ------------------------------ ---
BGRENN                         TEST                           NO


4)      Perform a full backup of the database


RMAN> backup incremental level 0 database;

Starting backup at 23-february -2018 12:45:45
using target database control file instead of recovery catalog

allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=1069 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_1: starting incremental level 0 datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00005 name=/oradata/db1/soe.dbf
channel ORA_DISK_7: starting incremental level 0 datafile backup set
channel ORA_DISK_7: specifying datafile(s) in backup set
input datafile file number=00004 name=/oradata/db1/users01.dbf
channel ORA_DISK_7: starting piece 1 at 23-february -2018 12:46:22
channel ORA_DISK_8: starting incremental level 0 datafile backup set
channel ORA_DISK_8: specifying datafile(s) in backup set
including current SPFILE in backup set
channel ORA_DISK_8: starting piece 1 at 23-february -2018 12:46:28
channel ORA_DISK_2: finished piece 1 at 23-february -2018 12:46:30

….
Finished backup at 23-february -2018 12:50:01


5)      Update the table no logging using append hint

SQL> insert into /*+ append */ bgrenn.test select * from dba_objects nologging;

68947 rows created.

SQL> Commit;



6)      Switch the logfile to ensure the changes are written to archive logs.

SQL> alter system switch logfile;




OK. Now we have done a Full backup of the database, and performed a nologging change to my table “bgrenn.test”.  I did a log switch to ensure the change is written to the archive log.

The next step is to reproduce the nologging “soft corruption” through a restore.
At this point, blocks containing my table were inserted into, but the actual changes were not logged.  The block numbers were written to the log file, and on recovery these blocks will be marked as being changed.


1)      Check for block corruption before restoring

SQL> select * from v$database_block_corruption;

no rows selected


2)       Retart the database mount and restore the database

RMAN> connect target /

connected to target database: DB16 (DBID=3618808394)

RMAN> shutdown immediate;

using target database control file instead of recovery catalog
startup mount;
database closed
database dismounted
Oracle instance shut down
using target database control file instead of recovery catalog
startup mount;
database closed
database dismounted
Oracle instance shut down


RMAN> startup mount;

connected to target database (not started)
Oracle instance started
database mounted

Total System Global Area   14564409344 bytes

Fixed Size                     2149720 bytes
Variable Size               6308233896 bytes
Database Buffers            8187281408 bytes
Redo Buffers                  66744320 bytes

RMAN> restore database;

Starting restore at 23-february -2018 12:55:54
allocated channel: ORA_SBT_TAPE_1
channel ORA_SBT_TAPE_1: SID=1110 device type=SBT_TAPE
channel ORA_SBT_TAPE_1: Oracle Secure Backup
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=1109 device type=DISK
channel ORA_DISK_5: starting datafile backup set restore
channel ORA_DISK_5: specifying datafile(s) to restore from backup set
channel ORA_DISK_5: restoring datafile 00004 to /oradata/db1116/users01.dbf
channel ORA_DISK_5: reading from backup piece /u01/app/oracle/flash_recovery_area/DB1116/backupset/2018_02_23/o1_mf_nnnd0_TAG20180223T12454
7_f90vwp3j_.bkp
Finished restore at 23-february -2018 12:57:41






3)      Recover database


RMAN> recover database;

Starting recover at 23-february -2018 12:58:22
using channel ORA_SBT_TAPE_1
using channel ORA_DISK_1
using channel ORA_DISK_2
using channel ORA_DISK_3
using channel ORA_DISK_4
using channel ORA_DISK_5
using channel ORA_DISK_6
using channel ORA_DISK_7
using channel ORA_DISK_8

starting media recovery
media recovery complete, elapsed time: 00:00:01

Finished recover at 23-february -2018 12:58:23

RMAN> alter database open;

database opened

RMAN>




4)      Check for corruption after restoring the database


SQL> select * from v$database_block_corruption;

no rows selected


5)      Select from the table in which we ran our nologging process



SQL> select * from bgrenn.test;
select * from bgrenn.test
                     *
ERROR at line 1:
ORA-01578: ORACLE data block corrupted (file # 4, block # 60)
ORA-01110: data file 4: '/oradata/db1/users01.dbf'
ORA-26040: Data block was loaded using the NOLOGGING option




6)      Check corruption again


SQL> select * from v$database_block_corruption;

no rows selected



7)       Validate datafile



RMAN> validate datafile 4;

Starting validate at 23-february -2018 14:15:36
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=1107 device type=DISK
channel ORA_DISK_8: SID=1073 device type=DISK
channel ORA_DISK_1: starting validation of datafile
channel ORA_DISK_1: specifying datafile(s) for validation
input datafile file number=00004 name=/oradata/db1/users01.dbf
channel ORA_DISK_1: validation complete, elapsed time: 00:00:01
List of Datafiles
=================



File Status Marked Corrupt Empty Blocks Blocks Examined High SCN
---- ------ -------------- ------------ --------------- ----------
4    OK     1016           307          1440            1760824163
  File Name: /oradata/db1/users01.dbf
  Block Type Blocks Failing Blocks Processed
  ---------- -------------- ----------------
  Data       0              63             
  Index      0              2              
  Other      0              1068           

Finished validate at 23-february -2018 14:15:42


Now this is where it gets interesting between version of Oracle.  Oracle 10g/11.1 reports this soft corruption differently from Oracle 11.2, and  both of these report it differently from 12.1+

Oracle 11.1   -
1)      Check for corruption in v$DATABASE_BLOCK_CORRUPTION  -- NOTE the corrupt_type only reports "CORRUPTION"



SQL> select * from v$database_block_corruption;

     FILE#     BLOCK#     BLOCKS CORRUPTION_CHANGE# CORRUPTIO
---------- ---------- ---------- ------------------ ---------
         4         60         13         1760822701 CORRUPT
         4         74         15         1760822716 CORRUPT
         4         90         15         1760822716 CORRUPT
         4        106         15         1760822730 CORRUPT
         4        122         15         1760822730 CORRUPT
         4        138         15         1760822745 CORRUPT
         4        154         15         1760822745 CORRUPT
         4        170         15         1760822759 CORRUPT
         4        267        126         1760822759 CORRUPT
         4        395        126         1760822763 CORRUPT
         4        523        126         1760822767 CORRUPT
         4        651        126         1760822771 CORRUPT
         4        779        126         1760822775 CORRUPT
         4        907        126         1760822779 CORRUPT
         4       1035        126         1760822784 CORRUPT
         4       1163         16         1760822788 CORRUPT


2)      Definition  of  CORRUPTION_TYPE

·         ALL ZERO - Block header on disk contained only zeros. The block may be valid if it was never filled and if it is in an Oracle7 file. The buffer will be reformatted to the Oracle8 standard for an empty block.
·         FRACTURED - Block header looks reasonable, but the front and back of the block are different versions.
·         CHECKSUM - optional check value shows that the block is not self-consistent. It is impossible to determine exactly why the check value fails, but it probably fails because sectors in the middle of the block are from different versions.
·         CORRUPT - Block is wrongly identified or is not a data block (for example, the data block address is missing)
·         LOGICAL - Specifies the range is for logically corrupt blocks. CORRUPTION_CHANGE# will have a nonzero values

3)      OEM Schedule Backup screen shows corruption

Oracle 11.2   -
1)      Check for corruption in v$DATABASE_BLOCK_CORRUPTION  -- NOTE 11.2 reports the soft corrupt in the view as "NOLOGGING" corruption.



SQL> select * from v$database_block_corruption;

     FILE#     BLOCK#     BLOCKS CORRUPTION_CHANGE# CORRUPTIO
---------- ---------- ---------- ------------------ ---------
         4         60         13         1760822701 NOLOGGING
         4         74         15         1760822716 NOLOGGING
         4         90         15         1760822716 NOLOGGING
         4        106         15         1760822730 NOLOGGING
         4        122         15         1760822730 NOLOGGING
         4        138         15         1760822745 NOLOGGING
         4        154         15         1760822745 NOLOGGING
         4        170         15         1760822759 NOLOGGING
         4        267        126         1760822759 NOLOGGING
         4        395        126         1760822763 NOLOGGING
         4        523        126         1760822767 NOLOGGING
         4        651        126         1760822771 NOLOGGING
         4        779        126         1760822775 NOLOGGING
         4        907        126         1760822779 NOLOGGING
         4       1035        126         1760822784 NOLOGGING
         4       1163         16         1760822788 NOLOGGING



2)      Definition  of  CORRUPTION_TYPE
·         ALL ZERO - Block header on disk contained only zeros. The block may be valid if it was never filled and if it is in an Oracle7 file. The buffer will be reformatted to the Oracle8 standard for an empty block.
·         FRACTURED - Block header looks reasonable, but the front and back of the block are different versions.
·         CHECKSUM - optional check value shows that the block is not self-consistent. It is impossible to determine exactly why the check value fails, but it probably fails because sectors in the middle of the block are from different versions.
·         CORRUPT - Block is wrongly identified or is not a data block (for example, the data block address is missing)
·         LOGICAL - Block is logically corrupt
·         NOLOGGING - Block does not have redo log entries (for example, NOLOGGING operations on primary database can introduce this type of corruption on a physical standby)

3)      OEM Schedule Backup screen shows corruption



Oracle 12.1   -
1)      Check for corruption in v$DATABASE_BLOCK_CORRUPTION  -- NOTE - 12.1 does not report it as corruption, but is reported in a new view  V$NONLOGGED_BLOCK


SQL> select * from v$database_block_corruption;

no rows selected



SQL> select file#,block#,blocks from v$nonlogged_block

     FILE#     BLOCK#     BLOCKS
---------- ---------- ----------
         6       6786        126
         6       6914        126
         6       7042        126
         6       7170        126
         6       7298        126
         6       7426        126
         6       7554        126
         6       7682        126
         6       7810        126
         6       7938        126
         6       8066        126

     FILE#     BLOCK#     BLOCKS
---------- ---------- ----------
         6       8194        126
         6       8322        126
         6       8512         64

47 rows selected.




2)      Definition  of  CORRUPTION_TYPE
·         ALL ZERO - Block header on disk contained only zeros. The block may be valid if it was never filled and if it is in an Oracle7 file. The buffer will be reformatted to the Oracle8 standard for an empty block.
·         FRACTURED - Block header looks reasonable, but the front and back of the block are different versions.
·         CHECKSUM - optional check value shows that the block is not self-consistent. It is impossible to determine exactly why the check value fails, but it probably fails because sectors in the middle of the block are from different versions.
·         CORRUPT - Block is wrongly identified or is not a data block (for example, the data block address is missing)
·         LOGICAL - Block is logically corrupt

3)      OEM Schedule Backup screen

Nothing appears


Now we have seen how to recreate “soft corruption” caused by nologging.  I have also shown how this is displayed in different versions of Oracle.

There are a few items to note that I have learned from this testing.

·        This is considered “soft corruption” so it is not reported when restoring a database.  This makes it very hard detect.

·        The ZDLRA does validation during backups, but since this is “soft corruption”, the database is backed up without no alerting.

·        OEM reports this corruption differently between versions.  With version 12.1 it is no longer reported in V$DATABASE_BLOCK_CORRUPTION, so OEM does not alert on this.

How to avoid Nologging corruption.

Ensure that you schedule backups when there isn’t any nologging operations occurring.This is a situation where the ZDLRA shines.  You can take an incremental backup before and after your nologging process then you have the capability to perform a full restore from either of these checkpoints.