fast incremental backup failed on standby database
一套Linux上的11.1.0.7的physical standby物理备库在使用fast incremental backup进行高于0级的增量备份时会出现ORA-19648错误，其出错记录如下：
RMAN> backup incremental level 1 database; Starting backup at 22-MAR-11 using channel ORA_DISK_1 using channel ORA_DISK_2 channel ORA_DISK_1: starting incremental level 1 datafile backup set channel ORA_DISK_1: specifying datafile(s) in backup set input datafile file number=00002 name=/standby/oradata/SBDB2/datafile/o1_mf_sysaux_22m6ov92_.dbf input datafile file number=00003 name=/standby/oradata/SBDB2/datafile/o1_mf_undotbs1_23m6ovap_.dbf input datafile file number=00006 name=/standby/oradata/SBDB2/datafile/o1_mf_enc_25m6ovba_.dbf channel ORA_DISK_1: starting piece 1 at 22-MAR-11 channel ORA_DISK_2: starting incremental level 1 datafile backup set channel ORA_DISK_2: specifying datafile(s) in backup set input datafile file number=00001 name=/standby/oradata/SBDB2/datafile/o1_mf_system_21m6ov92_.dbf input datafile file number=00005 name=/standby/oradata/SBDB2/datafile/o1_mf_example_24m6ovar_.dbf input datafile file number=00004 name=/standby/oradata/SBDB2/datafile/o1_mf_users_26m6ovba_.dbf channel ORA_DISK_2: starting piece 1 at 22-MAR-11 RMAN-03009: failure of backup command on ORA_DISK_1 channel at 03/22/2011 20:20:28 ORA-19648: datafile 2: incremental-start SCN equals checkpoint SCN ORA-19640: datafile checkpoint is SCN 1249353 time 03/09/2011 05:50:27 continuing other job steps, job failed will not be re-run RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03009: failure of backup command on ORA_DISK_2 channel at 03/22/2011 20:22:33 ORA-19648: datafile 1: incremental-start SCN equals checkpoint SCN ORA-19640: datafile checkpoint is SCN 1249352 time 03/09/2011 05:50:26经过分析该ORA-19640->ORA-19648是由11.1.0.7上的Bug引起的，MOS已经确认其为Bug 9288598:
Affects: Product (Component) Oracle Server (Rdbms) Range of versions believed to be affected Versions BELOW 12.1 Versions confirmed as being affected 11.1.0.7 Platforms affected Generic (all / most platforms affected) Fixed: This issue is fixed in 12.1 (Future Release) 11.2.0.2 (Server Patch Set) An RMAN Backup incremental level 1 can fail with ORA-19648 / ORA-19640 if the standby database is open read only rather than just skipping the datafile. PROBLEM: -------- Following error occurring during incremental backup: Starting backup at 02-DEC-2009 20:37:44 allocated channel: ORA_SBT_TAPE_1 channel ORA_SBT_TAPE_1: SID=154 device type=SBT_TAPE channel ORA_SBT_TAPE_1: Veritas NetBackup for Oracle - Release 6.5 (2008052301) channel ORA_SBT_TAPE_1: starting incremental level 1 datafile backup set channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set input datafile file number=00002 name=/data/ora_data01/KAHCB4P/KAHCB4P_sysaux_01.dbf input datafile file number=00001 name=/data/ora_data01/KAHCB4P/KAHCB4P_system_01.dbf input datafile file number=00003 name=/data/ora_data01/KAHCB4P/KAHCB4P_undo_01.dbf input datafile file number=00004 name=/data/ora_data01/KAHCB4P/KAHCB4P_users_01.dbf input datafile file number=00005 name=/data/ora_data01/KAHCB4P/KAHCB4P_tools_01.dbf input datafile file number=00006 name=/data/ora_data01/KAHCB4P/KAHCB4P_backup_test_01.dbf channel ORA_SBT_TAPE_1: starting piece 1 at 02-DEC-2009 20:38:01 RMAN-3009: failure of backup command on ORA_SBT_TAPE_1 channel at 12/02/2009 20:38:02 ORA-19648: datafile 2: incremental-start SCN equals checkpoint SCN ORA-19640: datafile checkpoint is SCN 3911695 time 11/27/2009 12:36:48 continuing other job steps, job failed will not be re-run channel ORA_SBT_TAPE_1: starting incremental level 1 datafile backup set channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set including current control file in backup set including current SPFILE in backup set DIAGNOSTIC ANALYSIS: -------------------- Same problem as Bug 6903819 Requested backport, but this hasn't fixed the problem. Confirmed that the patch was installed correctly and relinked successfully. RELEASE NOTES: ]]Backup incremental leve 1 fails with ORA-19648 and ORA-19640 when standby ]]database opened for read only. *** 02/14/10 02:43 pm *** (ADD: Impact/Symptom->FEATURE UNUSABLE ) REDISCOVERY INFORMATION: Backup incremental level 1 fails with ORA-19648 and the standby database is open read-only. WORKAROUND: None可以从上述Note中看到该Bug需要到11.2.0.2中才得到fix，那么在11.1.0.7上我们有什么办法能解决或者绕过该问题吗？ 经过测试，我得到2种workaround的方法： 该ORA-19648错误首先可以通过在物理备库(physical standby)上停止介质恢复(MRP)进程的方式来workaround：
SQL> alter database recover managed standby database cancel; Database altered. RMAN> backup incremental level 1 database; Starting backup at 22-MAR-11 using target database control file instead of recovery catalog allocated channel: ORA_DISK_1 channel ORA_DISK_1: SID=141 device type=DISK allocated channel: ORA_DISK_2 channel ORA_DISK_2: SID=140 device type=DISK channel ORA_DISK_1: starting incremental level 1 datafile backup set channel ORA_DISK_1: specifying datafile(s) in backup set input datafile file number=00002 name=/standby/oradata/SBDB2/datafile/o1_mf_sysaux_22m6ov92_.dbf input datafile file number=00004 name=/standby/oradata/SBDB2/datafile/o1_mf_users_26m6ovba_.dbf input datafile file number=00006 name=/standby/oradata/SBDB2/datafile/o1_mf_enc_25m6ovba_.dbf channel ORA_DISK_1: starting piece 1 at 22-MAR-11 channel ORA_DISK_2: starting incremental level 1 datafile backup set channel ORA_DISK_2: specifying datafile(s) in backup set input datafile file number=00001 name=/standby/oradata/SBDB2/datafile/o1_mf_system_21m6ov92_.dbf input datafile file number=00003 name=/standby/oradata/SBDB2/datafile/o1_mf_undotbs1_23m6ovap_.dbf input datafile file number=00005 name=/standby/oradata/SBDB2/datafile/o1_mf_example_24m6ovar_.dbf channel ORA_DISK_2: starting piece 1 at 22-MAR-11 channel ORA_DISK_1: finished piece 1 at 22-MAR-11 piece handle=/standby/backup/2km7suui_1_1.bak tag=TAG20110322T212538 comment=NONE channel ORA_DISK_1: backup set complete, elapsed time: 00:00:33 channel ORA_DISK_2: finished piece 1 at 22-MAR-11 piece handle=/standby/backup/2lm7suv1_1_1.bak tag=TAG20110322T212538 comment=NONE channel ORA_DISK_2: backup set complete, elapsed time: 00:00:27 Finished backup at 22-MAR-11 Starting Control File and SPFILE Autobackup at 22-MAR-11 piece handle=/standby/backup/c-157018592-20110322-01.bak comment=NONE Finished Control File and SPFILE Autobackup at 22-MAR-11该ORA-19648错误也可以通过禁用fast incremental backup的块跟踪(block change tracking)特性来绕过问题，当然在可能的情况下我们更推荐使用上面那种方法，因为毕竟还可以利用到fast incremental backup特性：
SQL> alter database recover managed standby database using current logfile disconnect; Database altered. SQL> select process,status from v$managed_standby; PROCESS STATUS --------- ------------ ARCH CONNECTED ARCH CONNECTED ARCH CONNECTED ARCH CONNECTED MRP0 APPLYING_LOG SQL> select status from v$block_change_tracking; STATUS ---------- ENABLED SQL> alter database disable block change tracking; Database altered. RMAN> backup incremental level 2 database; Starting backup at 22-MAR-11 using channel ORA_DISK_1 using channel ORA_DISK_2 channel ORA_DISK_1: starting incremental level 2 datafile backup set channel ORA_DISK_1: specifying datafile(s) in backup set input datafile file number=00002 name=/standby/oradata/SBDB2/datafile/o1_mf_sysaux_22m6ov92_.dbf input datafile file number=00004 name=/standby/oradata/SBDB2/datafile/o1_mf_users_26m6ovba_.dbf input datafile file number=00006 name=/standby/oradata/SBDB2/datafile/o1_mf_enc_25m6ovba_.dbf channel ORA_DISK_1: starting piece 1 at 22-MAR-11 channel ORA_DISK_2: starting incremental level 2 datafile backup set channel ORA_DISK_2: specifying datafile(s) in backup set input datafile file number=00001 name=/standby/oradata/SBDB2/datafile/o1_mf_system_21m6ov92_.dbf input datafile file number=00003 name=/standby/oradata/SBDB2/datafile/o1_mf_undotbs1_23m6ovap_.dbf input datafile file number=00005 name=/standby/oradata/SBDB2/datafile/o1_mf_example_24m6ovar_.dbf channel ORA_DISK_2: starting piece 1 at 22-MAR-11 channel ORA_DISK_1: finished piece 1 at 22-MAR-11 piece handle=/standby/backup/2nm7sv8d_1_1.bak tag=TAG20110322T213052 comment=NONE channel ORA_DISK_1: backup set complete, elapsed time: 00:00:15 channel ORA_DISK_2: finished piece 1 at 22-MAR-11 piece handle=/standby/backup/2om7sv8s_1_1.bak tag=TAG20110322T213052 comment=NONE channel ORA_DISK_2: backup set complete, elapsed time: 00:00:15 Finished backup at 22-MAR-11 Starting Control File and SPFILE Autobackup at 22-MAR-11 piece handle=/standby/backup/c-157018592-20110322-02.bak comment=NONE Finished Control File and SPFILE Autobackup at 22-MAR-11