flashback database遇到不能flashback的tablespace reset incarnation

How do we flashback a database to a guaranteed restore point (GRP) that has 2 tablespaces with flashback off after a RESETLOGS was done?

Flashback fails with the errors below.  It fails on the datafiles for the two tablespaces with flashback off.

RMAN-03002: failure of flashback command at 09/27/2013 15:14:24
ORA-38753: Cannot flashback data file 2229; no flashback log data.
ORA-01110: data file 2229: '+//datafile/.27472.826360617'

online情况下不给flashback

offline drop 掉不要这个文件也可以的

Solution

We need to restore the datafiles for the two tablespaces from a backup, offline those same files, flashback to GRP, online the files again then recover to the GRP SCN. 

Note: Because RESETLOGS was done we'll also have to reset the incarnation. Otherwise that step can be skipped.

1. Restore the datafiles for those two tablespaces from a backup.

Note: The restored files are dated Sept 26th while all other datafiles are current (Sept 28th). The GRP is Sept 27th.

STATUS      CHECKPOINT_CHANGE# FUZ CHECKPOINT_TIME                  COUNT(*)
------- ---------------------- --- ----------------------------- -----------
OFFLINE           104379205366 NO  26-SEP-2013 21:17:26                    1
OFFLINE           104379205549 NO  26-SEP-2013 21:19:06                    1
OFFLINE           104379205557 NO  26-SEP-2013 21:19:14                    1
OFFLINE           104379205610 NO  26-SEP-2013 21:19:57                    1
OFFLINE           104379206069 NO  26-SEP-2013 21:21:29                    1
OFFLINE           104379206276 NO  26-SEP-2013 21:22:23                    1
OFFLINE           104379206428 NO  26-SEP-2013 21:23:10                    1
OFFLINE           104379206445 NO  26-SEP-2013 21:23:21                    2
OFFLINE           104379206480 NO  26-SEP-2013 21:23:41                    2
OFFLINE           104379206520 NO  26-SEP-2013 21:24:16                    1
ONLINE            104398694790 YES 28-SEP-2013 08:42:03                 2221

11 rows selected.

Database is currently in MOUNT.

2. After the restore make sure those restored files are OFFLINE before the flashback is done or it will fail with the same errors.

ALTER DATABASE DATAFILE '+//datafile/.29237.826360609' offline;
ALTER DATABASE DATAFILE '+//datafile/.29345.826360597' offline;
ALTER DATABASE DATAFILE '+//datafile/.29341.826360579' offline;
ALTER DATABASE DATAFILE '+//datafile/.29338.826360575' offline;
ALTER DATABASE DATAFILE '+//datafile/.29337.826360571' offline;
ALTER DATABASE DATAFILE '+//datafile/.29238.826360611' offline;
ALTER DATABASE DATAFILE '+//datafile/.29239.826360615' offline;
ALTER DATABASE DATAFILE '+//datafile/.29240.826360615' offline;
ALTER DATABASE DATAFILE '+//datafile/.12744.826360617' offline;
ALTER DATABASE DATAFILE '+//datafile/.34180.826360617' offline;
ALTER DATABASE DATAFILE '+//datafile/.33245.826360617' offline;
ALTER DATABASE DATAFILE '+//datafile/.27472.826360617' offline;

3. Flashback to the guaranteed restore point (GRP).

SQL> FLASHBACK DATABASE TO RESTORE POINT '';

Starting flashback at 13-09-28 10:20
...
starting media recovery archived log for thread 1 with sequence 843 is already on disk as file +//archivelog/2013_09_27/thread_1_seq_843.33236.827237895
media recovery complete, elapsed time: 00:00:15
Finished flashback at 13-09-28 10:27

4. Online all the datafiles for the two tablespaces.

ALTER DATABASE DATAFILE '+//datafile/.29345.826360597' online;
ALTER DATABASE DATAFILE '+//datafile/.29341.826360579' online;
ALTER DATABASE DATAFILE '+//datafile/.29338.826360575' online;
ALTER DATABASE DATAFILE '+//datafile/.29337.826360571' online;
ALTER DATABASE DATAFILE '+//datafile/.29238.826360611' online;
ALTER DATABASE DATAFILE '+//datafile/.29239.826360615' online;
ALTER DATABASE DATAFILE '+//datafile/.29240.826360615' online;
ALTER DATABASE DATAFILE '+//datafile/.12744.826360617' online;
ALTER DATABASE DATAFILE '+//datafile/.34180.826360617' online;
ALTER DATABASE DATAFILE '+//datafile/.33245.826360617' online;
ALTER DATABASE DATAFILE '+//datafile/.27472.826360617' online;

5. Because the database was opened with resetlogs we have to reset the incarnation to before resetlogs was done.

RMAN> list incarnation;

List of Database Incarnations

DB Key Inc Key DB Name DB ID STATUS Reset SCN Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1       1            2189369331 PARENT 73600365279 11-12-28 01:45
2       2           2189369331 PARENT 104113554484 13-09-17 07:41 <<< We need to be in this incarnation to recover to 9-27-2013. Use the INC KEY.
3       3           2189369331 CURRENT 104390300886 13-09-28 07:04    9月28 reset过了

Note: The restored files are dated Sept 26th while all other datafiles are current (Sept 28th). The GRP is Sept 27th.

RMAN> reset database to incarnation 2;

database reset to incarnation 2

6. Do RECOVER UNTIL CHANGE using the SCN of the GRP +1（until 是-1）. The UNTIL recovers to right before the SCN specified.

SQL> select name,scn,time,database_incarnation#,guarantee_flashback_database,storage_size from v$restore_point;

NAME SCN TIME DATABASE_INCARNATION# GUARANTEE_FLASHBACK_DATABASE STORAGE_SIZE
-------------------- ------------------- -------------------------------------- --- ----------------
104390296851 27-SEP-13 12.14.11.000000000 PM 2 YES 15728640000

1 row selected.

SQL> RECOVER DATABASE UNTIL CHANGE 104390296852;

ORA-00279: change 104379205366 generated at 09/26/2013 21:17:26 needed for thread 1 ORA-00289:
suggestion : +//archivelog/2013_09_28/thread_1_seq_788.39097.827318825 ORA-00280:
change 104379205366 for thread 1 is in sequence #788 Specify log: {=suggested | filename | AUTO | CANCEL}
AUTO
ORA-00279: change 104379207088 generated at 09/26/2013 21:27:03 needed for thread 1 ORA-00289:
suggestion : +//archivelog/2013_09_28/thread_1_seq_789.23582.827318825 ORA-00280:
change 104379207088 for thread 1 is in sequence #789 ORA-00278: log file '+//archivelog/2013_09_28/thread_1_seq_788.39097.827318825' no longer needed for this recovery
....
ORA-00279: change 104390280294 generated at 09/27/2013 11:55:45 needed for thread 1 ORA-00289:
suggestion : +//archivelog/2013_09_27/thread_1_seq_843.33236.827237895 ORA-00280:
change 104390280294 for thread 1 is in sequence #843 ORA-00278: log file '+//archivelog/2013_09_27/thread_1_seq_842.21570.827236545' no longer needed for this recovery Log applied.
Media recovery complete.

其实这个有没有+1 都没有影响，因为整库recover的，最多就是整个DB前滚了1个scn

SYS> alter database open resetlogs;
Database altered.

7. To enable the flashback on tablespaces:

SQL> alter tablespace flashback on;

-----readonly 打开-------------但是不能是有问题的tablespace

Symptoms

Flashback is turned off for one or more tablespaces to avoid unnecessary flashback logs generation.

Now there is a need to do a 'FLASHBACK DATABASE' to some previous point in time to recover huge wrongly updated data
due to an incorrect run of batch job.

However, FLASHBACK DATABASE can not work if flashback is off for any of the tablespaces. It produces following errors :

SQL> flashback database to scn 8517809201835 ;
flashback database to scn 8517809201835
*
ERROR at line 1:
ORA-38753: Cannot flashback data file 8; no flashback log data.
ORA-01110: data file 8: '//.dbf'

While querying this datafile, we come to know that the flashback is turned off ( column flashback_on = NO ) for corresponding tablespace:

  

SQL> select a.file#, a.name file_name, b.ts#, b.name ts_name, b.flashback_on from v$datafile a, v$tablespace b where a.ts#=b.ts# ;

FILE#      FILE_NAME                   TS#        TS_NAME             FLA
---------- -------------------------------------------------- ---------- ------------------- ---
..
8          '//.dbf'     7                   NO  <==
..

  
9 rows selected.

Attempt to enable flashback for tablespace, fails with the same error :

SQL> alter tablespace flashback on ;
Tablespace altered.

SQL> flashback database to scn 8517809201835 ;
flashback database to scn 8517809201835
*
ERROR at line 1:
ORA-38753: Cannot flashback data file 8; no flashback log data.
ORA-01110: data file 8: '//.dbf'

Again, if we take this datafile offline, we can not  get it back when OPEN RESETLOGS is done after FLASHBACK DATABASE :

SQL> alter database datafile 8 offline ;

Database altered.

SQL> flashback database to scn 8517809201835 ;
flashback database to scn 8517809201835
*
ERROR at line 1:
ORA-38795: warning: FLASHBACK succeeded but OPEN RESETLOGS would get error below
ORA-01245: offline file 8 will be lost if RESETLOGS is done
ORA-01110: data file 6: '//.dbf'


Performing a TSPITR ( Tablespace Point in Time Recovery ) takes longer if multiple tablespaces are involved for recovery.

Cause

The main cause is that flashback logs are not available to flashback the datafile to previous point in time.

Error: ORA-38753
Text: Cannot flashback data file %s; no flashback log data.
---------------------------------------------------------------------------
Cause: An attempt to perform a FLASHBACK DATABASE failed because the
file does not have enough flashback log data to cover the time to
flash back. Either the file did not have flashback generation
enabled for it, or had flashback generation turned off for it some
time during the time span of the flashback.
Action: The file cannot be flashed back. The file must be taken offline
or the tablespace dropped before continuing with the FLASHBACK
DATABASE command.

There is a limitation of 'FLASHBACK DATABASE' if the flashback is disabled for tablespace. It is mentioned in Oracle Documentation :

CREATE TABLESPACE

"flashback_mode_clause

Use this clause in conjunction with the ALTER DATABASE FLASHBACK clause to specify whether the tablespace can participate in FLASHBACK DATABASE operations. This clause is useful if you have the database in FLASHBACK mode but you do not want Oracle Database to maintain Flashback log data for this tablespace.

This clause is not valid for temporary or undo tablespaces.

FLASHBACK ON
Specify FLASHBACK ON to put the tablespace in FLASHBACK mode. Oracle Database will save Flashback log data for this tablespace and the tablespace can participate in a FLASHBACK DATABASE operation. If you omit the flashback_mode_clause, then FLASHBACK ON is the default.

FLASHBACK OFF
Specify FLASHBACK OFF to take the tablespace out of FLASHBACK mode. Oracle Database will not save any Flashback log data for this tablespace. You must take the datafiles in this tablespace offline or drop them prior to any subsequent FLASHBACK DATABASE operation. Alternatively, you can take the entire tablespace offline. In either case, the database does not drop existing Flashback logs."
 

Solution

The Workaround to this problem is to do a 'FLASHBACK DATABASE' and open the database in READ ONLY mode to export the needed data. Then shutdown database
and recover back to current point in time using archived logs. This is a faster option than TSPITR ( Tablespace Point in Time Recovery )
in case if many tablespaces are needed to be brought back to some previous point in time.

Following is an in-house test-case to achieve the same. In this test-case, the flashback is turned off for tablespace 'APP' for which
the underlying datafile is file# 8 named '//app01.dbf'.
 

Example Test-case as a workaround for ORA-38753

SQL> select flashback_on from v$database ;

FLASHBACK_ON
------------------
YES

1 row selected.

SQL> select name, flashback_on from v$tablespace where flashback_on='NO';

NAME              FLA
------------------------------ ---
  NO
 

9 rows selected.

SQL> alter database open ;

Database altered.

SQL> create table . ( a number ) ;

Table created.

SQL> insert into . values ( 1 ) ;

1 row created.

SQL> insert into . values ( 2) ;

1 row created.

SQL> commit ;

Commit complete.

SQL> col systimestamp format a50
SQL> /

SYSTIMESTAMP CURRENT_SCN
-------------------------------------------------- ---------------
20-DEC-09 09.17.59.820753 PM +05:30 8517809201835

1 row selected.

SQL> alter system switch logfile ;

System altered.

SQL> /

System altered.

SQL> /

System altered.

SQL> /

System altered.

SQL> /

System altered.

SQL> alter tablespace flashback off ;

Tablespace altered.

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup mount
ORACLE instance started.

Total System Global Area 264241152 bytes
Fixed Size 1218844 bytes
Variable Size 109053668 bytes
Database Buffers 146800640 bytes
Redo Buffers 7168000 bytes
Database mounted.
SQL> alter database open ;

Database altered.

SQL> drop table . purge ;

Table dropped.

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup mount
ORACLE instance started.

Total System Global Area 264241152 bytes
Fixed Size 1218844 bytes
Variable Size 109053668 bytes
Database Buffers 146800640 bytes
Redo Buffers 7168000 bytes
Database mounted.
SQL> flashback database to scn 8517809201835 ;
flashback database to scn 8517809201835
*
ERROR at line 1:
ORA-38753: Cannot flashback data file 8; no flashback log data.
ORA-01110: data file 8: '//.dbf'

SQL> select a.file#, a.name file_name, b.ts#, b.name ts_name, b.flashback_on from v$datafile a, v$tablespace b where a.ts#=b.ts# ;

FILE# FILE_NAME             TS#            TS_NAME                FLA
---------- -------------------------------------------------- ---------- ------------------- ---
..
8 //.dbf        7          NO
 

9 rows selected.

SQL> alter database datafile 8 offline ;
Database altered.

SQL> flashback database to scn 8517809201835 ;

Flashback complete.

SQL> alter database open read only ;

Database altered.

SQL> select * from .; --- <
A
---------------
1
2

2 rows selected.

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup mount
ORACLE instance started.

Total System Global Area 264241152 bytes
Fixed Size 1218844 bytes
Variable Size 109053668 bytes
Database Buffers 146800640 bytes
Redo Buffers 7168000 bytes
Database mounted.
SQL> recover database ;  不用flashback前推，因为没有resetlogs
ORA-00279: change 8517809201836 generated at 12/20/2009 21:18:00 needed for thread 1
ORA-00289: suggestion :
//archivelog/2009_12_20/o1_mf_1_269_%u_.arc
ORA-00280: change 8517809201836 for thread 1 is in sequence #269

Specify log: {=suggested | filename | AUTO | CANCEL}
auto
ORA-00279: change 8517809201842 generated at 12/20/2009 21:18:09 needed for thread 1
ORA-00289: suggestion :
//archivelog/2009_12_20/o1_mf_1_270_%u_.arc
ORA-00280: change 8517809201842 for thread 1 is in sequence #270
ORA-00278: log file
'//archivelog/2009_12_20/o1_mf_1_269_5lwkssds_.arc' no longer needed for this recovery

ORA-00279: change 8517809201847 generated at 12/20/2009 21:18:10 needed for thread 1
ORA-00289: suggestion :
//archivelog/2009_12_20/o1_mf_1_271_%u_.arc
ORA-00280: change 8517809201847 for thread 1 is in sequence #271
ORA-00278: log file
'//archivelog/2009_12_20/o1_mf_1_270_5lwkstj2_.arc' no longer needed for this recovery

Log applied.
Media recovery complete. <
SQL> alter database open ;

Database altered.

SQL> alter database datafile 8 online ;

Database altered.

SQL> select * from .;
*
ERROR at line 1:
ORA-00942: table or view does not exist