Suspicion was some sort of data block corruption. Rman backup logs had not reported any corrpution, but dbverify did find some corrupt blocks. Those blocks were not associated with any record in dba_extents though.

All that is curious enough, but it is the fix which was the real surprise. Running:
SQL> alter system flush buffer_cache ;
Fixed the problem. Implication of that is that there is intermittent data block corruption, and that table was corrupt at the time it was first read into cache, and clean again (on disk) by the time I came along to look at it.

Dbverify later on also switched over to giving the datafiles a clean bill of health.

What’s the root cause? Don’t know (yet), but intermittent disk or memory hardware failure seems the most likely. Nothing reported in server logs though.

This on Oracle 10.1.0.4.0 Enterprise Edition on Microsoft Windows Server 2003 Standard Edition Service Pack 1 on Intel Xeon 3.2GHz.

The dbverify syntax I used was like:
dbv file=D:\ORACLE\ORADATA\SYSTEM01.DBF logfile=4.log
I didn’t have to specify block size because database was at the default block size for that version of dbv (seen with dbv help=y).

But they are somewhat misleading – ora-12514 implies listener issues, and the listener was actually working ok on the standby server.

Drcdbname.log had the right information:

DG 2007-03-07-07:12:42 0 2 0 RSM detected log transport problem: log transport for database 'dbase_DR' has the following error.
DG 2007-03-07-07:12:42 0 2 0 ORA-12514: Message 12514 not found; No message file for product=RDBMS, facility=ORA
DG 2007-03-07-07:12:42 0 2 0 RSM0: HEALTH CHECK ERROR: ORA-16737: the redo transport service for standby database "dbase_DR" has an error
DG 2007-03-07-07:12:42 0 2 0 NSV1: Failed to connect to remote database dbase_DR. Error is ORA-01034
DG 2007-03-07-07:12:42 0 2 0 NSV1: Message is dropped.
DG 2007-03-07-07:12:42 0 2 606440720 DMON: Database dbase_DR returned ORA-01034
DG 2007-03-07-07:12:42 0 2 606440720 for opcode = CTL_GET_STATUS, phase = BEGIN, req_id = 1.1.606440720
DG 2007-03-07-07:12:42 0 2 606440720 Operation CTL_GET_STATUS cancelled during phase 2, error = ORA-16778
DG 2007-03-07-07:12:42 0 2 606440720 Operation CTL_GET_STATUS cancelled during phase 2, error = ORA-16778

Checking Data Guard Manager on live server showed the same ORA-1034 error, with description added in:

C:\>dgmgrl
DGMGRL for 32-bit Windows: Version 10.2.0.1.0 - Production

Copyright (c) 2000, 2005, Oracle. All rights reserved.

Welcome to DGMGRL, type “help” for information.
DGMGRL> connect sys/password
Connected.
DGMGRL> show configuration;

Configuration
Name: db_DOCUMENTUM
Enabled: YES
Protection Mode: MaxPerformance
Fast-Start Failover: DISABLED
Databases:
dbase – Primary database
dbase_DR – Physical standby database

Current status for “db_DOCUMENTUM”:
Warning: ORA-16607: one or more databases have failed

DGMGRL> show database ‘dbase’;

Database
Name: dbase
Role: PRIMARY
Enabled: YES
Intended State: ONLINE
Instance(s):
dbase

Current status for “dbase”:
Error: ORA-16778: redo transport error for one or more databases

DGMGRL> show database ‘dbase_DR’;

Database
Name: dbase_DR
Role: PHYSICAL STANDBY
Enabled: YES
Intended State: ONLINE
Instance(s):
dbase

Current status for “dbase_DR”:
Error: ORA-01034: ORACLE not available

So the standby database had stopped running (although its windows service was still up and running ok). To fix, first startup nomount the standby database:

sqlplus sys/password as sysdba
SQL>


startup nomount

At this point dgmgrl on live server reported:

DGMGRL> show database 'dbase_DR';

Database
Name: dbase_DR
Role: PHYSICAL STANDBY
Enabled: YES
Intended State: ONLINE
Instance(s):
dbase

Current status for “dbase_DR”:
Error: ORA-16525: the Data Guard broker is not yet available

And secondly, again on the standby database:

SQL>


alter database mount standby database ;

Dgmgrl on live server now reported everything ok:

DGMGRL> show database 'dbase_DR';

Database
Name: dbase_DR
Role: PHYSICAL STANDBY
Enabled: YES
Intended State: ONLINE
Instance(s):
dbase

Current status for “dbase_DR”:
SUCCESS

DGMGRL> show configuration;

Configuration
Name: db_DOCUMENTUM
Enabled: YES
Protection Mode: MaxPerformance
Fast-Start Failover: DISABLED
Databases:
dbase – Primary database
dbase_DR – Physical standby database

Current status for “db_DOCUMENTUM”:
SUCCESS

DGMGRL>

Which alert logs for live and standby confirmed.

The standby alert log shows this command also:
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH ALL SWITCHOVER DISCONNECT NODELAY
But that was issued behind the scenes by the Data Guard Broker, not issued manually by the DBA.

As an aside, 10g logical standby databases (that replay SQL rather than redo change vectors) are availalable with the same protection modes and fast start failover options as physical standby databases (the traditional type that replay redo). That is according to documentation, I haven’t had a chance to use those myself yet, but hope to be soon. The big advantage of logical standby databases is that they are available for reporting use without halting the replication. In the absence of other drawbacks (not unknown in new oracle features), that would make logical standby the preferred option.

EDIT – for older databases or otherwise where data guard is not running, commands to use would instead be like:

recover managed standby database cancel;
recover standby database;
[auto]
recover managed standby database disconnect;

Database clones can also be done with rman, which has the advantage of avoiding the performance draining hot backup mode.

set pages 9999 lines 132 serverout on size 99999
BEGIN
  FOR t IN ( select distinct tablespace_name from dba_data_files )
  LOOP
    dbms_output.put_line('alter tablespace '||t.tablespace_name||' begin backup ;') ;
    FOR f IN ( select file_name from dba_data_files where tablespace_name = t.tablespace_name )
    LOOP
      dbms_output.put_line('host cp '||f.file_name||' '||f.file_name ) ;
    END LOOP ;
    dbms_output.put_line('alter tablespace '||t.tablespace_name||' end backup ;') ;
  END LOOP ;
END ;
/
set lines 80
select 'alter tablespace '||tablespace_name||' add tempfile '||file_name||' size '||bytes/1024/1024||' m ;'
from dba_temp_files
/
prompt alter system switch logfile ;;
prompt alter system backup controlfile to trace ;;

RMAN-03014: implicit resync of recovery catalog failed
RMAN-06004: ORACLE error from recovery catalog database: RMAN-20035: invalid high recid

You can switch the catalog over to a previous incarnation (before an open resetlogs):

RMAN> list incarnation of database <database name>;
RMAN> reset database to incarnation <incarnation number>;

But that is only valid where you want to go back in time to have the catalog work with an old incarnation. If you have mutiple copies of the same database, and want them all registered in the same rman catalog, you have to give the copies different DBIDs, usng the Nid utility:

SQL> shutdown immediate
SQL> startup mount
$ nid target=sys/password@testdb

(you don’t need to use a connect string here, but if you do, a remote login passwordfile is required)

SQL> shutdown immediate

(at least for 9i, is handled automatically with 10g according to documentation)

SQL> startup mount
SQL> alter database open resetlogs;
$ rman target / catalog=rman/password@catdb
RMAN> register database;

Nid comes with release 9iR2 and above, but can be run against older databases (at least with 8174 64 bit on Solaris), provided you use the connect string syntax with a remote login password file. It is

$ORACLE_HOME/bin/nid

. You have to set your full environment to the new (9iR2 or later) ORACLE_HOME before running nid that way – can use . oraenv naming a dummy (non existant) database and then typing in the new ORACLE_HOME when prompted.

Reference: Metalink Note:224266.1 or Utilities doc

RMAN-06059: expected archived log not found, lost of archived log compromises recoverability

If you can, you should bring back the missing archved redo logs to their original location and name, and let rman back them up. But if that is impossible, the workaround is to “crosscheck archivelog all”, like:

rman <<e1
connect target /
connect catalog username/password@catalog
run {
allocate channel c1 type disk ;
crosscheck archivelog all ;
release channel c1 ;
}
e1

You’ll get output like this:

validation succeeded for archived log
archive log filename=D:REDOARCHARCH_1038.DBF recid=1017 stamp=611103638

for every archived log as they are all checked on disk. That should be the catalog fixed, run an archivelog backup to make sure.

[Ref: Metalink ]