Home » Server Options » Data Guard » Inconsistent Dictionary on Standby after Reinstatement (Oracle 11.2.0.1.0 on Linux)
Inconsistent Dictionary on Standby after Reinstatement [message #531813] Thu, 17 November 2011 15:19 Go to next message
tolq
Messages: 1
Registered: November 2011
Location: Calgary, Canada
Junior Member
Hi,

My problems (as I understand it) is that after successful failover and successful reinstatement of new standby
database, redo logs do not applied to standby database because. Value on RESETLOGS_CHANGE#
is inconsistent on Standby database.

There are details:

1. Configuration
Server smphoracledb01 with Primary database DB_UNIQUE_NAME = NVCSOM1A
Server smphoracledb02 with Standby database DB_UNIQUE_NAME = NVCSOM1B
Server smphoracledb03 with Data Guard Observer running.
All servers run Linux with Oracle 11.2.0.1.0.

2. Server with Primary database failed and database failed over to Standby server.
No problem here. Primary database NVCSOM1B is up and running on server smphoracledb02.

3. After server smphoracledb01 was fixed, I started database NVCSOM1A 'nomount'. I expected
that it will be reinstate automatically by Observer, but it did not happen. I checked that
Observer can connect to both servers (Primary and Standby).

4. I've started DGMGRL on current Primary server smphoracledb02 and check configuration.


$> dgmgrl sys/xxxx
DGMGRL for Linux: Version 11.2.0.1.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected.
DGMGRL> show configuration

Configuration - nvcsom1

Protection Mode: MaxPerformance
Databases:
nvcsom1b - Primary database
Warning: ORA-16829: fast-start failover configuration is lagging

nvcsom1a - (*) Physical standby database (disabled)
ORA-16661: the standby database needs to be reinstated

Fast-Start Failover: ENABLED

Configuration Status:
WARNING

5. I started manual reinstatement and it succeeded. But configuration is still broken.

DGMGRL> reinstate database nvcsom1a;
Reinstating database "nvcsom1a", please wait...
Operation requires shutdown of instance "nvcsom1" on database "nvcsom1a"
Shutting down instance "nvcsom1"...
ORA-01109: database not open

Database dismounted.
ORACLE instance shut down.
Operation requires startup of instance "nvcsom1" on database "nvcsom1a"
Starting instance "nvcsom1"...
ORACLE instance started.
Database mounted.
Continuing to reinstate database "nvcsom1a" ...
Reinstatement of database "nvcsom1a" succeeded

DGMGRL> show configuration

Configuration - nvcsom1

Protection Mode: MaxPerformance
Databases:
nvcsom1b - Primary database
Error: ORA-16810: multiple errors or warnings detected for the database

nvcsom1a - (*) Physical standby database
Warning: ORA-16829: fast-start failover configuration is lagging

Fast-Start Failover: ENABLED

Configuration Status:
ERROR


6. Data Guard log file shows only these errors:

2011-11-14 08:55:10.944 RSM0: HEALTH CHECK WARNING: ORA-16829: fast-start failover configuration is lagging
2011-11-14 08:55:10.944 Operation HEALTH_CHECK canceled during phase 2, error = ORA-16810
2011-11-14 08:56:10.809 RSM0: HEALTH CHECK ERROR: ORA-16783: cannot resolve gap for database nvcsom1a
2011-11-14 08:56:10.809 Found unresolvable gap to database nvcsom1a.


7. From SQL*Plus on SMPHORACLEDB02 database NVCSOM1B (current Primary) I found
the problem. Archive redo logs were shipped to Standby but not applied:


SYS@nvcsom1> select db_unique_name, database_role, open_mode from v$database;

DB_UNIQUE_NAME DATABASE_ROLE OPEN_MODE
------------------------------ ---------------- --------------------
nvcsom1b PRIMARY READ WRITE


SYS@nvcsom1> select max(resetlogs_change#) from v$archived_log;

MAX(RESETLOGS_CHANGE#)
----------------------
9999251


SYS@nvcsom1> alter system switch logfile;

System altered.

SYS@nvcsom1> select sequence#, archived, applied, deleted, standby_dest
from v$archived_log
where resetlogs_change# = 9999251
order by 1,5;


SEQUENCE# ARC APPLIED DEL STA
---------- --- --------- --- ---
1 YES NO YES NO
2 YES NO YES NO
3 YES NO YES NO
3 YES NO NO YES
4 YES NO YES NO
4 YES NO NO YES
5 YES NO YES NO
5 YES NO NO YES
6 YES NO YES NO
6 YES NO NO YES
7 YES NO YES NO
7 YES NO NO YES
8 YES NO YES NO
8 YES NO NO YES



8. Investigation shows that probable cause of the problems is that after reinstatement new
Standby database has inconsistent information about resetlogs change number.
These are results from Standby:

SYS@nvcsom1> select resetlogs_change# from v$database;

RESETLOGS_CHANGE#
-----------------
773741

SYS@nvcsom1> select max(resetlogs_change#) from v$archived_log;

MAX(RESETLOGS_CHANGE#)
----------------------
9999251

SYS@nvcsom1> select RESETLOGS_CHANGE#, RESETLOGS_TIME, PRIOR_RESETLOGS_CHANGE#, PRIOR_RESETLOGS_TIME from v$database;

RESETLOGS_CHANGE# RESETLOGS PRIOR_RESETLOGS_CHANGE# PRIOR_RES
----------------- --------- ----------------------- ---------
773741 30-AUG-11 773166 30-AUG-11

SYS@nvcsom1> SELECT RESETLOGS_CHANGE# FROM V$DATABASE_INCARNATION WHERE STATUS = 'CURRENT';

RESETLOGS_CHANGE#
-----------------
9999251



9. The same queries on Primary database return these:

SYS@nvcsom1> select resetlogs_change# from v$database;

RESETLOGS_CHANGE#
-----------------
9999251


SYS@nvcsom1> select max(resetlogs_change#) from v$archived_log;

MAX(RESETLOGS_CHANGE#)
----------------------
9999251


SYS@nvcsom1> select RESETLOGS_CHANGE#, RESETLOGS_TIME, PRIOR_RESETLOGS_CHANGE#, PRIOR_RESETLOGS_TIME from v$database;

RESETLOGS_CHANGE# RESETLOGS PRIOR_RESETLOGS_CHANGE# PRIOR_RES
----------------- --------- ----------------------- ---------
9999251 13-NOV-11 773741 30-AUG-11



SYS@nvcsom1> SELECT RESETLOGS_CHANGE# FROM V$DATABASE_INCARNATION WHERE STATUS = 'CURRENT';

RESETLOGS_CHANGE#
-----------------
9999251

=========================================

To fix the problem I have recreated standby database.

Now I have two questions:
1. Why did it happen?
2. Is there the better way to fix the standby database?


Thank you,
Toli.
Re: Inconsistent Dictionary on Standby after Reinstatement [message #533038 is a reply to message #531813] Sun, 27 November 2011 14:52 Go to previous message
babuknb
Messages: 1736
Registered: December 2005
Location: NJ
Senior Member

Quote:
1. Why did it happen?
2. Is there the better way to fix the standby database?


Good question, Some time oracle arch process may hung. I give you two option here.

1, You need to create new arch process and then kill existing process

Work Arcund:

1, Find current ARC'n process ID

2, Increase your log_arch_process from 2 to 4

3, KILL existing arch process [ which you noted from step 1 ]

Method 2,

If you have prod window; then recycle your production database

- Babu
Previous Topic: changing password on primary database
Next Topic: Can dataguard rdbms version is not the same with production rdbms?
Goto Forum:
  


Current Time: Thu Mar 28 12:56:49 CDT 2024