Errors in file /ora10g/admin/dqb/bdump/dqb1_smon_185150.trc: ORA-00376: Message 376 not found; No message file for product=RDBMS, facility=ORA; arguments: [113] ORA-01110: Message 1110 not found; No message file for product=RDBMS, facility=
Errors in file /ora10g/admin/dqb/bdump/dqb1_smon_185150.trc:ORACLE Instance dqb1 (pid = 17) - Error 376 encountered while recovering transaction (87, 27) on object 3036334.
上述错误信息一直在后台报错，查询部分表报错，查询数据库的数据文件状态很多是offline状态。
SQL> select name ,status from v$datafile where status'ONLINE';
NAME
--------------------------------------------------------------------------------
STATUS
-------
/dev/rrac_system_1g
SYSTEM
/dev/rrlv_20g_data01
OFFLINE
/dev/rrlv_20g_data03
OFFLINE
NAME
--------------------------------------------------------------------------------
STATUS
-------
/dev/rrlv_20g_data04
OFFLINE
/dev/rrlv_20g_data05
OFFLINE
/dev/rrlv_20g_data06
OFFLINE
NAME
--------------------------------------------------------------------------------
STATUS
-------
/dev/rrlv_20g_data07
OFFLINE
/dev/rrlv_20g_data09
OFFLINE
/dev/rrlv_20g_data10
OFFLINE
NAME
--------------------------------------------------------------------------------
STATUS
-------
/dev/rrlv_20g_data11
OFFLINE
/dev/rrlv_20g_data12
OFFLINE
/dev/rrlv_20g_data13
OFFLINE
NAME
--------------------------------------------------------------------------------
STATUS
-------
/dev/rrlv_20g_data14
OFFLINE
/dev/rrac_20g20
SYSTEM
/dev/rrac_20g21
SYSTEM
NAME
--------------------------------------------------------------------------------
STATUS
-------
/dev/rrlv_20g_data15
OFFLINE
/dev/rrlv_20g_data17
OFFLINE
/dev/rrlv_20g_data16
OFFLINE
NAME
--------------------------------------------------------------------------------
STATUS
-------
/dev/rrlv_20g_data18
OFFLINE
19 rows selected.
仔细查看数据库日志发现，在1月12日做了表空间增加数据文件操作。
查看两个节点的数据文件权限
两个节点的裸设备都是这个样子
brw-rw---- 1 root system 83, 1 Feb 20 2014 /dev/rlv_20g_data01
brw-rw---- 1 root system 83, 2 Feb 20 2014 /dev/rlv_20g_data02
brw-rw---- 1 root system 83, 3 Feb 20 2014 /dev/rlv_20g_data03
brw-rw---- 1 root system 83, 4 Feb 20 2014 /dev/rlv_20g_data04
brw-rw---- 1 root system 83, 5 Feb 20 2014 /dev/rlv_20g_data05
brw-rw---- 1 root system 83, 6 Feb 20 2014 /dev/rlv_20g_data06
brw-rw---- 1 root system 83, 7 Feb 20 2014 /dev/rlv_20g_data07
brw-rw---- 1 root system 83, 8 Feb 20 2014 /dev/rlv_20g_data08
brw-rw---- 1 root system 83, 9 Feb 20 2014 /dev/rlv_20g_data09
brw-rw---- 1 root system 83, 10 Feb 20 2014 /dev/rlv_20g_data10
brw-rw---- 1 root system 83, 11 Feb 20 2014 /dev/rlv_20g_data11
brw-rw---- 1 root system 83, 12 Feb 20 2014 /dev/rlv_20g_data12
brw-rw---- 1 root system 83, 13 Feb 20 2014 /dev/rlv_20g_data13
brw-rw---- 1 root system 83, 14 Feb 20 2014 /dev/rlv_20g_data14
brw-rw---- 1 root system 83, 15 Feb 20 2014 /dev/rlv_20g_data15
brw-rw---- 1 root system 83, 16 Feb 20 2014 /dev/rlv_20g_data16
brw-rw---- 1 root system 83, 17 Feb 20 2014 /dev/rlv_20g_data17
brw-rw---- 1 root system 83, 18 Feb 20 2014 /dev/rlv_20g_data18
brw-rw---- 1 root system 83, 19 Feb 20 2014 /dev/rlv_20g_data19
brw-rw---- 1 root system 83, 20 Feb 20 2014 /dev/rlv_20g_data20
正确是以下：
crw-rw---- 1 ora10g dba 83, 1 Jan 20 08:19 /dev/rrlv_20g_data01
crw-rw---- 1 ora10g dba 83, 2 Jan 20 20:28 /dev/rrlv_20g_data02
crw-rw---- 1 ora10g dba 83, 3 Jan 20 08:19 /dev/rrlv_20g_data03
crw-rw---- 1 ora10g dba 83, 4 Jan 20 08:19 /dev/rrlv_20g_data04
crw-rw---- 1 ora10g dba 83, 5 Jan 20 08:19 /dev/rrlv_20g_data05
crw-rw---- 1 ora10g dba 83, 6 Jan 20 08:19 /dev/rrlv_20g_data06
crw-rw---- 1 ora10g dba 83, 7 Jan 20 08:19 /dev/rrlv_20g_data07
crw-rw---- 1 ora10g dba 83, 8 Jan 20 20:28 /dev/rrlv_20g_data08
crw-rw---- 1 ora10g dba 83, 9 Jan 20 08:19 /dev/rrlv_20g_data09
crw-rw---- 1 ora10g dba 83, 10 Jan 20 08:19 /dev/rrlv_20g_data10
crw-rw---- 1 ora10g dba 83, 11 Jan 20 08:19 /dev/rrlv_20g_data11
crw-rw---- 1 ora10g dba 83, 12 Jan 20 08:19 /dev/rrlv_20g_data12
crw-rw---- 1 ora10g dba 83, 13 Jan 20 08:19 /dev/rrlv_20g_data13
crw-rw---- 1 ora10g dba 83, 14 Jan 20 08:19 /dev/rrlv_20g_data14
crw-rw---- 1 ora10g dba 83, 15 Jan 20 08:19 /dev/rrlv_20g_data15
crw-rw---- 1 ora10g dba 83, 16 Jan 20 08:19 /dev/rrlv_20g_data16
crw-rw---- 1 ora10g dba 83, 17 Jan 20 08:19 /dev/rrlv_20g_data17
crw-rw---- 1 ora10g dba 83, 18 Jan 20 08:19 /dev/rrlv_20g_data18
crw-rw---- 1 ora10g dba 83, 19 Jan 20 20:28 /dev/rrlv_20g_data19
crw-rw---- 1 ora10g dba 83, 20 Jan 20 20:28 /dev/rrlv_20g_data20
这样子问题就清楚了，执行权限修改，后执行数据文件恢复
1、chown -R oracle;dba /dev/rrlv_20g_data*
2、停掉两个节点的数据库，启动第一个节点到mount，执行recover操作
[BEGIN] 2015-1-21 10:43:09
ORA-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92773_727466139.dbf
ORA-00280: change 14242883477423 for thread 2 is in sequence #92773
ORA-00278: log file '+ARCH/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
Log applied.
Media recovery complete.
SQL> RECOVER datafile 14;
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92772_727466139.dbf
ORA-00280: change 14242805827356 for thread 2 is in sequence #92772
Specify log: {
auto
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ORA-00289: suggestion : +ARCH/archive/arch1/arch1_30832_727466139.dbf
ORA-00280: change 14242805827356 for thread 1 is in sequence #30832
ORA-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92773_727466139.dbf
ORA-00280: change 14242883477423 for thread 2 is in sequence #92773
ORA-00278: log file '+ARCH/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
Log applied.
Media recovery complete.
SQL> RECOVER datafile 17;
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92772_727466139.dbf
ORA-00280: change 14242805827356 for thread 2 is in sequence #92772
Specify log: {
auto
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ORA-00289: suggestion : +ARCH/archive/arch1/arch1_30832_727466139.dbf
ORA-00280: change 14242805827356 for thread 1 is in sequence #30832
ORA-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92773_727466139.dbf
ORA-00280: change 14242883477423 for thread 2 is in sequence #92773
ORA-00278: log file '+ARCH/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
Log applied.
Media recovery complete.
SQL> RECOVER datafile 18;
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92772_727466139.dbf
ORA-00280: change 14242805827356 for thread 2 is in sequence #92772
Specify log: {
auto
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ORA-00289: suggestion : +ARCH/archive/arch1/arch1_30832_727466139.dbf
ORA-00280: change 14242805827356 for thread 1 is in sequence #30832
ORA-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92773_727466139.dbf
ORA-00280: change 14242883477423 for thread 2 is in sequence #92773
ORA-00278: log file '+ARCH/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
Log applied.
Media recovery complete.
SQL> RECOVER datafile 19;
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92772_727466139.dbf
ORA-00280: change 14242805827356 for thread 2 is in sequence #92772
Specify log: {
auto
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ORA-00289: suggestion : +ARCH/archive/arch1/arch1_30832_727466139.dbf
ORA-00280: change 14242805827356 for thread 1 is in sequence #30832
ORA-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92773_727466139.dbf
ORA-00280: change 14242883477423 for thread 2 is in sequence #92773
ORA-00278: log file '+ARCH/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
Log applied.
Media recovery complete.
SQL> RECOVER datafile 21;
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92772_727466139.dbf
ORA-00280: change 14242805827356 for thread 2 is in sequence #92772
Specify log: {
auto
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ORA-00289: suggestion : +ARCH/archive/arch1/arch1_30832_727466139.dbf
ORA-00280: change 14242805827356 for thread 1 is in sequence #30832
ORA-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92773_727466139.dbf
ORA-00280: change 14242883477423 for thread 2 is in sequence #92773
ORA-00278: log file '+ARCH/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
Log applied.
Media recovery complete.
SQL> RECOVER datafile 22;
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92772_727466139.dbf
ORA-00280: change 14242805827356 for thread 2 is in sequence #92772
Specify log: {
auto
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ORA-00289: suggestion : +ARCH/archive/arch1/arch1_30832_727466139.dbf
ORA-00280: change 14242805827356 for thread 1 is in sequence #30832
ORA-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92773_727466139.dbf
ORA-00280: change 14242883477423 for thread 2 is in sequence #92773
ORA-00278: log file '+ARCH/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
Log applied.
Media recovery complete.
SQL> RECOVER datafile 23;
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92772_727466139.dbf
ORA-00280: change 14242805827356 for thread 2 is in sequence #92772
Specify log: {
auto
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ORA-00289: suggestion : +ARCH/archive/arch1/arch1_30832_727466139.dbf
ORA-00280: change 14242805827356 for thread 1 is in sequence #30832
ORA-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92773_727466139.dbf
ORA-00280: change 14242883477423 for thread 2 is in sequence #92773
ORA-00278: log file '+ARCH/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
Log applied.
Media recovery complete.
SQL> RECOVER datafile 26;
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92772_727466139.dbf
ORA-00280: change 14242805827356 for thread 2 is in sequence #92772
Specify log: {
auto
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ORA-00289: suggestion : +ARCH/archive/arch1/arch1_30832_727466139.dbf
ORA-00280: change 14242805827356 for thread 1 is in sequence #30832
ORA-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92773_727466139.dbf
ORA-00280: change 14242883477423 for thread 2 is in sequence #92773
ORA-00278: log file '+ARCH/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
Log applied.
Media recovery complete.
SQL> RECOVER datafile 29;
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92772_727466139.dbf
ORA-00280: change 14242805827356 for thread 2 is in sequence #92772
Specify log: {
auto
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ORA-00289: suggestion : +ARCH/archive/arch1/arch1_30832_727466139.dbf
ORA-00280: change 14242805827356 for thread 1 is in sequence #30832
ORA-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92773_727466139.dbf
ORA-00280: change 14242883477423 for thread 2 is in sequence #92773
ORA-00278: log file '+ARCH/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
Log applied.
Media recovery complete.
SQL> RECOVER datafile 55;
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92772_727466139.dbf
ORA-00280: change 14242805827356 for thread 2 is in sequence #92772
Specify log: {
auto
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ORA-00289: suggestion : +ARCH/archive/arch1/arch1_30832_727466139.dbf
ORA-00280: change 14242805827356 for thread 1 is in sequence #30832
ORA-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92773_727466139.dbf
ORA-00280: change 14242883477423 for thread 2 is in sequence #92773
ORA-00278: log file '+ARCH/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
Log applied.
Media recovery complete.
SQL> RECOVER datafile 103;
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92772_727466139.dbf
ORA-00280: change 14242805827356 for thread 2 is in sequence #92772
Specify log: {
auto
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ORA-00289: suggestion : +ARCH/archive/arch1/arch1_30832_727466139.dbf
ORA-00280: change 14242805827356 for thread 1 is in sequence #30832
ORA-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92773_727466139.dbf
ORA-00280: change 14242883477423 for thread 2 is in sequence #92773
ORA-00278: log file '+ARCH/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
Log applied.
Media recovery complete.
SQL> RECOVER datafile 104;
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92772_727466139.dbf
ORA-00280: change 14242805827356 for thread 2 is in sequence #92772
Specify log: {
auto
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ORA-00289: suggestion : +ARCH/archive/arch1/arch1_30832_727466139.dbf
ORA-00280: change 14242805827356 for thread 1 is in sequence #30832
ORA-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92773_727466139.dbf
ORA-00280: change 14242883477423 for thread 2 is in sequence #92773
ORA-00278: log file '+ARCH/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
Log applied.
Media recovery complete.
SQL> RECOVER datafile 112;
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:33 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92772_727466139.dbf
ORA-00280: change 14242805827356 for thread 2 is in sequence #92772
Specify log: {
auto
ORA-00279: change 14242805827356 generated at 01/20/2015 08:19:29 needed for
thread 1
ORA-00289: suggestion : +ARCH/archive/arch1/arch1_30832_727466139.dbf
ORA-00280: change 14242805827356 for thread 1 is in sequence #30832
ORA-00279: change 14242883477423 generated at 01/20/2015 08:23:26 needed for
thread 2
ORA-00289: suggestion : +ARCH/archive/arch1/arch2_92773_727466139.dbf
ORA-00280: change 14242883477423 for thread 2 is in sequence #92773
ORA-00278: log file '+ARCH/archive/arch1/arch2_92772_727466139.dbf' no longer
needed for this recovery
Log applied.
Media recovery complete.
SQL> alter database open;
最后成功执行。
