OEL6.3上 Oracle RAC 上节点驱逐检查过程-mysql教程-PHP中文網

首頁

資料庫

mysql教程

OEL6.3上 Oracle RAC 上节点驱逐检查过程

WBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWB

Jun 07, 2016 pm 04:47 PM

oracle rac

RAC上是vmware vspere的虚拟机上有两个OEL6.3的虚拟机，上面跑的库是11.2.0.4

RAC上是vmware vspere的虚拟机上有两个OEL6.3的虚拟机，，上面跑的库是11.2.0.4

发现节点2挂掉了

1.检查节点1的alert
[root@racnode1 racnode1]# pwd
/u01/apps/grid/gridhome/11.2.0/grid/log/racnode1
[root@racnode1 racnode1]# tail -1000 alertracnode1.log
2014-04-15 09:41:20.815:
[crsd(27311)]CRS-2765:Resource 'ora.net1.network' has failed on server 'racnode1'.
2014-04-15 09:41:42.760:
[cssd(26972)]CRS-1612:Network communication with node racnode2 (2) missing for 50% of timeout interval. Removal of this node from cluster in 15.000 seconds
2014-04-15 09:41:50.763:
[cssd(26972)]CRS-1611:Network communication with node racnode2 (2) missing for 75% of timeout interval. Removal of this node from cluster in 7.000 seconds
2014-04-15 09:41:54.764:
[cssd(26972)]CRS-1610:Network communication with node racnode2 (2) missing for 90% of timeout interval. Removal of this node from cluster in 3.000 seconds
2014-04-15 09:41:57.766:
[cssd(26972)]CRS-1607:Node racnode2 is being evicted in cluster incarnation 291818318; details at (:CSSNM00007:) in /u01/apps/grid/gridhome/11.2.0/grid/log/racnode1/cssd/ocssd.log.
2014-04-15 09:42:06.052:
[cssd(26972)]CRS-1625:Node racnode2, number 2, was manually shut down
2014-04-15 09:42:06.059:
[cssd(26972)]CRS-1601:CSSD Reconfiguration complete. Active nodes are racnode1 .
2014-04-15 09:42:06.950:
[crsd(27311)]CRS-5504:Node down event reported for node 'racnode2'.
2014-04-15 09:42:24.882:
[crsd(27311)]CRS-2773:Server 'racnode2' has been removed from pool 'Generic'.
2014-04-15 09:42:24.882:
[crsd(27311)]CRS-2773:Server 'racnode2' has been removed from pool 'ora.pera'.
[root@racnode1 racnode1]#

节点2被驱逐时间：2014-04-15 09:41:57

2.查看ocssd.log
# more /u01/apps/grid/gridhome/11.2.0/grid/log/racnode1/cssd/ocssd.log |grep "2014-04-15 09:41"

2014-04-15 09:41:23.620: [ CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
2014-04-15 09:41:23.621: [ CSSD][906479360]clssnmSendingThread: sent 4 status msgs to all nodes
2014-04-15 09:41:28.758: [ CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
2014-04-15 09:41:28.758: [ CSSD][906479360]clssnmSendingThread: sent 5 status msgs to all nodes
2014-04-15 09:41:33.145: [GIPCHGEN][919283456] gipchaInterfaceFail: marking interface failing 0x7fc01821d4c0 { host '', haName 'CSS_racnode-cluster', local (nil), ip '172.168.1.11:52955', subnet '172.168.1.0', mask '255.255.255.0', mac '00-50-56-a1-7b-e2', ifname 'eth1', numRef 1, numFail 0, idxBoot 0, flags 0x184d }
2014-04-15 09:41:33.760: [ CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
2014-04-15 09:41:33.760: [GIPCHGEN][920860416] gipchaInterfaceFail: marking interface failing 0x7fc024040a20 { host 'racnode2', haName 'CSS_racnode-cluster', local 0x7fc01821d4c0, ip '172.168.1.12:60678', subnet '172.168.1.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 4, flags 0x6 }
2014-04-15 09:41:33.760: [ CSSD][906479360]clssnmSendingThread: sent 5 status msgs to all nodes
2014-04-15 09:41:33.760: [GIPCHGEN][920860416] gipchaInterfaceDisable: disabling interface 0x7fc01821d4c0 { host '', haName 'CSS_racnode-cluster', local (nil), ip '172.168.1.11:52955', subnet '172.168.1.0', mask '255.255.255.0', mac '00-50-56-a1-7b-e2', ifname 'eth1', numRef 0, numFail 1, idxBoot 0, flags 0x19cd }
2014-04-15 09:41:33.760: [GIPCHGEN][920860416] gipchaInterfaceDisable: disabling interface 0x7fc024040a20 { host 'racnode2', haName 'CSS_racnode-cluster', local 0x7fc01821d4c0, ip '172.168.1.12:60678', subnet '172.168.1.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 4, flags 0x86 }
2014-04-15 09:41:33.761: [GIPCHALO][920860416] gipchaLowerCleanInterfaces: performing cleanup of disabled interface 0x7fc024040a20 { host 'racnode2', haName 'CSS_racnode-cluster', local 0x7fc01821d4c0, ip '172.168.1.12:60678', subnet '172.168.1.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 4, flags 0xa6 }
2014-04-15 09:41:33.885: [GIPCHDEM][920860416] gipchaWorkerCleanInterface: performing cleanup of disabled interface 0x7fc01821d4c0 { host '', haName 'CSS_racnode-cluster', local (nil), ip '172.168.1.11:52955', subnet '172.168.1.0', mask '255.255.255.0', mac '00-50-56-a1-7b-e2', ifname 'eth1', numRef 0, numFail 0, idxBoot 0, flags 0x19ed }
2014-04-15 09:41:34.145: [GIPCHDEM][919283456] gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x1e22410 [0000000000000010] { gipchaContext : host 'racnode1', name 'CSS_racnode-cluster', luid '8bbba732-00000000', numNode 1, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
2014-04-15 09:41:37.993: [GIPCHALO][920860416] gipchaLowerProcessNode: no valid interfaces found to node for 5240 ms, node 0x7fc018226540 { host 'racnode2', haName 'CSS_racnode-cluster', srcLuid 8bbba732-d4368a71, dstLuid 0fc9914c-ec29f220 numInf 0, contigSeq 1787047, lastAck 1786995, lastValidAck 1787046, sendSeq [1786996 : 1787014], createTime 1683284, sentRegister 1, localMonitor 1, flags 0x2408 }
2014-04-15 09:41:38.762: [ CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
2014-04-15 09:41:38.762: [ CSSD][906479360]clssnmSendingThread: sent 5 status msgs to all nodes
2014-04-15 09:41:40.146: [GIPCHDEM][919283456] gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x1e22410 [0000000000000010] { gipchaContext : host 'racnode1', name 'CSS_racnode-cluster', luid '8bbba732-00000000', numNode 1, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
2014-04-15 09:41:42.760: [ CSSD][908056320]clssnmPollingThread: node racnode2 (2) at 50% heartbeat fatal, removal in 15.000 seconds
2014-04-15 09:41:42.760: [ CSSD][908056320]clssnmPollingThread: node racnode2 (2) is impending reconfig, flag 2294796, misstime 15000
2014-04-15 09:41:42.760: [ CSSD][908056320]clssnmPollingThread: local diskTimeout set to 27000 ms, remote disk timeout set to 27000, impending reconfig status(1)
2014-04-15 09:41:42.764: [ CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
2014-04-15 09:41:42.764: [ CSSD][906479360]clssnmSendingThread: sent 4 status msgs to all nodes
2014-04-15 09:41:43.764: [GIPCHALO][920860416] gipchaLowerProcessNode: no valid interfaces found to node for 11010 ms, node 0x7fc018226540 { host 'racnode2', haName 'CSS_racnode-cluster', srcLuid 8bbba732-d4368a71, dstLuid 0fc9914c-ec29f220 numInf 0, contigSeq 1787047, lastAck 1786995, lastValidAck 1787046, sendSeq [1786996 : 1787025], createTime 1683284, sentRegister 1, localMonitor 1, flags 0x2408 }
2014-04-15 09:41:43.822: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526103/1203750934
2014-04-15 09:41:43.906: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1193997, LATS 1203751024, lastSeqNo 1175771, uniqueness 1396323841, timestamp 1397526102/1203310914
2014-04-15 09:41:45.102: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526105/1203752214
2014-04-15 09:41:45.772: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526105/1203752884
2014-04-15 09:41:46.147: [GIPCHDEM][919283456] gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x1e22410 [0000000000000010] { gipchaContext : host 'racnode1', name 'CSS_racnode-cluster', luid '8bbba732-00000000', numNode 1, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
2014-04-15 09:41:46.181: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1193999, LATS 1203753294, lastSeqNo 1193997, uniqueness 1396323841, timestamp 1397526104/1203313164
2014-04-15 09:41:46.782: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526106/1203753894
2014-04-15 09:41:47.120: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194001, LATS 1203754234, lastSeqNo 1193999, uniqueness 1396323841, timestamp 1397526106/1203314834
2014-04-15 09:41:47.622: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526107/1203754734
2014-04-15 09:41:47.765: [ CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
2014-04-15 09:41:47.765: [ CSSD][906479360]clssnmSendingThread: sent 5 status msgs to all nodes
2014-04-15 09:41:48.004: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194002, LATS 1203755124, lastSeqNo 1194001, uniqueness 1396323841, timestamp 1397526107/1203315684
2014-04-15 09:41:48.507: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526108/1203755624
2014-04-15 09:41:48.774: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194003, LATS 1203755894, lastSeqNo 1194002, uniqueness 1396323841, timestamp 1397526108/1203316514
2014-04-15 09:41:49.213: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526109/1203756324
2014-04-15 09:41:49.766: [GIPCHALO][920860416] gipchaLowerProcessNode: no valid interfaces found to node for 17010 ms, node 0x7fc018226540 { host 'racnode2', haName 'CSS_racnode-cluster', srcLuid 8bbba732-d4368a71, dstLuid 0fc9914c-ec29f220 numInf 0, contigSeq 1787047, lastAck 1786995, lastValidAck 1787046, sendSeq [1786996 : 1787037], createTime 1683284, sentRegister 1, localMonitor 1, flags 0x2408 }
2014-04-15 09:41:49.893: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194004, LATS 1203757004, lastSeqNo 1194003, uniqueness 1396323841, timestamp 1397526109/1203317334
2014-04-15 09:41:49.903: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526109/1203757014
2014-04-15 09:41:50.528: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194005, LATS 1203757644, lastSeqNo 1194004, uniqueness 1396323841, timestamp 1397526109/1203317964
2014-04-15 09:41:50.548: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526110/1203757664
2014-04-15 09:41:50.763: [ CSSD][908056320]clssnmPollingThread: node racnode2 (2) at 75% heartbeat fatal, removal in 7.000 seconds
2014-04-15 09:41:50.875: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194006, LATS 1203757994, lastSeqNo 1194005, uniqueness 1396323841, timestamp 1397526110/1203318664
2014-04-15 09:41:51.113: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526111/1203758224
2014-04-15 09:41:51.396: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194007, LATS 1203758514, lastSeqNo 1194006, uniqueness 1396323841, timestamp 1397526110/1203319204
2014-04-15 09:41:51.692: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526111/1203758804
2014-04-15 09:41:52.148: [GIPCHDEM][919283456] gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x1e22410 [0000000000000010] { gipchaContext : host 'racnode1', name 'CSS_racnode-cluster', luid '8bbba732-00000000', numNode 1, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
2014-04-15 09:41:52.294: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194008, LATS 1203759404, lastSeqNo 1194007, uniqueness 1396323841, timestamp 1397526111/1203319754
2014-04-15 09:41:52.305: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526112/1203759424
2014-04-15 09:41:52.767: [ CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
2014-04-15 09:41:52.767: [ CSSD][906479360]clssnmSendingThread: sent 5 status msgs to all nodes
2014-04-15 09:41:52.772: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194009, LATS 1203759884, lastSeqNo 1194008, uniqueness 1396323841, timestamp 1397526112/1203320364
2014-04-15 09:41:52.974: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526112/1203760084
2014-04-15 09:41:53.793: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194010, LATS 1203760904, lastSeqNo 1194009, uniqueness 1396323841, timestamp 1397526112/1203321034
2014-04-15 09:41:53.934: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526113/1203761044
2014-04-15 09:41:54.401: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194012, LATS 1203761514, lastSeqNo 1194010, uniqueness 1396323841, timestamp 1397526114/1203322404
2014-04-15 09:41:54.494: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526114/1203761604
2014-04-15 09:41:54.764: [ CSSD][908056320]clssnmPollingThread: node racnode2 (2) at 90% heartbeat fatal, removal in 3.000 seconds, seedhbimpd 1
2014-04-15 09:41:55.044: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526115/1203762154
2014-04-15 09:41:55.134: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194013, LATS 1203762254, lastSeqNo 1194012, uniqueness 1396323841, timestamp 1397526114/1203322964
2014-04-15 09:41:55.640: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526115/1203762754
2014-04-15 09:41:55.768: [GIPCHALO][920860416] gipchaLowerProcessNode: no valid interfaces found to node for 23010 ms, node 0x7fc018226540 { host 'racnode2', haName 'CSS_racnode-cluster', srcLuid 8bbba732-d4368a71, dstLuid 0fc9914c-ec29f220 numInf 0, contigSeq 1787047, lastAck 1786995, lastValidAck 1787046, sendSeq [1786996 : 1787048], createTime 1683284, sentRegister 1, localMonitor 1, flags 0x2408 }
2014-04-15 09:41:56.498: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526116/1203763614
2014-04-15 09:41:57.145: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194015, LATS 1203764254, lastSeqNo 1194013, uniqueness 1396323841, timestamp 1397526116/1203324554
2014-04-15 09:41:57.645: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526117/1203764764
2014-04-15 09:41:57.764: [ CSSD][908056320]clssnmPollingThread: Removal started for node racnode2 (2), flags 0x23040c, state 3, wt4c 0
2014-04-15 09:41:57.764: [ CSSD][908056320]clssnmMarkNodeForRemoval: node 2, racnode2 marked for removal
2014-04-15 09:41:57.765: [ CSSD][908056320]clssnmDiscHelper: racnode2, node(2) connection failed, endp (0x3938), probe(0x7fc000000000), ninf->endp 0x3938
2014-04-15 09:41:57.765: [ CSSD][908056320]clssnmDiscHelper: node 2 clean up, endp (0x3938), init state 5, cur state 5
2014-04-15 09:41:57.765: [GIPCXCPT][908056320] gipcInternalDissociate: obj 0x7fc024056a70 [0000000000003938] { gipcEndpoint : localAddr 'gipcha://racnode1:nm2_racnode-cluster/da4b-8718-50e0-b51', remoteAddr 'gipcha://racnode2:7b5c-4672-87e8-565', numPend 1, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 1, wobj 0x7fc02402dd90, sendp (nil)flags 0x138606, usrFlags 0x0 } not associated with any container, ret gipcretFail (1)
2014-04-15 09:41:57.765: [GIPCXCPT][908056320] gipcDissociateF [clssnmDiscHelper : clssnm.c : 3485]: EXCEPTION[ ret gipcretFail (1) ] failed to dissociate obj 0x7fc024056a70 [0000000000003938] { gipcEndpoint : localAddr 'gipcha://racnode1:nm2_racnode-cluster/da4b-8718-50e0-b51', remoteAddr 'gipcha://racnode2:7b5c-4672-87e8-565', numPend 1, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 1, wobj 0x7fc02402dd90, sendp (nil)flags 0x138606, usrFlags 0x0 }, flags 0x0
2014-04-15 09:41:57.765: [ CSSD][904902400]clssnmDoSyncUpdate: Initiating sync 291818318
2014-04-15 09:41:57.765: [ CSSD][904902400]clssscCompareSwapEventValue: changed NMReconfigInProgress val 1, from -1, changes 7
2014-04-15 09:41:57.765: [ CSSD][904902400]clssnmDoSyncUpdate: local disk timeout set to 27000 ms, remote disk timeout set to 27000
2014-04-15 09:41:57.765: [ CSSD][904902400]clssnmDoSyncUpdate: new values for local disk timeout and remote disk timeout will take effect when the sync is completed.
2014-04-15 09:41:57.765: [ CSSD][903325440]clssnmDiscEndp: gipcDestroy 0x3938
2014-04-15 09:41:57.765: [ CSSD][904902400]clssnmDoSyncUpdate: Starting cluster reconfig with incarnation 291818318
2014-04-15 09:41:57.765: [ CSSD][904902400]clssnmSetupAckWait: Ack message type (11)
2014-04-15 09:41:57.765: [ CSSD][904902400]clssnmSetupAckWait: node(1) is ALIVE
2014-04-15 09:41:57.765: [ CSSD][904902400]clssnmSendSync: syncSeqNo(291818318), indicating EXADATA fence initialization complete
2014-04-15 09:41:57.765: [ CSSD][904902400]List of nodes that have ACKed my sync: NULL
2014-04-15 09:41:57.765: [ CSSD][904902400]clssnmSendSync: syncSeqNo(291818318)
2014-04-15 09:41:57.765: [ CSSD][903325440]clssnmHandleSync: Node racnode1, number 1, is EXADATA fence capable
2014-04-15 09:41:57.765: [ CSSD][904902400]clssnmWaitForAcks: Ack message type(11), ackCount(1)
2014-04-15 09:41:57.765: [ CSSD][903325440]clssscUpdateEventValue: NMReconfigInProgress val 1, changes 8
2014-04-15 09:41:57.765: [ CSSD][903325440]clssnmHandleSync: local disk timeout set to 27000 ms, remote disk timeout set to 27000
2014-04-15 09:41:57.765: [ CSSD][903325440]clssnmHandleSync: initleader 1 newleader 1
2014-04-15 09:41:57.765: [ CSSD][903325440]clssnmQueueClientEvent: Sending Event(2), type 2, incarn 291818317
2014-04-15 09:41:57.765: [ CSSD][903325440]clssnmQueueClientEvent: Node[1] state = 3, birth = 291818316, unique = 1396323475
2014-04-15 09:41:57.765: [ CSSD][903325440]clssnmQueueClientEvent: Node[2] state = 5, birth = 291818317, unique = 1396323841
2014-04-15 09:41:57.765: [ CSSD][903325440]clssnmHandleSync: Acknowledging sync: src[1] srcName[racnode1] seq[11] sync[291818318]
2014-04-15 09:41:57.765: [ CSSD][903325440]clssnmSendAck: node 1, racnode1, syncSeqNo(291818318) type(11)
2014-04-15 09:41:57.765: [ CSSD][903325440]clssnmHandleAck: Received ack type 11 from node racnode1, number 1, with seq 0 for sync 291818318, waiting for 0 acks
2014-04-15 09:41:57.765: [ CSSD][904902400]clssnmSendSync: syncSeqNo(291818318), indicating EXADATA fence initialization complete
2014-04-15 09:41:57.765: [ CSSD][941274880]clssgmStartNMMon: node 1 active, birth 291818316
2014-04-15 09:41:57.765: [ CSSD][941274880]clssgmStartNMMon: node 2 active, birth 291818317
2014-04-15 09:41:57.765: [ CSSD][941274880]NMEVENT_SUSPEND [00][00][00][06]
2014-04-15 09:41:57.765: [ CSSD][941274880]clssgmCompareSwapEventValue: changed CmInfo State val 5, from 11, changes 21
2014-04-15 09:41:57.765: [ CSSD][941274880]clssgmSuspendAllGrocks: Issue SUSPEND
2014-04-15 09:41:57.765: [ CSSD][904902400]List of nodes that have ACKed my sync: 1
2014-04-15 09:41:57.765: [ CSSD][904902400]clssnmWaitForAcks: done, syncseq(291818318), msg type(11)
2014-04-15 09:41:57.765: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(IGPERAperaXDB) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
2014-04-15 09:41:57.765: [ CSSD][904902400]clssnmSetMinMaxVersion:node1 product/protocol (11.2/1.4)
2014-04-15 09:41:57.766: [ CSSD][904902400]clssnmSetMinMaxVersion: properties common to all nodes: 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17
2014-04-15 09:41:57.766: [ CSSD][904902400]clssnmSetMinMaxVersion: min product/protocol (11.2/1.4)
2014-04-15 09:41:57.766: [ CSSD][904902400]clssnmSetMinMaxVersion: max product/protocol (11.2/1.4)
2014-04-15 09:41:57.766: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(IG+ASMSYS$USERS) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
2014-04-15 09:41:57.766: [ CSSD][904902400]clssnmNeedConfReq: No configuration to change
2014-04-15 09:41:57.766: [ CSSD][904902400]clssnmDoSyncUpdate: Terminating node 2, racnode2, misstime(30000) state(5)
2014-04-15 09:41:57.766: [ CSSD][904902400]clssnmDoSyncUpdate: Wait for 0 vote ack(s)
2014-04-15 09:41:57.766: [ CSSD][904902400]clssnmCheckDskInfo: Checking disk info...
2014-04-15 09:41:57.766: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(IGPERASYS$USERS) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
2014-04-15 09:41:57.766: [ CSSD][904902400]clssnmCheckSplit: Node 2, racnode2, is alive, DHB (1397526116, 1203324554) more than disk timeout of 27000 after the last NHB (1397526085, 1203293884)
2014-04-15 09:41:57.766: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(crs_version) count(3) master(0) event(2), incarn 3, mbrc 3, to member 0, events 0x20, state 0x0
2014-04-15 09:41:57.766: [ CSSD][904902400]clssnmCheckDskInfo: My cohort: 1
2014-04-15 09:41:57.766: [ CSSD][904902400]clssnmRemove: Start
2014-04-15 09:41:57.766: [ CSSD][904902400](:CSSNM00007:)clssnmrRemoveNode: Evicting node 2, racnode2, from the cluster in incarnation 291818318, node birth incarnation 291818317, death incarnation 291818318, stateflags 0x234000 uniqueness value 1396323841
2014-04-15 09:41:57.766: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(crs_version) count(3) master(0) event(2), incarn 3, mbrc 3, to member 1, events 0x0, state 0x0
2014-04-15 09:41:57.767: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(CRF-) count(4) master(0) event(2), incarn 4, mbrc 4, to member 0, events 0x38, state 0x0
2014-04-15 09:41:57.767: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(CRF-) count(4) master(0) event(2), incarn 4, mbrc 4, to member 1, events 0x38, state 0x0
2014-04-15 09:41:57.767: [ default][904902400]kgzf_gen_node_reid2: generated reid cid=a2998261b3ccff8abf36841a04ffe27b,icin=291818316,nmn=2,lnid=291818317,gid=0,gin=0,gmn=0,umemid=0,opid=0,opsn=0,lvl=node hdr=0xfece0100
2014-04-15 09:41:57.767: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(CLSN.ONSPROC.MASTER) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0xa0, state 0x0
2014-04-15 09:41:57.767: [ CSSD][904902400]clssnmrFenceSage: Fenced node racnode2, number 2, with EXADATA, handle 0
2014-04-15 09:41:57.767: [ CSSD][904902400]clssnmSendShutdown: req to node 2, kill time 1203764884
2014-04-15 09:41:57.767: [ CSSD][904902400]clssnmsendmsg: not connected to node 2
2014-04-15 09:41:57.767: [ CSSD][904902400]clssnmSendShutdown: Send to node 2 failed
2014-04-15 09:41:57.767: [ CSSD][904902400]clssnmWaitOnEvictions: Start
2014-04-15 09:41:57.767: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(DB+ASM) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x68, state 0x0
2014-04-15 09:41:57.767: [ CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526116, 1203324554, 1201720), seedhbimpd TRUE
2014-04-15 09:41:57.767: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(DG+ASM) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x0, state 0x0
2014-04-15 09:41:57.767: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(IG+ASMSYS$BACKGROUND) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
2014-04-15 09:41:57.767: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(DBPERA) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x68, state 0x0
2014-04-15 09:41:57.767: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(VT+ASM) count(2) master(1) event(2), incarn 6, mbrc 2, to member 1, events 0x60, state 0x0
2014-04-15 09:41:57.767: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(IGPERASYS$BACKGROUND) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
2014-04-15 09:41:57.767: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(DG_FRA) count(2) master(1) event(2), incarn 5, mbrc 2, to member 1, events 0x4, state 0x0
2014-04-15 09:41:57.767: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(DG+ASM0) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x0, state 0x0
2014-04-15 09:41:57.768: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(GR+GCR1) count(4) master(0) event(2), incarn 26, mbrc 4, to member 0, events 0x280, state 0x0
2014-04-15 09:41:57.768: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(GR+GCR1) count(4) master(0) event(2), incarn 26, mbrc 4, to member 2, events 0x280, state 0x0
2014-04-15 09:41:57.768: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(DG_CRS) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x4, state 0x0
2014-04-15 09:41:57.768: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(DGPERA-) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x0, state 0x0
2014-04-15 09:41:57.768: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(DGPERA0) count(2) master(0) event(2), incarn 2, mbrc 2, to member 0, events 0x0, state 0x0
2014-04-15 09:41:57.768: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(DG_CRS1) count(2) master(1) event(2), incarn 5, mbrc 2, to member 1, events 0x4, state 0x0
2014-04-15 09:41:57.768: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(DG_DATA) count(2) master(1) event(2), incarn 5, mbrc 2, to member 1, events 0x4, state 0x0
2014-04-15 09:41:57.768: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(CLSN.RLB.pera.MASTER) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0xa0, state 0x0
2014-04-15 09:41:57.768: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(CLSFRAME) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x8, state 0x0
2014-04-15 09:41:57.768: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(EVMDMAIN) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x8, state 0x0
2014-04-15 09:41:57.768: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(CRSDMAIN) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x8, state 0x0
2014-04-15 09:41:57.768: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(EVMDMAIN2) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x8, state 0x0
2014-04-15 09:41:57.769: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(IGPERAALL) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
2014-04-15 09:41:57.769: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(CTSSGROUP) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x8, state 0x0
2014-04-15 09:41:57.769: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(CLSN.AQPROC.pera.MASTER) count(2) master(2) event(2), incarn 2, mbrc 2, to member 1, events 0xa0, state 0x0
2014-04-15 09:41:57.769: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(IGPERApera) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
2014-04-15 09:41:57.769: [ CSSD][941274880]clssgmQueueGrockEvent: groupName(ocr_racnode-cluster) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x78, state 0x0
2014-04-15 09:41:57.769: [ CSSD][941274880]clssgmSuspendAllGrocks: done
2014-04-15 09:41:57.769: [ CSSD][941274880]clssgmCompareSwapEventValue: changed CmInfo State val 2, from 5, changes 22
2014-04-15 09:41:57.769: [ CSSD][941274880]clssgmUpdateEventValue: ConnectedNodes val 291818317, changes 7
2014-04-15 09:41:57.769: [ CSSD][941274880]clssgmCleanupNodeContexts(): cleaning up nodes, rcfg(291818317)
2014-04-15 09:41:57.769: [ CSSD][941274880]clssgmCleanupNodeContexts(): successful cleanup of nodes rcfg(291818317)
2014-04-15 09:41:57.769: [GIPCHAUP][920860416] gipchaUpperDisconnect: initiated discconnect umsg 0x7fc024054c60 { msg 0x7fc0240565c8, ret gipcretRequestPending (15), flags 0x2 }, msg 0x7fc0240565c8 { type gipchaMsgTypeDisconnect (5), srcCid 00000000-000038ed, dstCid 00000000-0000059b }, endp 0x7fc0240553a0 [00000000000038ed] { gipchaEndpoint : port 'nm2_racnode-cluster/da4b-8718-50e0-b51b', peer 'racnode2:7b5c-4672-87e8-565a', srcCid 00000000-000038ed, dstCid 00000000-0000059b, numSend 29, maxSend 100, groupListType 2, hagroup 0x2021010, usrFlags 0x4000, flags 0x21c }
2014-04-15 09:41:57.769: [ CSSD][941274880]clssgmStartNMMon: completed node cleanup
2014-04-15 09:41:57.769: [ CSSD][909633280]clssgmUpdateEventValue: HoldRequest val 1, changes 5
2014-04-15 09:41:57.775: [ CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
2014-04-15 09:41:57.775: [ CSSD][906479360]clssnmSendingThread: sent 5 status msgs to all nodes
2014-04-15 09:41:57.867: [ CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526116, 1203324554, 1201720), seedhbimpd TRUE
2014-04-15 09:41:57.934: [ CSSD][914388736]clssnmvDHBValidateNcopy: node 2, racnode2, has a disk HB, but no network HB, DHB has rcfg 291818318, wrtcnt, 1194016, LATS 1203765044, lastSeqNo 1194015, uniqueness 1396323841, timestamp 1397526117/1203325474
2014-04-15 09:41:57.934: [ CSSD][914388736]clssnmvDiskEvict: Kill block write, file /dev/asm-diskb flags 0x00010004, kill block unique 1396323841, stamp 1203764884/1203764884
2014-04-15 09:41:57.934: [ CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526117, 1203325474, 1201721), seedhbimpd TRUE
2014-04-15 09:41:58.035: [ CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526117, 1203325474, 1201721), seedhbimpd TRUE
2014-04-15 09:41:58.135: [ CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526117, 1203325474, 1201721), seedhbimpd TRUE
2014-04-15 09:41:58.149: [GIPCHDEM][919283456] gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x1e22410 [0000000000000010] { gipchaContext : host 'racnode1', name 'CSS_racnode-cluster', luid '8bbba732-00000000', numNode 1, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
2014-04-15 09:41:58.205: [ CSSD][917706496]clssnmvDiskPing: Writing with status 0x3, timestamp 1397526118/1203765314
2014-04-15 09:41:58.235: [ CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526117, 1203325474, 1201721), seedhbimpd TRUE
2014-04-15 09:41:58.335: [ CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526117, 1203325474, 1201721), seedhbimpd TRUE
2014-04-15 09:41:58.435: [ CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526117, 1203325474, 1201721), seedhbimpd TRUE
2014-04-15 09:41:58.535: [ CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526117, 1203325474, 1201721), seedhbimpd TRUE
2014-04-15 09:41:58.551: [ CSSD][916129536]clssnmvDiskKillCheck: not evicted, file /dev/asm-diskb flags 0x00000000, kill block unique 0, my unique 1396323475
2014-04-15 09:41:58.636: [ CSSD][904902400]clssnmWaitOnEvictions: node 2, undead 1, EXADATA fence handle 0 kill reqest id 0, last DHB (1397526117, 1203325474, 1201721), seedhbimpd TRUE