Troubleshooting ORA-00600: [kcladdle_1], [], [] in RAC 11.2.0.3

内容简介：Troubleshooting ORA-00600: [kcladdle_1], [], [] in RAC 11.2.0.3

突然数据库出现了较高的 enq : TX – row lock contention 的等待事件，这是一个APPLICATION CLASS的wait event, 通常出现在DML事务级的行或块竞争，这次的MODE为6，语句只是简单的delete xx where rowid=:bind; 理论上这个事务应该是非常快，但是查看数据库和应用日志都出现了ORA-600的错误，判断问题就不再是简单的行事务竞争，手动模拟语句也还原出问题，但并不是所有记录每次都能触发，现象为运行DELETE SQL时失败并出现ora-600 [kcladdle_1]. 数据库中DB Alert log 如下：

2017-06-03 19:43:50.563000 +08:00
Thread 1 advanced to log sequence 26913 (LGWR switch)
  Current log# 1 seq# 26913 mem# 0: +DATADG/anbob/onlinelog/group_1.257.923576165
2017-06-03 19:49:03.062000 +08:00
Thread 1 advanced to log sequence 26914 (LGWR switch)
  Current log# 2 seq# 26914 mem# 0: +DATADG/anbob/onlinelog/group_2.258.923576167
2017-06-03 19:53:45.570000 +08:00
ERROR: Unable to normalize symbol name for the following short stack (at offset 316):
dbgexProcessError()+180<-dbgeExecuteForError()+72<-dbgePostErrorKGE()+2048<-dbkePostKGE_kgsf()+68<-kgeadse()+380<-kgerinv_internal()+48<-kgerinv()+48<-kserin()+76
<-kclladdle()+128<-kclcls()+18764<-kclgrlk()+328<-kcbzib()+18480<-kcbgcur()+5440<-ktspgfblk3()+376<-ktspstchg1()+128<-ktspstchg()+20<-kddchg()+1212<-IPRA.$kdddrp()+1980<-IPRA.$kdddel()+472<-kdddel()+96<-kaudel()+120<-delrow()+1348<-qerdlFetch()+688<-delexe()+936<-opiexe()+14672<-opiodr()+720<-ttcpip()+1
028<-opitsk()+1508<-opiino()+940<-opiodr()+720<-opidrv()+1132<-sou2o()+136<-opimai_real()+608<-ssthrdmain()+268<-main()+204<-__start()+112
Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_60686692.trc  (incident=374057):
ORA-00600: internal error code, arguments: [kcladdle_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_374057/anbob1_ora_60686692_i374057.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Non critical error ORA-48913 caught while writing to trace file "/oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_374057/anbob1_ora_60686692_i374057.trc"
Error message: ORA-48913: Writing into trace file failed, file size limit [1024000] reached
Writing to the above trace file is disabled for now on...
2017-06-03 19:54:01.034000 +08:00
Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_66978102.trc  (incident=374073):
ORA-00600: internal error code, arguments: [kcladdle_1], [], [], [], [], [], [], [], [], [], [], []
Dumping diagnostic data in directory=[cdmp_20170603195401], requested by (instance=1, osid=60686692), summary=[incident=374057].
Incident details in: /oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_374073/anbob1_ora_66978102_i374073.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2017-06-03 19:54:02.219000 +08:00
Sweep [inc][374057]: completed
Non critical error ORA-48913 caught while writing to trace file "/oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_374073/anbob1_ora_66978102_i374073.trc"
Error message: ORA-48913: Writing into trace file failed, file size limit [1024000] reached
Writing to the above trace file is disabled for now on...
Sweep [inc2][374057]: completed
Sweep [inc][374073]: completed
Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_61473276.trc  (incident=374033):
ORA-00600: internal error code, arguments: [kcladdle_1], [], [], [], [], [], [], [], [], [], [], []
Dumping diagnostic data in directory=[cdmp_20170603195403], requested by (instance=1, osid=66978102), summary=[incident=374073].
Incident details in: /oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_374033/anbob1_ora_61473276_i374033.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2017-06-03 19:54:03.461000 +08:00
Sweep [inc2][374073]: completed
Sweep [inc][374033]: completed
Non critical error ORA-48913 caught while writing to trace file "/oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_374033/anbob1_ora_61473276_i374033.trc"
Error message: ORA-48913: Writing into trace file failed, file size limit [1024000] reached
Writing to the above trace file is disabled for now on...
2017-06-03 19:54:04.567000 +08:00
Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_5440550.trc  (incident=374081):
ORA-00600: internal error code, arguments: [kcladdle_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_374081/anbob1_ora_5440550_i374081.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Dumping diagnostic data in directory=[cdmp_20170603195404], requested by (instance=1, osid=61473276), summary=[incident=374033].
Sweep [inc2][374033]: completed
Sweep [inc][374081]: completed
Non critical error ORA-48913 caught while writing to trace file "/oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_374081/anbob1_ora_5440550_i374081.trc"
Error message: ORA-48913: Writing into trace file failed, file size limit [1024000] reached
Writing to the above trace file is disabled for now on...
2017-06-03 19:54:06.212000 +08:00
Dumping diagnostic data in directory=[cdmp_20170603195406], requested by (instance=1, osid=5440550), summary=[incident=374081].
Errors in file /oracle/app/oracle/diag/rdbms/anbob/anbob1/trace/anbob1_ora_1902024.trc  (incident=374065):
ORA-00600: internal error code, arguments: [kcladdle_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_374065/anbob1_ora_1902024_i374065.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Non critical error ORA-48913 caught while writing to trace file "/oracle/app/oracle/diag/rdbms/anbob/anbob1/incident/incdir_374065/anbob1_ora_1902024_i374065.trc"
...
...

错误时的CALL Stack，确认BUG常用的

----- Call Stack Trace -----
skdstdst <- ksedst1 <- ksedst <- dbkedDefDump <- ksedmp
<- ksfdmp <- dbgexPhaseII <- dbgexProcessError <- dbgeExecuteForError <- dbgePostErrorKGE
<- 2048 <- dbkePostKGE_kgsf <- kgeade <- kgefec <- kgefic
<- ksefic <- ksupop <- opiodr <- ttcpip <- opitsk
<- opiino <- opiodr <- opidrv <- sou2o <- opimai_real
<- ssthrdmain <- main <- start

TIP:

日志错误一致在持续，根据错误函数kcladdle_1判断， KCL=>Kernel Cache, (RAC) Lock,判断是cache级错误，MOS中也没有关于该错误的更多的信息只有一条BUG描述。判断CACHE级错误，尝试把在1节点delete报错的 SQL 去2节点执行无任何错误，后建议应用调整之2节点，同时刷新2个节点的Buffer cache, 问题现象得到了解决。后期经SR确认为

Bug 12644270，当转换buffer cache k中的 block PastImages 锁时发生。

ORA-00600: [kcladdle_1]

Rediscovery Notes:

The problem can occur while downconverting a lock to NULL because there is just a PI attached – the PI may be written to disk and be removed from the lock.

Versions confirmed as being affected

11.2.0.3

Platforms affected Generic (all / most platforms affected)

Fixed:

The fix for 12644270 is first included in

12.1.0.1 (Base Release)

11.2.0.4 (Server Patch Set)

以上就是本文的全部内容，希望本文的内容对大家的学习或者工作能带来一定的帮助，也希望大家多多支持码农网