Oracle 10g ORA-07445 RAC环境下一个案例

Oracle 10g ORA-07445 RAC环境下一个案例，主要排查以下错误日志：tomcat日志、数据库alter日志、数据表日志错误、数据库转储文件

Oracle 10gORA-07445 1.   数据库使用的软件及其版本

序号

名称

内容

1

操作系统

CentOS release 5.6 (Final) x86-64

2

数据库版本

Oracle Database 10g Release 2

3

数据库安装包：

补丁包：

10201_database_linux_x86_64.cpio.gz

p6810189_10204_Linux-x86-64.zip

4

OEMbug解决补丁

p8350262_10204_Generic.zip

 

2.   数据库结构

3.   现象

现在出现的报表系统goldengate同步进程异常中断，还有修改部分product表内容rollback。数据库后台报严重的错误。

主要排查以下错误日志：tomcat日志、数据库alter日志、数据表日志错误、数据库转储文件、goldengate进程、goldengate同步日志。

3.1  tomcat日志错误

jdbc版本：ojdbc14-10.2.0.4.0.jar

2012-05-14 16:15:36,755 [http-8083-3] ERROR net.sf.hibernate.transaction.JDBCTransaction - Rollback failed

java.sql.SQLException: No more data to read from socket

            at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112) ~[ojdbc14.jar:Oracle JDBC Driver version - "10.2.0.4.0"]

                at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146) ~[ojdbc14.jar:Oracle JDBC Driver version - "10.2.0.4.0"]

 

3.2  数据库alter日志

Mon May 14 14:37:33 2012

Errors in file /oracle/ora10g/admin/jscn/udump/jscn2_ora_28001.trc:

ORA-07445: exception encountered: core dump [_intel_fast_memcpy.A()+10] [SIGSEGV] [Address not mapped to object] [0x2B09D44BE000] [] []

Mon May 14 14:48:14 2012

Process 0x0x2d8367dc0 appears to be hung while dumping

Current time = 485605312, process death time = 485541208 interval = 60000

                从上面的日志可以看出进程hung住，，让我查看/oracle/ora10g/admin/jscn/udump/jscn2_ora_28001.trc转存文件

3.3  查看转存文件

/oracle/ora10g/admin/jscn/udump/jscn2_ora_28001.trc转存文件比较长，大概有5M，我们就看关键部分

Exception signal: 11 (SIGSEGV), code: 1 (Address not mapped to object), addr: 0x2b09d44be000, PC: [0x2b09d26ee608, _intel_fast_memcpy.A()+10]

*** 2012-05-14 14:37:33.145

ksedmp: internal or fatal error

ORA-07445: exception encountered: core dump [_intel_fast_memcpy.A()+10] [SIGSEGV] [Address not mapped to object] [0x2B09D44BE000] [] []

Current SQL statement for this session:

update product set type=:1, ship_type=:2, stock_warning2=:3, stock_warning=:4, store_id=:5, PRODUCT_NAME=:6, UNIT_PRICE=:7, LIST_PRICE=:8, QUANTITY=:9, BRAND=:10, MODEL=:11, Spec=:12, UNIT=:13, MANUFACTORY=:14, PRODUCINGAREA=:15, RESELLER=:16, supplier_id=:17, CREATE_TIME=:18, UPDATE_TIME=:19, DESCRIPTION=:20, BIG_PICTURE=:21, SMALL_PICTURE=:22, CENTER_PICTURE=:23, POP_PICTURE=:24, POP_PICTURE1=:25, POP_PICTURE2=:26, POP_PICTURE3=:27, POP_PICTURE4=:28, POP_PICTURE5=:29, note=:30, TEMPLATE_ID=:31, CHECKSTATUS=:32, check_man=:33, check_time=:34, STATUS=:35, SEARCHKEY=:36, MAIN_CATEGORY=:37, PRODUCT_NO=:38, sid=:39, displayId=:40, SORTINDEX=:41, corpid=:42, LINKURL=:43, spec_name1=:44, spec_name2=:45, spec_value1=:46, spec_value2=:47, POP_TYPE=:48, tag=:49, visitCount=:50, startTime=:51, endTime=:52, refer_price=:53, base_price=:54, parent_id=:55, isconfig=:56, short_name=:57, custom_image=:58, app_usercount=:59, app_totalCount=:60, buy_totalCount=:61, weight=:62, integral=:63, integral_Count=:64, is_userPrice=:65, short_name_color=:66, short_name_bold=:67, p_color=:68, p_color_pic=:69, charges_flag=:70, EXT_FIELD1=:71, EXT_FIELD2=:72, extFileld3=:73, extFileld4=:74, ext_fileld5=:75, ext_fileld6=:76, ext_fileld7=:77, ext_fileld8=:78, ext_fileld9=:79, ext_fileld10=:80, ext_fileld11=:81, ext_fileld12=:82, ext_fileld13=:83, ext_fileld14=:84, ext_fileld15=:85, ext_fileld16=:86, ext_fileld17=:87, ext_fileld18=:88, ext_fileld19=:89, ext_fileld20=:90, esalesno=:91, classtype=:92, game_type=:93, each_buy_count=:94, limit_buy_count=:95, start_limit_time=:96, end_limit_time=:97, channel_flag=:98, YDPRICE=:99, TQYD_DAYS=:100, SCOPE=:101, VD_ID1=:102, VD_ID2=:103, VD_ID3=:104, VD_ID4=:105, VD_ID5=:106, AUTO_GTCARD=:107, TV_NOTE=:108, MOBILE_NOTE=:109, IS_YYPRODUCT=:110, EXT_FILELD41=:111, WARRANTY=:112, AGENT_ID=:113, min_buynum=:114, MODEL_NO=:115, tp_spec_value1=:116, tp_spec_value2=:117 where PRODUCT_ID=:118

----- Call Stack Trace -----

ORA-07445这个是个内部错误，查看oracle官方文档解释 

ORA-07445:exception encountered: core dump [string] [string] [string] [string]

[string][string]

Cause:  An OS exception occurred which should resultin the creation of a core file.

This is aninternal error.

Action:  Contact your customer support representative.