环境: OEL 5.7 + Oracle 11.2.0.3
1. 模拟 ORA-600 [4194][][]故障
2. 使用 bbed 处理
3. 尝试启动数据库
1. 模拟 ORA-600 [4194][][]故障
本次实验环境模拟的是相关的 undo/rollback block 在 system rollback segment 上的场景:
- SQL> alter database open;
- alter database open
- *
- ERROR at line 1:
- ORA-01092: ORACLE instance terminated. Disconnection forced
- ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [],
- [], [], [], []
- Process ID: 6670
- Session ID: 26 Serial number: 37
注意:
1. 这种场景无法通过使用隐藏参数 (_offline_rollback_segments 和_corrupted_rollback_segments) 来绕过问题;
2. 我这里实验验证是通过 bbed 修改 system 文件 128 号块的 ktuxc.ktuxcfbp[0].ktufbuba.kubarec 来模拟 4194 故障;
3. 数据库版本不一样, 所在块号可能不同, 比如 9i/10g 就都是 9 号块;
4. 这个实验我在 11.2.0.3 这个版本的测试, 4194 后面参数是没有任何值的, 而在 10.2.0.4 测试模拟同样故障后面是有参数值的;
5. 请一定不要在任何生产环境尝试做此类操作.
查询 system 的回滚段头所在块:
select HEADER_FILE, HEADER_BLOCK, BLOCKS, EXTENTS from dba_segments where segment_name = 'SYSTEM' and SEGMENT_TYPE = 'ROLLBACK';
查看数据库的 alert 日志:
- Fri May 17 04:59:13 2019
- SMON: enabling cache recovery
- Errors in file /opt/App/oracle/diag/rdbms/test/test/trace/test_ora_6670.trc (incident=4969):
- ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
- Incident details in: /opt/App/oracle/diag/rdbms/test/test/incident/incdir_4969/test_ora_6670_i4969.trc
- Use ADRCI or Support Workbench to package the incident.
- See Note 411.1 at My Oracle Support for error and packaging details.
- Fri May 17 04:59:17 2019
- Dumping diagnostic data in directory=[cdmp_20190517045917], requested by (instance=1, osid=6670), summary=[incident=4969].
- Block recovery from logseq 2, block 66292 to scn 1006025
- Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0
- Mem# 0: +DATADG/test/onlinelog/group_2.341.1008434769
- Block recovery stopped at EOT rba 2.66294.16
- Block recovery completed at rba 2.66294.16, scn 0.1006022
- Block recovery from logseq 2, block 66292 to scn 1006021
- Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0
- Mem# 0: +DATADG/test/onlinelog/group_2.341.1008434769
- Block recovery completed at rba 2.66294.16, scn 0.1006022
- Undo initialization errored: err:600 serial:0 start:1356834 end:1362534 diff:5700 (57 seconds)
- Errors in file /opt/App/oracle/diag/rdbms/test/test/trace/test_ora_6670.trc:
- ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
- Errors in file /opt/App/oracle/diag/rdbms/test/test/trace/test_ora_6670.trc:
- ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
- Error 600 happened during db open, shutting down database
- USER (ospid: 6670): terminating the instance due to error 600
- Instance terminated by USER, pid = 6670
- ORA-1092 signalled during: alter database open...
- opiodr aborting process unknown ospid (6670) as a result of ORA-1092
- Fri May 17 04:59:20 2019
- ORA-1092 : opitsk aborting process
2. 使用 bbed 处理
需要处理:
- ktuxc.ktuxcnfb
- ktuxc.ktuxcfbp[0].ktufbuba.kubadba
- kcbh.flg_kcbh
- kcbh.chkval_kcbh
核心命令:
- set offset ktuxc.ktuxcnfb
- modify 0x0000
- set offset ktuxc.ktuxcfbp[0].ktufbuba
- modify 0x00000000
- set offset kcbh.flg_kcbh
- modify 0x00
- set offset kcbh.chkval_kcbh
- modify 0x0000
实际处理过程如下:
- [oracle@OEL-ASM tmp]$ bbed parfile=bbed.par
- Password:
- BBED: Release 2.0.0.0.0 - Limited Production on Fri May 17 05:05:22 2019
- Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
- ************* !!! For Oracle Internal Use only !!! ***************
- BBED> map
- File: /tmp/system01.bak (1)
- Block: 1 Dba:0x00400001
- ------------------------------------------------------------
- Data File Header
- struct kcvfh, 860 bytes @0
- ub4 tailchk @8188
- BBED> set dba 1,128
- DBA 0x00400080 (4194432 1,128)
- BBED> map
- File: /tmp/system01.bak (1)
- Block: 128 Dba:0x00400080
- ------------------------------------------------------------
- Unlimited Undo Segment Header
- struct kcbh, 20 bytes @0
- struct ktech, 72 bytes @20
- struct ktemh, 16 bytes @92
- struct ktetb[6], 48 bytes @108
- struct ktuxc, 104 bytes @4148
- struct ktuxe[255], 10200 bytes @4252
- ub4 tailchk @8188
- BBED> p kcbh
- struct kcbh, 20 bytes @0
- ub1 type_kcbh @0 0x0e
- ub1 frmt_kcbh @1 0xa2
- ub1 spare1_kcbh @2 0x00
- ub1 spare2_kcbh @3 0x00
- ub4 rdba_kcbh @4 0x00400080
- ub4 bas_kcbh @8 0x000f59bb
- ub2 wrp_kcbh @12 0x0000
- ub1 seq_kcbh @14 0x01
- ub1 flg_kcbh @15 0x04 (KCBHFCKV)
- ub2 chkval_kcbh @16 0x3626
- ub2 spare3_kcbh @18 0x0000
- BBED> p ktuxc
- struct ktuxc, 104 bytes @4148
- struct ktuxcscn, 8 bytes @4148
- ub4 kscnbas @4148 0x000f26af
- ub2 kscnwrp @4152 0x0000
- struct ktuxcuba, 8 bytes @4156
- ub4 kubadba @4156 0x0040021f
- ub2 kubaseq @4160 0x001b
- ub1 kubarec @4162 0x03
- sb2 ktuxcflg @4164 1 (KTUXCFSK)
- ub2 ktuxcseq @4166 0x001b
- sb2 ktuxcnfb @4168 1
- ub4 ktuxcinc @4172 0x00000000
- sb2 ktuxcchd @4176 3
- sb2 ktuxcctl @4178 91
- ub2 ktuxcmgc @4180 0x8002
- ub4 ktuxcopt @4188 0x7ffffffe
- struct ktuxcfbp[0], 12 bytes @4192
- struct ktufbuba, 8 bytes @4192
- ub4 kubadba @4192 0x0040021f
- ub2 kubaseq @4196 0x001b
- ub1 kubarec @4198 0x0d
- sb2 ktufbext @4200 3
- sb2 ktufbspc @4202 5510
- struct ktuxcfbp[1], 12 bytes @4204
- struct ktufbuba, 8 bytes @4204
- ub4 kubadba @4204 0x00000000
- ub2 kubaseq @4208 0x001a
- ub1 kubarec @4210 0x02
- sb2 ktufbext @4212 2
- sb2 ktufbspc @4214 7958
- struct ktuxcfbp[2], 12 bytes @4216
- struct ktufbuba, 8 bytes @4216
- ub4 kubadba @4216 0x00000000
- ub2 kubaseq @4220 0x0018
- ub1 kubarec @4222 0x3a
- sb2 ktufbext @4224 0
- sb2 ktufbspc @4226 2062
- struct ktuxcfbp[3], 12 bytes @4228
- struct ktufbuba, 8 bytes @4228
- ub4 kubadba @4228 0x00000000
- ub2 kubaseq @4232 0x0000
- ub1 kubarec @4234 0x00
- sb2 ktufbext @4236 0
- sb2 ktufbspc @4238 0
- struct ktuxcfbp[4], 12 bytes @4240
- struct ktufbuba, 8 bytes @4240
- ub4 kubadba @4240 0x00000000
- ub2 kubaseq @4244 0x0000
- ub1 kubarec @4246 0x00
- sb2 ktufbext @4248 0
- sb2 ktufbspc @4250 0
- BBED> set offset 4168
- OFFSET 4168
- BBED> p
- ktuxc.ktuxcnfb
- --------------
- sb2 ktuxcnfb @4168 1
- BBED> modify 0x0000
- Warning: contents of previous BIFILE will be lost. Proceed? (Y/N) y
- File: /tmp/system01.bak (1)
- Block: 128 Offsets: 4168 to 4679 Dba:0x00400080
- ------------------------------------------------------------------------
- 00000000 00000000 03005b00 02800100 68000000 feffff7f 1f024000 1b000d00
- 03008615 00000000 1a000200 0200161f 00000000 18003a00 00000e08 00000000
- 00000000 00000000 00000000 00000000 00000000 14000000 1e024000 28570f00
- 00000000 09005800 00000000 00000000 00000000 01000000 00000000 13000000
- 1a024000 bc260f00 00000000 09005f00 00000000 00000000 00000000 01000000
- 00000000 13000000 1b024000 db260f00 00000000 09005c00 00000000 00000000
- 00000000 01000000 00000000 13000000 1a024000 b2260f00 00000000 09000400
- 00000000 00000000 00000000 01000000 00000000 13000000 1a024000 b4260f00
- 00000000 09000600 00000000 00000000 00000000 01000000 00000000 13000000
- 1b024000 e5260f00 00000000 09000a00 00000000 00000000 00000000 01000000
- 00000000 13000000 1a024000 b7260f00 00000000 09006100 00000000 00000000
- 00000000 01000000 00000000 13000000 1b024000 c2260f00 00000000 09005900
- 00000000 00000000 00000000 02000000 00000000 14000000 1f024000 30570f00
- 00000000 09005b00 00000000 00000000 00000000 01000000 00000000 13000000
- 1b024000 ce260f00 00000000 09001200 00000000 00000000 00000000 01000000
- 00000000 13000000 1b024000 e7260f00 00000000 09001000 00000000 00000000
- <32 bytes per line>
- BBED> p
- ktuxc.ktuxcnfb
- --------------
- sb2 ktuxcnfb @4168 0
- BBED> set offset 4192
- OFFSET 4192
- BBED> p
- ktuxc.ktuxcfbp[0].ktufbuba.kubadba
- ----------------------------------
- ub4 kubadba @4192 0x0040021f
- BBED> modify 0x00000000
- File: /tmp/system01.bak (1)
- Block: 128 Offsets: 4192 to 4703 Dba:0x00400080
- ------------------------------------------------------------------------
- 00000000 1b000d00 03008615 00000000 1a000200 0200161f 00000000 18003a00
- 00000e08 00000000 00000000 00000000 00000000 00000000 00000000 14000000
- 1e024000 28570f00 00000000 09005800 00000000 00000000 00000000 01000000
- 00000000 13000000 1a024000 bc260f00 00000000 09005f00 00000000 00000000
- 00000000 01000000 00000000 13000000 1b024000 db260f00 00000000 09005c00
- 00000000 00000000 00000000 01000000 00000000 13000000 1a024000 b2260f00
- 00000000 09000400 00000000 00000000 00000000 01000000 00000000 13000000
- 1a024000 b4260f00 00000000 09000600 00000000 00000000 00000000 01000000
- 00000000 13000000 1b024000 e5260f00 00000000 09000a00 00000000 00000000
- 00000000 01000000 00000000 13000000 1a024000 b7260f00 00000000 09006100
- 00000000 00000000 00000000 01000000 00000000 13000000 1b024000 c2260f00
- 00000000 09005900 00000000 00000000 00000000 02000000 00000000 14000000
- 1f024000 30570f00 00000000 09005b00 00000000 00000000 00000000 01000000
- 00000000 13000000 1b024000 ce260f00 00000000 09001200 00000000 00000000
- 00000000 01000000 00000000 13000000 1b024000 e7260f00 00000000 09001000
- 00000000 00000000 00000000 01000000 00000000 13000000 1c024000 15290f00
- <32 bytes per line>
- BBED> p
- ktuxc.ktuxcfbp[0].ktufbuba.kubadba
- ----------------------------------
- ub4 kubadba @4192 0x00000000
- BBED> map
- File: /tmp/system01.bak (1)
- Block: 128 Dba:0x00400080
- ------------------------------------------------------------
- Unlimited Undo Segment Header
- struct kcbh, 20 bytes @0
- struct ktech, 72 bytes @20
- struct ktemh, 16 bytes @92
- struct ktetb[6], 48 bytes @108
- struct ktuxc, 104 bytes @4148
- struct ktuxe[255], 10200 bytes @4252
- ub4 tailchk @8188
- BBED> p kcbh
- struct kcbh, 20 bytes @0
- ub1 type_kcbh @0 0x0e
- ub1 frmt_kcbh @1 0xa2
- ub1 spare1_kcbh @2 0x00
- ub1 spare2_kcbh @3 0x00
- ub4 rdba_kcbh @4 0x00400080
- ub4 bas_kcbh @8 0x000f59bb
- ub2 wrp_kcbh @12 0x0000
- ub1 seq_kcbh @14 0x01
- ub1 flg_kcbh @15 0x04 (KCBHFCKV)
- ub2 chkval_kcbh @16 0x3626
- ub2 spare3_kcbh @18 0x0000
- BBED> set offset 15
- OFFSET 15
- BBED> p
- kcbh.flg_kcbh
- -------------
- ub1 flg_kcbh @15 0x04 (KCBHFCKV)
- BBED> modify 0x00
- File: /tmp/system01.bak (1)
- Block: 128 Offsets: 15 to 526 Dba:0x00400080
- ------------------------------------------------------------------------
- 00263600 00000000 00000000 00000000 00000000 00060000 002f0000 00201000
- 00030000 00070000 00080000 001f0240 00000000 00030000 00000000 00000000
- 00000000 00000000 00000000 00060000 00000000 00000000 00000000 40810040
- 00070000 00880040 00080000 00100240 00080000 00180240 00080000 00200240
- 00080000 00280240 00080000 00000000 00000000 00000000 00000000 00000000
- 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
- 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
- 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
- 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
- 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
- 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
- 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
- 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
- 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
- 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
- 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
- <32 bytes per line>
- BBED> p
- kcbh.flg_kcbh
- -------------
- ub1 flg_kcbh @15 0x00 (NONE)
- BBED> set offset 16
- OFFSET 16
- BBED> p
- kcbh.chkval_kcbh
- ----------------
- ub2 chkval_kcbh @16 0x3626
- BBED> modify 0x0000
- File: /tmp/system01.bak (1)
- Block: 128 Offsets: 16 to 527 Dba:0x00400080
- ------------------------------------------------------------------------
- 00000000 00000000 00000000 00000000 00000000 06000000 2f000000 20100000
- 03000000 07000000 08000000 1f024000 00000000 03000000 00000000 00000000
- 00000000 00000000 00000000 06000000 00000000 00000000 00000040 81004000
- 07000000 88004000 08000000 10024000 08000000 18024000 08000000 20024000
- 08000000 28024000 08000000 00000000 00000000 00000000 00000000 00000000
- 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
- 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
- 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
- 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
- 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
- 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
- 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
- 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
- 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
- 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
- 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
- <32 bytes per line>
- BBED> p
- kcbh.chkval_kcbh
- ----------------
- ub2 chkval_kcbh @16 0x0000
- BBED> sum apply
- Check value for File 1, Block 128:
- current = 0x0000, required = 0x0000
- BBED> exit
- [oracle@OEL-ASM tmp]$
3. 尝试启动数据库
bbed 修改完成, 再次尝试启动数据库成功:
- SQL> startup mount
- ORACLE instance started.
- Total System Global Area 1703624704 bytes
- Fixed Size 2229024 bytes
- Variable Size 989859040 bytes
- Database Buffers 704643072 bytes
- Redo Buffers 6893568 bytes
- Database mounted.
- SQL> alter database open;
- Database altered.
其他类 ORA-600 [4194]的场景, 可参考之前的随笔:
Oracle 创建新 undo 表空间最佳实践(包含段检查)
来源: https://www.cnblogs.com/jyzhao/p/10879253.html