机房断电后,启动WEB服务器后,几台考试WEB始终报错,无法连接数据库。
再看小机状态正常,这台驾驶人考试监管DB上运行着AIX 6系统,Oracle软件安装在本地目录/oracle,数据库文件则使用挂载的存储空间(FC+富士通DX90 S2)。
开始排查数据库,telnet到小机上:
AIX Version 6
Copyright IBM Corporation, 1982, 2010.
login: root
root's Password:
*******************************************************************************
* *
* Welcome to AIX Version 6.1! *
* *
* Please see the README file in /usr/lpp/bos for information pertinent to *
* this release of the AIX Operating System. *
* *
* *
*******************************************************************************
先去oracle用户下,查看实例发布库没有正常启动。再看磁盘空间,/oradata目录并没有正常mount,原来问题在这,实例异常原因在这。
# df -g
Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/hd4 10.00 9.33 7% 2288 1% /
/dev/hd2 5.00 2.87 43% 38144 3% /usr
/dev/hd9var 3.00 2.34 22% 748 1% /var
/dev/hd3 5.00 4.02 20% 2974 1% /tmp
/dev/fwdump 10.00 9.69 4% 37 1% /var/adm/ras/platform
/dev/hd1 5.00 4.84 4% 37 1% /home
/proc - - - - - /proc
/dev/hd10opt 5.00 4.73 6% 3209 1% /opt
/dev/oracle_lv 20.00 14.07 30% 100753 3% /oracle
/dev/oradata_lv 7190.00 35.27 100% 205 1% /oradata
查看vg,vg状态正常:
# lsvg
rootvg
Datavg
# lsvg -l datavg
datavg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
oradata_lv jfs2 2048 2048 3 closed/syncd /oradata
oracle_lv jfs2 40 40 1 open/syncd /oracle
install_lv jfs2 20 20 1 open/syncd /oracle_install
loglv00 jfs2log 1 1 1 open/syncd N/A
到此,发现datavg的lv state为closed,尝试手动挂载oradata_lv,报错。
# mount /oradata
Replaying log for /dev/oradata_lv.
mount: 0506-324 Cannot mount /dev/oradata_lv on /oradata: The media is not formatted or the format is not correct.
0506-342 The superblock on /dev/oradata_lv is dirty. Run a full fsck to fix.
显然,断电关机后,造成分区损坏,无法正确识别该分区,只能执行fsck对磁盘分区进行修复。
# fsck /oradata
The current volume is: /dev/oracle_lv
Primary superblock is valid.
*** Phase 1 - Initial inode scan
*** Phase 2 - Process remaining directories
*** Phase 3 - Process remaining files
*** Phase 4 - Check and repair inode allocation map
*** Phase 5 - Check and repair block allocation map
File system is clean.
Superblock is marked dirty; FIX? y
All observed inconsistencies have been repaired.
修复完成后,mount该lv到挂载点。
# mount /oradata
然后重启实例,恢复WEB服务。至此,这起因断电引起的数据库异常得以解决。