crashed centos server

We had a generator failure, a centos 5.x x64 server went down, when it came up, no booty booty... I tried booted to rescue mode, it would mount the LVM volume, no errors. But the chroot always failed.

So, I've decided to put the CD back in, run the upgrade, and let it install again... Looks good until it gets to glibc, when it suddenly gives me a readonly filesystem error.

Traceback (most recent call first): File "/tmp/treedir.28080/instimage/usr/lib/anaconda/yuminstall.py", line 137, in callback self.instLog.flush() IOError: [Errno 30] Read-only file system

Local variables in innermost frame: what: 4 hdr: h: glibc-2.5-42.x86_64 self: repo: anaconda-base-200910011652.x86_64 amount: 0

It seems to get through Quite a lot before it fails...

/mnt/sysimage/root/install.log:

Installing libgcc - 4.1.2-14.el5.i386

warning: libgcc-4.1.2-14.el5: Header V3 DSA signature: NOKEY, key ID e8562897

Installing libgcc - 4.1.2-14.el5.x86_64

Installing setup - 2.5.58-1.el5.noarch

Installing filesystem - 2.4.0-1.el5.centos.x86_64

Installing basesystem - 8.0-5.1.1.el5.centos.noarch

Installing pciutils-devel - 2.2.3-4.i386

Installing cracklib-dicts - 2.8.9-3.3.x86_64

Installing termcap - 1:5.5-1.20060701.1.noarch

A lot more before that, and it appears the newsest is at the top.

Thanks

From serverfault

• check dmesg, which while booting should be already on one of the consoles, try alt+f3 4 or 5.

  oods are there's a disk problem and ext3 is detecting it and going read-only.

  wzzrd : O RLY? "[Errno 30] Read-only file system" :-D

  : Yeah, I didn't miss it... but the filesystem was NOT readonly before it crashed, and I'm not sure why it's being mounted as RO now that I'm running the upgrade.
  From cagenut

• You may want to try badblocks and memtest86+ to make sure your physical hardware is still OK.

  From Dave