I was called in recently to help with the recovery of an old SCO 3.2v4.2 system that had crashed. The crash had initially been caused by a power supply failure, but after replacing that hardware, the machine would not boot – it just hung after the kernel i/o buffers message. As a common cause for that is simply a missing or damaged inittab, I thought we might be able to fix it by booting from floppies and doing a manual repair. Unfortunately, there were no emergency boot floppies.
After some scouting around, we did find the original install floppies, so I used those to boot with. As the hard disks were on an AMI Raid controller, we needed an amird btld, but that was easy to find on the internet. I followed the install prompts until it got to
1. Initial Install
2. Update
3. Exit
and there I typed “shell” to get to a shell prompt. I then did:
fsck -y /dev/hd0root
mount /dev/hd0root /mnt
Damage assesment
On these old systems, you don’t have access to niceties like “ls” from the boot floppy – you need to do things like “/hd0root/mnt/bin/ls”. Unfortunately, I couldn’t do that, and an “echo *” showed why: there was no bin directory! It was just not there. The /etc directory was intact, and I could cd to it and with “echo *” see that there was little or nothing missing there, but the rest of the system was in quite a mess. In addition to /bin being gone, the /usr directory was no longer a directory. It was just a regular file so I couldn’t cd to it or anything under it (the difference between a file and a directory is just bits set in the inode – damage to the inode table can cause this easily). No wonder the poor machine couldn’t boot. Given the amount of damage, it was surprising that we were able to fsck and mount the disk at all!
At this point we stopped to discuss options. I could, of course, just use the install media to reinstall. Unfortunately, the client was not sure of the state of his backups and there was the possibility that some important data might be on the root filesystem and not backed up. I had seen data directories, but no one could remember what should and shouldn’t have been there. I had also noticed in /etc that LoneTarhad been at least installed on this system. If they had been actually using it too, that would be great news, but the client did not know and no Airbag boot diskettes were to be found. It’s not unusual to see expired software on these old systems. It could have been installed as a demo and not been used, so I wasn’t too excited by this
Reading the tapes
We decided to take the system’s tape drive and attempt to read the tapes on a Linux system. Toward that end, I installed a demo copy of Microlite Edge that I had with me on CD. If the tapes were Lone-Tar or any supertar or just tar, the Edge demo would be able to read it. Of course I also could have just downloaded a LoneTar demo, but I had the Microlite product in my pocket, so I used that. The tapes were able to be read, and I quickly determined that they were indeed LoneTar tapes. That was good news, as the verify capability of this type of software meant there was far less chance of a bad backup. I also determined that the newest tape was a few weeks old, which was not really good news as it meant that we still couldn’t chance a fresh install and restore of tapes.
OSR5 to the rescue!
As I expected problems like this, I had also brought a 5.0.6 demo set with me. We found an unused IDE machine of recent vintage, and I did a quick basic install of the SCO 5.0.6 on that. We then removed the drive, and brought it to the crashed machine. We attached it to the ide controller, reattached the tape drive, and booted. I then put in the amird btld for OSR5, and did:
mount /dev/fd0135ds18 /mnt
btldinstall /mnt
to install the latest amird driver (5.0.6 does have a built-in version but I wanted to be sure we had the latest). I followed this with a “mkdev hd” to add in the crashed system’s root disk. You have to be careful with this. You might want to read:
Transferring a hard drive
Adding a hard drive (covers drives with existing data also)
Drive Geometry
I had no problem adding the hard drive and mounting it. I considered using fsdb to put the /usr filesystem back, but the examination of the tape on the Linux box had shown us that actual user data was elsewhere and /usr was just system files, so I just removed it entirely. Why look for trouble when you don’t need to? Examination of /oldysys/etc/default/filesys and /oldsys/etc/conf/cf.d/mscsi showed what other scsi devices and filesystems had been present on the old system, and also told me where the tape was located so that I could quickly add that. There was no need to bother with the other filesystems at this time; our concern was the root.
Because I still had concerns about the actual hardware, I decided to restore the root to the ide drive first. I used the Edge software to do this, excluding the directories that were on other filesystems (gained from looking at /etc/default/filesys on the mounted drive). The easy way to do this is to create a file containing the directories to be excluded, one per line. I then created a directory to hold the restore, cd’d to that directory, and did:
edge xvXf /tmp/excludethese /dev/rStp0
The purpose of this was just to save time should we have hardware problems down the road. So now I had the old crashed system mounted at /oldsys and a restore of it at /restored on the ide. I then did:
cd /restored
find . | cpio -pdmv /oldys
This copied everything over, but would not overwrite anything newer that was already there. We then shutdown the system, removed the ide drive, and rebooted. The other filesystems needed to be fsck’d, but after that, everything came up successfully and an examination through application reports seemed to indicate no missing data. I showed them how to make boot diskettes from the LoneTar menu (“ltmenu”) and suggested that they might want to consider upgrading to the latest version for that system. I also suggested regularly copying important data across the network to another system for more redundant backup and that the software would probably run better on a 5.0.6 system, but this machine is just for historical access so they prefer to leave it as is.
We later discovered that the power supply failure had also killed the secondary ide controller on this system, but we were able to switch the CD to be a slave off the master to get by that.
The entire recovery took almost six hours, though a lot of that was just waiting for tapes to read.
A.P. Lawrence provides SCO Unix and Linux consulting services http://www.pcunix.com
