Cannot boot V490 with cannot assemble drivers for root, 1

[mrberry]

I have two identical V490s running Solaris 10. I just installed Veritas Storage Foundation Cluster File System Enterprise HA V4.1 on both. After the install one rebooted fine the other hangs and will not even boot into single user mode. I can boot off the cdrom.

The is what I get:

----------------start here---------------------------

ok boot -s
Boot device: /pci@9,600000/SUNW,qlc@2/fp@0,0/disk@w21000014c33be9d9,0:a File and args: -s
SunOS Release 5.10 Version Generic_118822-27 64-bit
Copyright 1983-2005 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
Cannot assemble drivers for root /pci@9,600000/SUNW,qlc@2/fp@0,0/disk@w21000014c33be9d9,0:a
Cannot mount root on /pci@9,600000/SUNW,qlc@2/fp@0,0/disk@w21000014c33be9d9,0:a fstype ufs

panic[cpu2]/thread=180e000: vfs_mountroot: cannot mount root

000000000180b950 genunix:vfs_mountroot+290 (800, 200, 200, 1836ac0, 11dc400, 1882000)
%l0-3: 0000000000000000 00000000010a886c 00000000010a8800 00000000010a8950
%l4-7: 00000000010a8800 00000000010a866c 00000000018b0400 0000000000000600
000000000180ba10 genunix:main+98 (1813d18, 1013400, 1835c00, 18aac00, 180e000, 1813c00)
%l0-3: 0000000070002000 0000000000000001 000000000180c000 000000000180e000
%l4-7: 0000000000000001 0000000001072c00 0000000000000060 0000000000000000

skipping system dump - no dump device configured
rebooting...
Resetting ...


RSC Alert: Host System has Reset

---------------------end here---------------------------

It then goes into a loop: it resets, tries to boot and resets again and so on.


I have a case open with Sun and the support guy is telling me to re-install (what is this windoze?!). He is now telling me (without any evidence to back it up) that the kernel must be corrupted. Clutching and straws come to mind......

I spoke with Veritas and they have no way for me to uninstall unless I can bring the machine up normally at least into single use mode (becuase the paths are not right under a cdrom boot). So I can't even try this.

Any help with this would be greatly appreicated.

 

[comtec17]

Unfortunately I would have to agree with the SUN support person if you cannot even get it to boot into single user or maintanence mode. With Solaris 10 I believe you can boot to the CD and do a repair install.

 

[Annihilannic]

becuase the paths are not right under a cdrom boot

What exactly do you mean here? Can you not mount the filesystems manually under /mnt, for example?

Annihilannic.

 

[mrberry]

Annihilannic, yes I can mount the root file system under /mnt. What I meant by this is that even if I could get it onto this machine the Veritas uninstall script would not work because the paths are now under /mnt (eg /opt is now /mnt/opt). The uninstall would fail and Veritas support tell me that there is no manual procedure to do a complete uninstall.

 

[Annihilannic]

Oh, I think they're lying to you again. Or else they've put it in the "too hard" basket...

Try commenting out rootdev:/pseudo/vxio@0:0 and set vxio:vol_rootdev_is_volume=1 in /etc/system and modifying /etc/fstab to point to the underlying slices on your disk.

If the partition table has partially gone missing (it should normally have been left more-or-less alone for the root disk, apart from adding the two Veritas slices) the Veritas install should have saved it in /etc/vx/reconfig.d/disk.d/cNtNdN/vtoc in a suitable format to be reapplied using fmthard -s vtocfile. Might be an idea to save the existing partition table (prtvtoc /dev/rdsk/cNtNdNs2 > some_file) before applying it so that you can go back if required.

If you manage to get the system booted you should be able to pkgrm the VRTS* packages.

Annihilannic.

 

[mrberry]

Annihilannic, that you for your reply.

The disks were never put under Veritas control. I just did the install without any configuration, then did the reboot as requested. One machine came up OK (and has booted OK again since), but this one failed to boot with errors above. I tried removing the Veritas entires in system, but that did not work either. They are:

forceload: drv/vxdmp
forceload: drv/vxio
forceload: drv/vxspec

I agree that this has been put in the "too hard" basket and I not willing to give up just yet. There has to be an underlying problem and my concern is if I just reinstall the OS from scratch what is to say that it won't happen again?

 

[Annihilannic]

Hmm... not so easy then. SUNW,qlc, what kind of device is that?

I wonder would rebuilding /etc/path_to_inst help, try a boot -ars perhaps (can't recall whether you have to remove it first) and experiment with the options.

I'm starting to flounder now... if you're in a hurry Sun's and comtec17's recommendations might be the quickest option. If the same happens again you won't have lost that much time; and then you can dig further if necessary?

Annihilannic.

 

[mrberry]

I wonder would rebuilding /etc/path_to_inst help

Already tried that and it didn't help :-(

try a boot -ars

Briefly tried this, but perhaps it needs a little more investiagtion.

Thanks.

 

[mrberry]

OK, got to the bottom of this one - and I did not have to re-install. If you just re-install you don't find the root cause, learn nothing and could end up in the same place again. Its too easy for support people to cop out with the reinstall line instead of doing their job - this is what we pay big money for? Sometimes it is the only solution which I am perfectly fine to accept, but the Sun support tech had given me no evidence that the kernel was corrupt.

The problem was that path_to_inst_file was corrupted. It was not enough to remove this file, I also had to remove the file path_to_inst.old as it was also currupt and gets used if the path_to_inst is not present. Once I removed these two files the path_to_inst was rebuilt on boot and the system now boots.

I hope this information is of use to someone else.

 

[mrberry]

I wonder would rebuilding /etc/path_to_inst help, try a boot -ars perhaps (can't recall whether you have to remove it first) and experiment with the options.

Annihilannic, thank you for your assistance, you were on the right track.