transport rejected (-2) server crashed

[stevenriz]

Hi, this is the second time the server crashed in two days. First time I did a cold restart of the system. Couldn't shut it down so I just turned the key. All was well until I passed a lot of data over the bus.... Then lower down you will see results of the iostat -En command. I just can't tell yet if it is the disks or the scsi card or both or something else alltogether! Help if you can. I will be doing my own diagnostics in the mean time. Will return results either way. thanks!!

I am getting these messages...
Apr 2 02:46:48 application unix: sf1: sf1: Target 0x4 Reset Failed. Ret=105
Apr 2 02:46:50 application unix: sf1: sf:Target driver initiated lip
Apr 2 02:46:50 application unix: ID[SUNWssa.socal.link.5010] socal0: port 1: Fi
bre Channel is OFFLINE
Apr 2 02:46:50 application unix: ID[SUNWssa.socal.link.6010] socal0: port 1: Fi
bre Channel Loop is ONLINE
Apr 2 02:46:50 application unix: sf1: target 0x4 al_pa 0xe1 offlined
Apr 2 02:46:50 application unix: WARNING: /sbus@2,0/SUNW,socal@d,10000/sf@1,0/s
sd@w22000020379c3242,0 (ssd2):
Apr 2 02:46:50 application unix: requeue of command fails (fffffffe)
Apr 2 02:46:50 application unix:
Apr 2 02:46:50 application unix: WARNING: /sbus@2,0/SUNW,socal@d,10000/sf@1,0/s
sd@w22000020379c3242,0 (ssd2):
Apr 2 02:46:51 application unix: requeue of command fails (fffffffe)
Apr 2 02:46:51 application unix:
Apr 2 02:46:51 application unix: WARNING: /sbus@2,0/SUNW,socal@d,10000/sf@1,0/s
sd@w22000020379c3242,0 (ssd2):
Apr 2 02:46:51 application unix: requeue of command fails (fffffffe)
Apr 2 02:46:51 application unix:
Apr 2 02:46:51 application unix: WARNING: /sbus@2,0/SUNW,socal@d,10000/sf@1,0/s
sd@w22000020379c3242,0 (ssd2):
Apr 2 02:46:51 application unix: transport rejected (-2)
Apr 2 02:46:51 application unix:
Apr 2 02:46:51 application unix: WARNING: /sbus@2,0/SUNW,socal@d,10000/sf@1,0/s
sd@w22000020379c3242,0 (ssd2):
Apr 2 02:46:51 application unix: transport rejected (-2)

=============
# iostat -En








c2t6d0 Soft Errors: 0 Hard Errors: 3 Transport Errors: 0
Vendor: TOSHIBA Product: XM6201TASUN32XCD Revision: 1103 Serial No: 12/12/97
RPM: 0 Heads: 0 Size: 18446744073.71GB <-8589934591 bytes>
Media Error: 0 Device Not Ready: 2 No Device: 1 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0

c0t2d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST318203FSUN18G Revision: 034A Serial No: 0031J80486
RPM: 7200 Heads: 19 Size: 18.11GB <18110967808 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0

c0t0d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST318203FSUN18G Revision: 034A Serial No: 0031J95276
RPM: 7200 Heads: 19 Size: 18.11GB <18110967808 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0

c1t4d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 7942
Vendor: SEAGATE Product: ST318203FSUN18G Revision: 034A Serial No: 0031J95352
RPM: 7200 Heads: 19 Size: 18.11GB <18110967808 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0

c1t6d0 Soft Errors: 0 Hard Errors: 1 Transport Errors: 0
Vendor: SEAGATE Product: ST318203FSUN18G Revision: 034A Serial No: 0031J85712
RPM: 7200 Heads: 19 Size: 18.11GB <18110967808 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 1 Recover

#

THANKS!!!!

 

[Chacalinc]

It looks like as disk drive failure. I can see you have more than one disk drive in the same bus, but only one device is giving you errors. Get the drive out of the system and reproduce the error with another disk drive. I'm pretty sure you will not have errors/crash.

Cheers.

 

[stevenriz]

Yes I Was thinking that Chacalinc but that drive has transport errors only on it. Wouldn't that be the bus or the scsi board? Now c2t6d0 and c1t6d0 both have hard errors. Do you know what that could mean? Thanks I am still working on it!
Steve

 

[Chacalinc]

transport error could be an electronic issue at circuit board on disk drive... if the same error is replicated to all disk drives then the problem could be in the scsi cable or the scsi card.

Now, regarding to the hard errors it could be sectors in the disk, sector reassigned automatically.. not a problem after all, it's a common issue... if a sector is found with problem, it's marked as bad and reassigned, all OS have that capability.

Cheers.

 

[stevenriz]

Ok I understand now. I went to replace the disk with an unused disk from another system. No go. The bad disk is in an e3500 and the other systems we have are e250s. So I poked around a bit more and found I didn't need some data on one of the drives, wiped it out and re-file-system-ed it and am copying the data off the bad disk now. I can do that since after a cold restart, the bad disk came up as "stable" rather then "clean" I guess I am very lucky there....... Thanks for all the help!!

 

[Chacalinc]

great! hope my posts helped you.

Cheers.

 

[stevenriz]

Sure did. Come to find out the e3500 has fiber drives. The reason why I couldn't use an e250 drive in it. Oh well, I got a couple used for $100ea to have on hand. The system crashed a couple times during the copy but it is still copying. Whew, no having to reinstall the app!!

 

[Chacalinc]

hehehe.. I'm sure re-installing the app should be a pain.
Post your result after you finish.

 

[stevenriz]

All is well after I unmounted that disk. New disks coming tomorrow and will replace. NO LOSS OF DATA!!! How lucky can one get!! I feel blessed by the Gods!

Thanks!
Steve

 

[Chacalinc]

hehehe great!!

 

[stevenriz]

I am impressed with Sun or whoever for the way they have their drives built. To have many states a drive can be in. That when a PC drive fails, it really fails. Thre usually isn't any inbetween there.

 

[Chacalinc]

It must be so !! If you think about it, a SUN server (or other Unix) are made for critical services so if a simple drive failure marks the drive as totally bad, what happen with the critical data? you don't use a PC for critical data... (or you should not) because of PC are not done for that porpuse.

Cheers.