9840 and 9940B-Tapes got marked as full

[spaceball]

Hi,
since about 1 week I've noticed that lot's of our 9840 Tapes (20GB) and 9940B (200GB) Tapes got marked as full even if they have written less than 20 % of its (possible) capacity. No matter if new labeled or already in use Tapes.
Some of them have some savesets others just a single one, than it got marked as full.
There is no relation to a networker problem during the shown savesets got saved, all I can say is that we had some media-db problems last week and had to cross-check the db by nsrim -X and nsrck -m.
There are also no aborted savesets located on those volumes.
Today I relized, that right after networker finished writing one single saveset of 2,5GB totalsize, the volume got marked as full and readonly in the NW-GUI.

Our environment:
Networker server 7.1.2 and Alphastor 3 SR2 incl. Patches, both togehter installed on one Solaris-Host; STK-Powderhorn controlled by ACSLS; 9840 and 9940B tapedrives connected by Fibrechannel (SAN)

We upgraded the microcode of our drives more than 3 weeks ago and backups went fine afterwards until last week...
Today we switched off and on again, the power of all tapedrives, one by one: No change.

Any suggestions?
Kind Regards Thorsten

 

[605]

Did you check the logs files when the problem occured first time?

 

[spaceball]

Yes, the daemon.log says:
04/20/05 07:22:24 nsrd: media notice: 9940 tape 200162 on /dev/rmt/22cbn is full
04/20/05 07:22:24 nsrd: media notice: 9940 tape 200162 used 13 GB of 200 GB capacity.

As soon as I set the volume manually on apendable, it would be re-used and data would be written on it. But after one written saveset it say again that it is full.
It looks like the tape got labeled in a situation where the LNS and AS worked not stable. Up to now I had no chance to bring the system down again because of big delayed backup-jobs to do. Once I get the chance, I'd like to crunch the /nsr/tmp/ files and do a nwasdbsync.
So, right now It looks like I have no alternative than label new tapes on and on...

 

[605]

Unless your system date is incorrect, this entry can not be the FIRST one when the problem occured.

 

[spaceball]

You're right, this is just an actual log-entry, showing that the fault is still there.
Our actual guess is that it might have to do with a firmware update of our 9840 and 9940B tape-drive, done the week after easter.
The reason why the fault comes up so late could be that we start manualy relabeling jobs around April, 15th. We've several "scratch-pools" (one for each robot-location and type of tape) and around the 15th, some produktiv pools grow's so much that we had to fill them up a bit. (no expired tapes in those prod-pools available for relabeling...) So If the "scratch-volumes" once got labeled under an old, bugy drive-firmware, it could cause a problem during relabeleing this time, with the actual loaded firmware-version. (faulty blocksize???) We already escallate it to Storagetek.

Kind Regards Thorsten

 

[spaceball]

Hi,

We solved the problem: There was a visual-SAN agent (damon) for Solaris running on the Legato Networker-Server.
The daemon was installed some days ago, close to the time when the volume-problem occures first time. So we deinstalled the daemon and from that time on all tapes got written full again, as expected.

Kind Regards Thorsten