NetWorker won't work: bad database header

[angelshark]

NetWorker crash!



Our NetWorker server won’t work anymore. It has stopped working apparently as a result of a media database corruption since the “daemon.log” has the following entry:

05/18/04 11:36:05 nsrmmdbd: WISS error: Unable to mount /nsr/mm/mmvolume6: bad database header

We have uninstalled all NetWorker 6.1.3 packages from our Sun Solaris 2.6 box, deleted everything from the /nsr folder except the /nsr/index and then we have done a clean reinstallation. We have configured our 2 jukeboxes and everything seemed to work fine, but without our current settings. We have recovered the media database and the server’s configuration (/nsr/res) from the bootstrap but the NetWorker services hung again. We have tried recovering from different bootstraps on different tapes but the result is similar.

Has anyone seen anything like this?

Do I have to start with a clean /nsr/res configuration and reconfigure everything from scratch?

Thank you for your help.

 

[605]

Recovering the resource files should not be a problem running the "automated" recovery:

recover -S bootstrap_ssid -a /nsr/res

As usual, you need to activate these changes.

What i am more concerned is the repeated failing of the media db. This would mean that you were working with a bad
index for a while.

With respect to media db issues, i urge you to update to NW 7.x.

 

[Chapter11]

Shouldn't his media database be repaired before upgrading?

angelshark, you're going to have to call your support vendor to deal with this, but it's likely they're going to tell you to keep recovering previous media databases until you get to one that isn't corrupted. Depending on how far back that is, it can be EXTREMELY nasty since you will have to rescan every tape that has had any changes since then.

 

[605]

Updating the media db should not an issue - he recovered the media db
already with no success - why do you want to keep it?

 

[Chapter11]

If someone has a few thousand volumes, writing off the entire media database is not something you do without making a very serious effort to recover what you can.

 

[605]

I agree of course. The question simply is - how far does he need to go back in time until he will find a good bootstrap. I would recommend to recover those from weekly fulls, if possible.

 

[wallace88]

Before you try to recover an older media database, try the media scavenging procedure:

(This was posted in an earlier article, use keyword search "scavenging")


First, check the file system to make sure that it is 'sane' before performing the media scavenging procedure. For example, in UNIX use fsck.

If the file system that has /nsr is out of space, then move some of the indexes to another file system as described in the tech bulletin, and also described later in this post.

Then, in your case, perform the following to reindex the media database:

1) In /nsr/mm remove: cmprssd
2) in /nsr/mm/mmvolume6 delete all files except the following: vol.0, ss.0, clients.0, vol.1, ss.1, clients.1, vol.2, ss.2, clients.2, vol.3, ss.3, clients.3, etc.
3) delete the /nsr/tmp directory
4) start NetWorker
5) run ps -ef | grep -i nsr, and see if nsrd, nsrmmdbd, nsrindexd, nsrmmds, nsrexecd have all started.
6) look at the daemon.log to see if NetWorker has started without problems.
7) If it stared correctly then run: nsrck -m and nsrim -X

If NetWorker fails to start, and you are still getting WISS errors, or any other errors during the startup, then you will have to recover the bootstrap, which will recover the res and media database from tape.

After NetWorker restarts (hopefully) then look at the media database, either using nwadmin to look at volume information, or with:

mminfo -B
mminfo -mv

If it doesn't look right.. then you may still have to recover the bootstrap to recover a non corrupted database.




Try looking at the /nsr/logs/daemon.log, or daemon.001-004 to try to determine when the WISS errors started. Then recover the bootstrap from before that date.

 

[605]

In general scavenging is a good idea. I just doubt whether it will work in this case.

As far as i know, scavenging rebuilds the media db's internal index (searchtables) but i am not sure whether it also can rebuild the header.