Cannot restore from incomplete drive backup 1

[billy1]

Hi,

When a backup fails during the backup of a certain drive on a server. It seems that we cannot restore anything from that drive.
e.g. 90% of drive E: on server X completes before it crashes for some reason.
We go to restore a file from the E:\ drive - the entire E:\ drive is not available for restore.

Any ideas ?

Using Netbackup 5.1 MP6

 

[billy1]

OK, can anybody even verify if this is the case with Netbackup ?
That you can only restore from a drive as long as the entire drive is backed up ?? Does anybody else have this issue ?

Thanks.

 

[Stumpr]

A backup has to finish in order to be available for a restore. If a backup fails to finish then whatever has been backed up to that point is expired and is not available for restores. I confirm that is how NetBackup works.



Bob Stump
VERITAS
"Ain't it the truth? Ain't it the truth?" Bert Lahr, Wizard of Oz 1939

 

[billy1]

Thanks for confirming that Bob, I appreciate it.

Just one more thing. Are you aware of any clients that have issues with their drives that constantly fall into a DOWN state ?

We have that and have tried everything. Our backups start in the evening but as soon as they do, our drives (shared drives) will show as being DOWN on certain media servers. We can UP the drive and it re-enables but we have to keep UPing them as they keep going down. And completely randomly aswell, different media servers all the time.

We have updated tape drivers
Patched netbackup to the latest version
Had the hardware checked by an engineer
Replaced an MDR we had connecting to the SAN
Set up zoning on our brocades
Updated firmware on the library

This seemed to fix the problem as we were running with 100% backups for a couple of weeks. But it looks like the problem has come back again.

Any ideas ?
Thanks !

 

[Stumpr]

The most common problem is drive mapping the Netbackup definition to the physical library definition and the corresponding OS definition. This will sound tedious but what I would do is physicallly mount a tape in the drive and confirm the proper mapping on all media servers.



Bob Stump
VERITAS
"Ain't it the truth? Ain't it the truth?" Bert Lahr, Wizard of Oz 1939

 

[billy1]

Thanks Bob.

We can mount a tape no problem but how do we confirm the proper mapping on each server ? Where is this seen ?

We have many backups that run to a library of 4 tape drives and so overnight tapes are being mounted and unmounted many times. If it is trying to map the netbackup definition to the library definition and corresponding OS definition every time it tries to mount a tape then if it works the first time should it not work every other time - given that the definitions haven't changed ?

Thanks again for your response !

 

[Stumpr]

you need to set up persistent binding. Do you have a mixture of UNIX and windows?



Bob Stump
VERITAS
"Ain't it the truth? Ain't it the truth?" Bert Lahr, Wizard of Oz 1939

 

[billy1]

No these are all windows servers.
What is persistent binding and how can we set it up ?

Cheers!

 

[Stumpr]

Billy1,
I like you. I am going to give you the hook up.

DOCUMENTATION: How to delete "ghost" or "phantom" devices from the Windows Device Manager when using Veritas NetBackup (tm)

http://support.veritas.com/docs/259595

Details:
Manual:
Veritas NetBackup (tm) 5.0 Media Manager Device Configuration Guide for UNIX and Windows
Veritas NetBackup (tm) 5.1 Media Manager Device Configuration Guide for UNIX and Windows
Veritas NetBackup (tm) 6.0 Media Manager Device Configuration Guide for UNIX and Windows

Page: N/A

Modification Type: addition

Modification:
Several things can contribute to the addition of ghost devices in the Windows Device Manager. Some of these include adding or removing hardware, changes to tape drives or tape libraries, failure to use persistent binding, static indexing or hard ALPAs on storage area network (SAN) equipment. These sort of things allow changes to the SCSI device path presentation of a device to the operating system. These "ghost" devices can retain Port, SCSI, Target, and logical unit number (LUN) information that conflict with the active devices being used by Windows. Under these circumstances, hardware instability can result.

Running the device discovery wizard in NetBackup, for example, may list 20 tape drives when only 10 tape drives physically exist. This information is being pulled from the operating system and can incorrectly associate tape device information with the NetBackup hardware configuration. Also, when there are ghost devices, tape device drivers may "disappear" or roll back to the previous drivers after a tape device driver update.

To identify and remove the ghost devices from the Windows Device Manager, do the following:

1. From the command prompt on the problem media server, run:
C:\>set devmgr_show_nonpresent_devices=1
C:\>start devmgmt.msc

2. Then, select View from the drop down and select to Show Hidden Devices.

At this point, any ghost tape devices will be seen with lighter, transparent icon and can be removed. This is done by right-clicking the ghost tape device and selecting "Uninstall". A reboot of the machine should be performed following this action. It may also be necessary to delete and re-add any of these devices being used by NetBackup, as the current NetBackup configuration may have been pointing to one of these ghost devices, rather then the active one. Prior to re-configuring any devices for use with NetBackup, ensure no additional ghost entries have been created following the reboot of the server.



Bob Stump
VERITAS
"Ain't it the truth? Ain't it the truth?" Bert Lahr, Wizard of Oz 1939

 

[Stumpr]

What is Persistent Binding? What are the advantages of Persistent Binding?

http://support.veritas.com/docs/266892

Details:
How does Persistent Binding Work?
A fibre channel target is assigned its World Wide Node Name (WWNN) at loop initialization time; the SCSI target ID for that target is assigned by the device driver when the device is first discovered. It is possible for the WWNN to change between one loop initialization and the next. Every time a system boots or a target is added to or removed from the fibre channel, the loop will be re-initialized. After a system has booted, it will maintain a constant view of the same target ID because the driver software remaps the SCSI target ID to the new WWNN on the fly.
In an environment that utilizes persistent binding, once a system has booted, it will maintain a constant view of the same target ID because the driver software remaps the SCSI target ID to the new WWNN on the fly.

Persistent Binding Advantages:
A second system may use a different SCSI target ID for that target. Thus, an administrator seeking to work with the same target across multiple hosts must be prepared to encounter the situation where the same fibre channel target is known by different SCSI target IDs. Use persistent binding to maintain a consistent target ID on all systems. A common scenario for this type of situation would be a network administrator who needs to consistently map to a tape drive for backup purposes.
Please refer to this Emulex TechNote for additional information:

http://www.emulex.com/ts/fc/docs/winnt4/port/technotes/persbind.htm

.




Bob Stump
VERITAS
"Ain't it the truth? Ain't it the truth?" Bert Lahr, Wizard of Oz 1939

 

[Stumpr]

Best Practice guidelines for using persistent binding in NetBackup Shared Storage Option

http://support.veritas.com/docs/256261

Details:
Without persistent binding, the following problems with tape drives may occur:

Media not ejecting, the wrong media mounting in drives, and tape drives down.

Solution:

This is a narrative description of NetBackup (tm) operation on a Windows 2000 operating system.
The Fibre Host Bus Adaptor (HBA) featured in this TechNote is the Emulex LP8000.

Ensuring proper tape drive configuration in a Storage Area Network (SAN) environment is critical to the proper operation of NetBackup Shared Storage Option (SSO). Often, Storage Area Network (SAN) hardware is configured to use automatic device discovery. This practice works well with disk storage devices, but is not recommended for tape storage sub-systems where persistent hardware device configuration or identification is necessary.

The Windows operating system detects the SAN hardware SCSI devices via a Fibre HBA. This detection phase then stores or maps SCSI drive information in the Windows registry.

The Windows operating system must properly detect and map all SCSI devices in the registry before NetBackup device configuration. In NetBackup device configuration, SCSI device information is gathered from the operating system and placed into the NetBackup device database. The database stores the exact SCSI location of every device detected by the HBA.

When the SAN hardware is configured for automatic detection, any change in the SAN system will trigger the operating system to re-scan the HBA and remap the devices in the registry. In such an event, the new tape device mapping can present a conflict within the NetBackup device database. As a result, NetBackup will no longer communicate with the proper tape device and device errors will occur.

Example: NetBackup may attempt to mount media into a tape device from the previous configuration. NetBackup is configured to mount media into drive 1, but after a hardware re-scan the device is mapped to drive 2. This new configuration, although detected by the operating system, is not automatically updated in NetBackup. This will cause the backup to fail.
A second example is when NetBackup queries the SCSI device and cannot sense the device on the bus. This condition will result in a missed drive signal stating that the "device is not ready" or that it is "in use" which can be interpreted as a hardware problem, and will down the drive.

A third example in this situation is that a media eject may not occur and is presented to the operator as a hardware problem. This condition may interrupt NetBackup processes and ultimately NetBackup will refuse to release the drive for further NetBackup drive operations.

To counteract the effect of SAN Hardware automatic device detection, it is important to ensure that consistent SCSI device path and Logical Unit Number (LUN) mapping information remains consistent even after a reboot.

Note: NetBackup 4.5 also counteracts the effect of SAN Hardware automatic device detection.

Reference:
NetBackup 4.5 Release Notes for UNIX and Windows
page 1: NetBackup 4.5 improves the product's robustness in SAN environments
page 29: Windows NT and Windows 2000 path re-mapping for tape drives
During device manager startup, serialized drives are verified, and if necessary, the paths are corrected in the Media Manager device database.


Below are some examples of HBA configurations when using Emulex LP8000 series cards.

Clearing the Automatically Map SCSI Devices check box will prevent the SCSI device path information from changing. Changes in the SCSI device path information could conflict with other SAN equipment and result in tape devices going down or cause interruption to backup or restore operations.

Clearing the Automatic LUN Mapping check box will allow proper configuration of your mapped SCSI IDs to the appropriate World Wide Name (WWN) of the HBA. Differences in LUN mapping information can also cause problems similar to those caused by inconsistencies in the SCSI device paths listed above.

LP9000 series HBAs share a similar configuration utility. The rules of persistent binding, LUN mapping, and SCSI device path consistencies vary between HBA manufacturers, but the need for consistent hardware mapping information is essential for proper communication.

Additional setup and configuration information of LP8000 HBAs including screen shots can be found at:

http://www.emulex.com/ts/fc/docs/winnt4/port/technotes/persbind.htm

Below is an excerpt from Emulex which illustrates the advantages of persistent binding and usefulness in backups directed to fibre attached tape devices:

"A Fibre Channel target is assigned its WWNN at loop initialization time; the SCSI target ID for that target is assigned by the device driver when the device is first discovered. It is possible for the WWNN to change between one loop initialization and the next. Every time a system boots or a target is added to or removed from the Fibre Channel, the loop will be re-initialized. After a system has booted, it will maintain a constant view of the same target ID because the driver software remaps the SCSI target ID to the new WWNN on the fly.
In an environment that utilizes persistent binding, once a system has booted, it will maintain a constant view of the same target ID because the driver software remaps the SCSI target ID to the new WWNN on the fly.
A second system may use a different SCSI target ID for that target. Thus, an administrator seeking to work with the same target across multiple hosts must be prepared to encounter the situation where the same Fibre Channel target is known by different SCSI target IDs. Use persistent binding to maintain a consistent target ID on all systems. A common scenario for this type of situation would be a network administrator who needs to consistently map to a tape drive for backup purposes. "



Bob Stump
VERITAS
"Ain't it the truth? Ain't it the truth?" Bert Lahr, Wizard of Oz 1939

 

[Stumpr]

Now your probably wondering how someone could get so much information so quickly.

HERE IS THE HOOK UP

http://www.symantec.com/enterprise/support/overview.jsp?pid=15143

simply enter the phrase
persistent binding
....and search to your heart's content



Bob Stump
VERITAS
"Ain't it the truth? Ain't it the truth?" Bert Lahr, Wizard of Oz 1939

 

[billy1]

Bob

Thats fantastic !

I'll have to take some time to get through all of that !

Cheers !