cms backup tape (unix commands) 2

[earnshaw]

Hello all,

I have noticed a couple of times that the cms does not backup (timetable backups full and inc). I know that the timetable is set correctly and that all is fine. I think that the reason may be that the tapes are out of sync. Is there a way of looking at the tape header (name etc) and formating the tape and starting again. If so what are the unix commands!

The cmsadm backup is not a problem, just done one now. I can perform a full backup, but sometimes it does not like the tapes.

many thanks

Jason

 

[earnshaw]

don't worry about it found this.....

Table of Contents

I. Introduction ----------------------------------------------------- 1

II. Backup and Restore Components ---------------------------- 2

1. Hardware and Tape Device -------------------------------- 2

2. Software -------------------------------------------------------- 2

3. CMSADM Backup and Restore Procedures ------------ 3

A. CMSADM Backup -------------------------------------- 3

B. CMSADM Restore for Solaris and CMS ----------- 3

C. CMSADM Restore for CMS application ----------- 4

4. Maintenance Backup and Restore Procedures -------- 4

A. Full Maintenance Backup ----------------------------- 4

B. Incremental Maintenance Backup ------------------- 5

C. Full Maintenance Restore ----------------------------- 6

D. Incremental Maintenance Restore ------------------- 7

III. Tape Check Commands --------------------------------------- 9

IV. Summary -------------------------------------------------------- 11

V. Customer Case Studies --------------------------------------- 12




Backup and Crash Recovery for CMS Systems


WHEN YOUR SYSTEM CRASHES WITH A FATAL HARD DISK ERROR, DO YOU HAVE THE MATERIALS ON HAND TO RECOVER AS MUCH OF YOUR DATA AS POSSIBLE AS WELL AS GET BACK ON-LINE QUICKLY?

For too often we in CMS Customer Support are finding that the answer to this question is a resounding "No!" . To help ensure the integrity of your backups, we are making this document available on-line to our R3 CMS customers. It provides supplemental information extracted from the CMS Administration and the CMS Installation and Maintenance manuals relative to performing backups and restores on the CMS system.

Remember that the material in this document is supplemental to the CMS manuals. To get a full scope of backup and restore process please read the pertinent sections in CMS documentation manuals.

This document is organized into sections relative to CMS release and each section describes important information to ensure integrity in the backup and recovery operations. We request that your CMS administration staff read the CMS Administration and CMS Installation and Maintenance manuals.

The tone of this document is largely informal. This is the, CMS Customer Support group, talking to you, our customers, about how we can work together to make sure that your CMS system would experience minimum down time after a system crash. We are aware that in many cases this is your whole operation at stake, and there is nothing more painful or frustrating to us than knowing that if only the backups were done, if only the software was at hand and well organized, we could have had your CMS on-line in hours instead of days.

After reading this document, please let us know if the information is useful to your operation. In addition, please let us know if you would like to see changes in this document regarding your recommendations, errors detected or general comments. Call our CMS Helpline at 1-800-242-2121 to give us your feedback. Your request will be forward to the appropriate department for possible updates.

I. Introduction

This document is a quick reference guide to provide backup and recovery techniques for CMS customers. Information is provided to indicate required resources necessary for a CMS user to maintain quality backups to be prepared for an unexpected recovery. We will explain how to maintain backups on a routine schedule and how to recover your CMS system from an inadvertent disk failure.

Maintaining quality backups is economical in nature and good common business sense for your call center operation. When system backups are unavailable or without data integrity then it will cost your call center in system down time, data loss and unnecessary financial expense to restore your CMS. On the other hand, if you maintained good quality backups then your CMS system could be on-line within hours instead of days. Backups are extremely important and necessary in order to maintain a quality and cost effective CMS system.

The backup operation is simple, easy and an excellent investment to safeguard the sanity of your CMS system and the restore operation provides the return on investment when you maintain quality stable backups.

Your CMS system administrators should monitor the CMS tape device unit(s) and software to make sure that the backup process is working properly. A check of the actual backup tapes should be performed to verify data integrity. You want a stable quality backup to transition a down CMS to the on-line state within a minimum time period.

Maintaining quality backups is mandatory to maintain a stable CMS.

It is the responsibility of the customer to "safe-guard" all original factory shipped software and documentation in a SAFE accessible location. In addition, it is highly recommended to never leave a "good backup tape" in the tape device overnight, because tape rotation is essential in maintaining a quality set of backup tapes for system recovery.


II. Backup and Restore Components


Your CMS system is shipped with functioning hardware and software components to allow you to backup and restore the system. The CMS systems have backup and restore software to perform a CMSADM and a Maintenance backup/restore operations for all CMS software releases. The hardware tape device components are specific to the hardware platform.

1. Hardware and Tape Device

The CMS system is configured with specific tape devices according to the CMS hardware platform. The following tape devices are configured with the indicate hardware platforms:

Tape Device Description Hardware Platform

DC6320 320 MB - QIC Sparc5, Sparc10

2.5G QIC 2.5 GB - QIC Sparc5

160mXL 14 GB, 8mm tape Sparc20, UE3000

SLR5 SLR5, 4/8GB QIC tape Sparc5, Ultra5

170m AME Mammoth 20/40GB UE3500

4mm DDS-150 DDS4 20/40GB tape Sparc5, Ultra5

M 2.5GB Magnus 2.5GB tape Sparc5

The customer should have on hand tape-cleaning cartridges and clean the tape device routinely. Normally a tape device should be cleaned monthly but this depends on device usage. The backup process will fail if the tape device is not clean.


2. Software

The CMS system contains backup and restore software specific for a CMSADM and Maintenance operation. This document supports CMS software release R3V6, R3V8 and R3V9. The CMSADM backup and restore software will backup and restore the Solaris Operating System, CMS applications and configuration data. The CMSADM backup will backup all data on the CMS system, excluding the Informix database.

3. CMSADM Backup and Restore Procedures

The CMSADM operation is executed at the UNIX command prompt to start a backup or restore process. It is recommended to perform a CMSADM backup every 30 days and 3-4 tapes should be rotated. To perform a CMSADM backup or restore process the user must perform the following procedures:

A. CMSADM Backup

Log onto the CMS server as root.

Insert a write enable tape into tape device.

At the "#" prompt, execute "cmsadm" command.

Select the "backup" option for backup.

Respond to the prompt from the backup operation.

After receiving backup successful message remove tape and label.

Exit from UNIX and logoff system, your backup is successful.

Store your valuable backup tape(s) in a safe location.


There are several conditions which would require a CMSADM restore operation. One condition is damage to the boot disk device on the CMS server, another condition is damage to the "cms" file system or damage to a non-boot disk device. Under each condition a restore is required for the Solaris operating system and cms file system or for the cms file system only. Reference the Software Installation and Setup Manual sections "Performing a CMSADM restore" for detail instructions.

B. CMSADM Restore for Solaris and CMS application (V9 mirrored N/A)

Replace the damage boot disk device, if necessary.

Boot from CDROM and perform Solaris Operating System installation.

Perform the "cpio" command to restore system configuration data.

Setup disk configuration based on mirrored or non-mirrored system.

Perform the "cpio" command to restore data to the cms file system.

Reboot the system in multi-user mode via "init 6" command.

Run CMS setup via cmssvc or setup the "cms.install" flat file.

Perform the CMS setup function via "cmssvc" command.

Perform a maintenance restore using most recent full backup.

Transition system via "/usr/sbin/shutdown -y -g0 -i0" command.

The CMS system should boot with "boot -r" command at OK prompt.

The CMS will boot and all acd links transitioned to up state.


C. CMSADM Restore for CMS application (V9 mirrored N/A)

1. Replace the damage non-boot disk device if necessary.

2. Boot the system in single-user mode from boot disk.

3. Setup disk partitioning based on mirrored or non-mirrored system.

4. Setup disk configuration based on mirrored or non-mirrored system.

5. Perform the "cpio" command to restore data to the cms file system.

6. Reboot the system in multi-user mode via "init 6" command.

7. Perform a maintenance restore using most recent full backup.

8. Reboot system via "/usr/sbin/shutdown -y -g0 -i6" command.

9. The CMS system will boot and acd links transitioned to up state.

Please reference your CMS Administration Manual for detail description of the CMSADM backup and restore procedures.


4. Maintenance Backup and Restore Procedures

The CMS Maintenance backup and restore software will backup and restore the Informix database. The maintenance backup provides an option for a full or incremental backup and these operations are executed from the Backup Up Data operation via the CMS Maintenance menu window. In addition, there are timetable entries for maintenance backups execution. They need to be activated based on the customer's scheduled request. It is recommended to perform a full maintenance backup for every day of business operations and rotate 7-8 tapes. An incremental backup is a subset of a full maintenance backup. A full maintenance backup must be done before an incremental backup. To perform a maintenance backup and restore process the user must perform the following procedures:

A. Full Maintenance Backup

1. The user must log onto the CMS and execute the CMS application.

2. Insert a write enable tape into tape device.

3. Select the Maintenance option to display sub-screen.

4. Select the Back Up Data option to display backup screen.

5. Enable the following options:

Device name: default

Verify tape can be read after backup? (y,n): y

ACD(s) to back up (Select one):

All ACDs < > Current ACD

Data to back up (Select any you wish):

[x] Local system administration data

[x] CMS system administration data

[x] ACD-specific administration data

[x] Historical data,

Select one:

Full < > Incremental

[x] Non-CMS data

[ ] Specific tables

6. Hit the &quot;ENTER&quot; key and select the RUN option.

7. The CMS application should start the full maintenance backup.

8. After successful completion remove the tape and label properly.

9. Exit from the CMS menu and logoff system.

10. Store your valuable backup tape(s) in a safe location.

Rule of Thumb: If all of your Maintenance backup fits one tape then, Incremental backups are NOT NECESSARY.

B. Incremental Maintenance Backup

1. The user must log onto CMS and execute the CMS application.

2. Insert a write enable tape into tape device.

3. Select the Maintenance option to display sub-screen.

4. Select the Back Up Data option to display backup screen.

5. Enable the following options:

Device name: default

Verify tape can be read after backup? (y,n): y

ACD(s) to back up (Select one):

All ACDs < > Current ACD

Data to back up (Select any you wish):

[x] Local system administration data

[x] CMS system administration data

[x] ACD-specific administration data

[x] Historical data,

Select one:

< > Full Incremental

[x] Non-CMS data

[ ] Specific tables

6. Hit the &quot;ENTER&quot; key and select the RUN option.

7. The CMS application will start the incremental maintenance backup.

8. After successful completion remove the tape and label properly.

9. Exit the CMS menu and logoff system.

10. Store your valuable backup tape(s) in a safe location.


C. Full Maintenance Restore

1. The user must log onto the CMS server and execute the CMS application.

2. Insert the most recent valid full maintenance backup into tape device.

3. Select the Maintenance option to display sub-screen.

4. Select the Restore Data option to display backup screen.

5. Enable the following options:

Status:

Errors:

Device name: default

Restore from last backup (y,n): n

Restore historical data from:

Start date:
Start time:
Stop date:
Stop time:
ACD(s) to restore (select one): All ACDs < > Current ACD

Data to restore (Select any you wish):

[x] Local system administration data
[x] CMS system administration data
[x] ACD-specific administration data
[x] Historical data
[x] Non-CMS data
[x ] Specific tables
6. Hit the &quot;ENTER&quot; key and select the RUN option.

7. The CMS application should start the full maintenance restore.

8. After successful completion remove the tape from device.

9. Exit from the CMS menu and log off system.

10. Log onto CMS server as root and reboot system.

11. The CMS server should boot with restored data.

12. Return your valuable backup tape(s) to a safe location.

D. Incremental Maintenance Restore

1. A full restore must be done prior to incremental restore.

2. After full maintenance restore insert 1st incremental backup tape.

3. Insert next incremental backup tape in order of backup.

4. Select the Maintenance option to display sub-screen.

5. Select the Restore Data option to display backup screen.

6. Enable the following options:

Status:

Errors:

Device name: default

Restore from last backup (y,n): n

Restore historical data from:

Start date:
Start time:
Stop date:
Stop time:
ACD(s) to restore (select one): All ACDs < > Current ACD

Data to restore (Select any you wish):

[x] Local system administration data
[x] CMS system administration data
[x] ACD-specific administration data
[x] Historical data
[x] Non-CMS data
[x] Specific tables
7. Hit the &quot;ENTER&quot; key and select the RUN option.

8. The CMS application should start the incremental maintenance restore.

9. After successful completion remove the tape from device.

10. Exit from the CMS menu and log off system.

11. Log onto CMS server as root and reboot system.

12. The CMS server should boot with restored data.

13. Return your valuable backup tape(s) to a safe location.


III. Tape Check Commands

The CMS system has several commands which will allow the user to check and verify the tape status and data integrity of backup tapes. The user can execute the following commands at the UNIX &quot;#&quot; prompt to perform check procedures:

1. Tape Status

The &quot;mt&quot; command will report the current status of a tape in the tape device.

Example 1: # mt -f /dev/rmt/0 status

Response 1: /dev/rmt/0: no tape loaded or drive offline

Note 1: No tape is loaded into device &quot;/dev/rmt/0&quot;



Example 2: # mt -f /dev/rmt/0 status

Response 2: /dev/rmt/0: no tape loaded or drive offline

Note 2: Tandberg 8 Gig QIC tape drive:

sense key(0x0)= No Additional Sense residual= 0 retries= 0

file no= 0 block no= 0

2. Tape Check

The &quot;br_check&quot; command will verify backup quality of a CMS maintenance backup tape.

Example : # /cms/bin/br_check

Response: Enter device type ['q' for qtape, 'f' for floppy]: q

Enter device path: /dev/rmt/0

Opening the volume...

Label: CMS-010713-01-LSAC-00-F-01

CMS Database Version: r3v8ai.b.0

Machine name: r3gpbxx

ACD(s) backed up on the volume:

ACD Switch EAS Vectoring

1 R8 n y

Enter 'l' to list the tables or 'v' to also verify the volume: v

htrunk Full 1 12/31/1899(0) 07/13/2001(230)

hvdn (V) Full 1 12/31/1899(0) 07/13/2001(300)

hvector (V) Full 1 12/31/1899(0) 07/13/2001(300)

m_secs Full 1 12/31/1899 06/30/2001

magent Full 1 12/31/1899 06/30/2001

mcwc Full 1 12/31/1899 06/30/2001

msplit Full 1 12/31/1899 06/30/2001

mtkgrp Full 1 12/31/1899 06/30/2001

mtrunk Full 1 12/31/1899 06/30/2001

mvdn (V) Full 1 12/31/1899 06/30/2001

mvector (V) Full 1 12/31/1899 06/30/2001

w_secs Full 1 12/31/1899 07/07/2001

wagent Full 1 12/31/1899 07/07/2001

wcwc Full 1 12/31/1899 07/07/2001

wsplit Full 1 12/31/1899 07/07/2001

wtkgrp Full 1 12/31/1899 07/07/2001

wtrunk Full 1 12/31/1899 07/07/2001

wvdn (V) Full 1 12/31/1899 07/07/2001

wvector (V) Full 1 12/31/1899 07/07/2001

End of Backup.

Terminating...

The &quot;cpio&quot; command will list a table of contents of a CMSADM backup tape. This list contains hundreds of files such as the response below.

Example : # cpio -icvt -C 10240 -I /dev/rmt/0

Response: drwxr-xr-x 3 root other 0 Mar 14 17:58 2001, cms/cc

drwxr-xr-x 3 root other 0 Mar 14 17:58 2001, cms/cc/c

drwxr-xr-x 6 root root 0 Mar 14 16:11 2001, cms/cow

drwxr-xr-x 2 root root 0 Mar 14 16:13 2001, cms/cow/l


IV. Summary

Performing routine backup of your CMS system is extremely important and cost effective to say the least. The CMS software and hardware platform provide the necessary resources to generate quality backups of your CMS system and if you the user generate these backups on a routine schedule then whenever your system experience a hard disk crash then you will have the necessary resources to minimize down time.

The backups are critical and without them the restore process is not possible. But, when you perform quality backups then the restore process is the insurance to maintain a stable CMS system.

The customer should maintain a crash recovery kit. The crash recovery kit should have the following items:

1. All of the software and passwords shipped with the CMS system.

2. The current CMS software load tape.

3. The CMS cmsadm backups.

4. The CMS full maintenance backups.

5. The CMS Incremental maintenance backups.

V. Customer Case Studies

Here are several examples of the many customers who have found themselves unable to recover from a disk crash in a timely manner.

Customer A was running CMS Full Maintenance backups every night. A combination of three unfortunate circumstances conspired to make their recovery from a hard disk crash take two days to recover enough data to be operational, and nearly two weeks to recover all their data:

1. They re-used the same two tapes on alternate days for over a year. This exceeded the tape media usage by considerable margin. The over usage of tape media can cause data corruption and loss of data. The CMS Support group had to apply extraordinary efforts to recover most of the data.

2. The customer was doing the right thing by performing a CMS Full Maintenance backup on a nightly interval. Every customer should perform a backup nightly. Unfortunately, over usage of two caused data corruption and the data was not retrievable by standard means.

3. Software needed to recover from catastrophic failure was not available at the customer site and took considerable time to obtain. Also, a vital password needed to activate critical software was not available. A considerable amount of time was spent on retrieving password data.


Customer B also had a system disk crash and needed to recover quickly. Their hardware was quickly replaced and the system became operational, but recovering the software took a considerable amount of time, and the lingering effects of trying to take shortcuts were only resolved a month later. What happened to make the recovery so difficult? Here are some of the items that contributed:

1. The total system backup, or cmsadm backup was not available. The cmsadm backup that was shipped with the system was unavailable.

2. Via a timetable, the full maintenance backup was scheduled to perform on a regular basis. This is good practice. Unfortunately, the same tape was repeatedly used for over four months. A database table on disk became corrupt and was subsequently backed-up to the tape. After the corrupt table was written to tape the backup became corrupt and essentially unusable.

3. When it came time to restore the customer's system disk, the customer's copy of necessary OS and application software packages were unavailable to restore the CMS system.

Important Issues:

1. An adequate number of backup tapes should be rotated on a timely schedule.

2. The customer should maintain some knowledge of tape usage.

3. Periodically set aside copies of backup tapes, especially when major changes are applied to the CMS system.

4. Keep all software, documentation and system configuration data in a safe location.

5. Reference your CMS documentation to maintain your CMS system.