September 1999

How to Deal with Mirrored Disks

HP’s add-on product provides vital disaster recovery, but you’ll need advice on set-up, disk errors and split-volumes

By Andreas Schmidt

Mirrored Disk/iX is an optional subsystem for HP 3000 mission critical systems, and it’s vital to guaranteeing high availability of your company’s data. This article explains the fundamentals of HP’s Mirrored Disk/iX: how to set up volume sets, how to deal with disk errors and how to establish a split-volume backup.

The complete resource on this subject is, of course, the HP manual for Mirrored Disk/iX (User’s Guide, HP Part No. 30349-90003.) Some of what follows is based on this. But we all know that a summary sometimes will help better than reading through a complete HP manual — especially if you are under pressure in a delicate situation.

What are mirrored disks?

Mirrored Disk/iX is a subsystem for HP 3000s which needs to be ordered separately. The installation follows the normal subsystem Installation Process as documented in the Installation Manual. Mirrored Disk/iX is designed to work only with non-system volumes. To make it very clear: Mirrored Disk/iX does not support mirroring the HP 3000’s system volumes.

It supports disk drives that use HP-FL cards or NIO SCSI cards. But mirrored partners must be the same model of fiber-link drive or NIO SCSI drive, and mirrored partners must be connected to different HP-FL cards or NIO SCSI. Otherwise a single point of failure would still exist.

Mirrored disks are designed to provide high data availability by automatically maintaining identical information on two partner disks. When an application writes to a disk, disk mirroring causes the information to be written to both drive partners. When an application reads from a disk, there are two places to access the requested data. This may give performance benefits on large systems which do a lot of reads for queries but only a few writes to the same data. Applications running on the system are unaware that disk mirroring is present.

Once disk mirrors have been established using the VOLUTIL utility, a mirrored disk acts just like any other disk connected to the system, until a disk failure occurs. If either disk of any pair fails, normal system operation continues. When the partner is ready to resume operation, the system copies data from the good disk, bringing the pair to a consistent state, and normal mirroring resumes.

Once mirrored disks have been installed, you can use them like any other disks connected to the system. Additionally, you can perform split-volume backup of mirrored disk data while still accessing the data.

So, Mirrored Disk/iX supports the following features:

The installation of Mirrored Disk/iX is easy:

The subsystem installation of Mirrored Disk/iX enhances the HP 3000’s VOLUTIL commands. HP provides both commands VOLUTIL and MIRVUTIL to make life easier for the System Manager. The functionality is the same when Mirrored Disk/iX has been installed.

Please be careful: The Create Volumes (CV) capability is required to use VOLUTIL to initialize mirrored volumes. You also need it to input system commands from the system console to perform split-volume backups.

How to set up volume sets

Assuming that the subsystem has been installed and the hardware as been plugged in and is configured, the new volumes will be in state SCRATCH or UNKNOWN. Verify this via :DTSTAT ALL

LDEV-TYPE
30-079370
31-079370
32-079370
33-079370		 STATUS
SCRATCH
SCRATCH
SCRATCH
SCRATCH		 VOLUME (VOLUME SET - GEN)

Now invoke :VOLUTIL or :MIRVUTIL and create the new set’s master disk as mirrored pair:

volutil: NEWMIRRSET PROD_SET:MEMBER1 (30,31)

and verify via DSTAT:

volutil: :DSTAT ALL
LDEV-TYPE
30-079370
31-079370
32-079370
33-079370		 STATUS
MASTER-MD
MASTER-MD
SCRATCH
SCRATCH		 VOLUME (VOLUME SET - GEN)
MEMBER1 (PROD_SET-0)
MEMBER1 (PROD_SET-0)
 
 

Now add volumes in this mirrored set. These volumes must be in state SCRATCH or UNKNOWN.

volutil: NEWMIRRVOL PROD_SET:MEMBER2 (32,33)

and check via DSTAT again:

volutil: :DSTAT ALL

LDEV-TYPE
30-079370
31-079370
32-079370
33-079370		 STATUS
MASTER-MD
MASTER-MD
MEMBER-MD
MEMBER-MD		 VOLUME (VOLUME SET - GEN)
MEMBER1 (PROD_SET-0)
MEMBER1 (PROD_SET-0)
MEMBER2 (PROD_SET-0)
MEMBER2 (PROD_SET-0)

In VOLUTIL, the command SHOWSET with the MIRROR option (this option has been installed via the added subsystem) will show the state of the mirrored set:

volutil: SHOWSET PROD_SET MIRROR

Volume Name
MEMBER1
MEMBER1
MEMBER2
MEMBER2		 Vol Status
MASTER
MASTER
MEMBER
MEMBER		 Mirr Status
NORMAL
NORMAL
NORMAL
NORMAL		 Ldev
30
31
32
33		 Mirr Ldev
31
30
33
32
How to deal with Disk Errors

There are two types of disk errors: Disk errors after mount (in normal operation) and a disk cannot be mounted (already defective in booting the box)

Disk Error after Mount:
If a disk has a problem after the mount, the system will continue to work automatically with only one disk of the affected pair.

Possibilities for Disk Errors are:

Disk reports ERRORS — Disk will immediately become DISABLED and the System will continue to work without a mirroring for the affected volume set without any interruption.

Disk does not answer any longer — The system waits about two minutes for an answer by the disk. During this period, all I/O processes are suspended. If the disk will answer in this interval, the system will continue to work with mirroring for this pair. If the disk will not answer the disk will become DISABLED. The system will continue without mirroring for the affected volume.

Here’s an example: During normal operation LDEV 32 fails. The following message will appear on the Console:

?09:09/12/MIRRORED VOLUME DISABLED ON LDEV#32
?09:09/22/ACKNOWLEDGE MIRRORED VOLUME DISABLED ON LDEV#32 (Y/N)?

:REPLY 22 ,Y

The system will continue to run. The problem may be that this message and the reply will be overlooked on big systems. It’s recommended to have a monitor in place. HP’s OpenView Operations Center (a.k.a. IT/O OpC) may be one option.

A check using DSTAT and VOLUTIL will show the following:

:DSTAT ALL

LDEV-TYPE
30-079370
31-079370
32-079370
33-079370		 STATUS
MASTER-MD
MASTER-MD
*DISABLED-MD
MEMBER-MD		 VOLUME (VOLUME SET - GEN)
MEMBER1 (PROD_SET-0)
MEMBER1 (PROD_SET-0)
MEMBER2 (PROD_SET-0)
MEMBER2 (PROD_SET-0)

volutil: SHOWSET PROD_SET MIRROR

Volume Name
MEMBER1
MEMBER1
MEMBER2
MEMBER2		 Vol Status
MASTER
MASTER
MEMBER
MEMBER		 Mirr Status
NORMAL
NORMAL
DISABLED
NON-MIRROR		 Ldev
30
31
32
33		 Mirr Ldev
31
30
33
32

Having repaired or exchanged the disk, you must continue by issuing the command: volutil: REPLACEMIRRVOL PROD_SET:MEMBER2 32 and re-establish the mirroring. Check it via

volutil: SHOWSET PROD_SET MIRROR
Volume Name
MEMBER1
MEMBER1
MEMBER2
MEMBER2		 Vol Status
MASTER
MASTER
MEMBER
MEMBER		 Mirr Status
NORMAL
NORMAL
REPAIR-DEST
REPAIR-SRCE		 Ldev
30
31
32
33		 Mirr Ldev
31
30
33
32

The repair will happen automatically. No further intervention will be needed. This process is fully transparent for applications and users of the data on the affected volume set.

Disk Error before Mount:
Here’s another example: LDEV 33 cannot be mounted during the system’s startup. Following message will appear on the console:

?09:09/12/MIRRORED PARTNER MISSING FOR LDEV# 32

The system sets the “good” mirrored disk LDEV 32 on PENDING and waits for SUSPENDMIRRVOL to allow LDEV 32 to work without the mirrored partner:

?09:09/22/ACKNOWLEDGE MIRRORRED PARTNER MISSING FOR LDEV# 32(Y/N)?

:REPLY 22 ,Y

This volume (here: MEMBER2) will stay unavailable until either the mirrored pair (here: LDEV 33) will become available again, or the mirror will become suspended via SUSPENDMIRRVOL command. DSTAT will show the following:

:DSTAT ALL

LDEV-TYPE
30-079370
31-079370
32-079370		 STATUS
MASTER-MD
MASTER-MD
*PENDING-MD		 VOLUME (VOLUME SET - GEN)
MEMBER1 (PROD_SET-0)
MEMBER1 (PROD_SET-0)
MEMBER2 (PROD_SET-0)

By using SUSPENDMIRRVOL the mirror will become suspended, and the disk will work without a mirror for it. Now, this disk should not become faulty — it’s a single point of failure now! SUSPENDMIRRVOL will work only for disks in state PENDING!

volutil: SUSPENDMIRRVOL PROD_SET:MEMBER2 32

MEMBER2 is now available again, but without a mirror:

volutil: SHOWSET PROD_SET MIRROR

Volume Name
MEMBER1
MEMBER1
MEMBER2		 Vol Status
MASTER
MASTER
MEMBER		 Mirr Status
NORMAL
NORMAL
SUSPEND-MIRR		 Ldev
30
31
32		 Mirr Ldev
31
30
33

Having repaired the disk (LDEV 33), the mirror must become active again:

:DSTAT ALL
LDEV-TYPE
30-079370
31-079370
32-079370
33-079370		 STATUS
MASTER-MD
MASTER-MD
*PENDING-MD
SCRATCH		 VOLUME (VOLUME SET - GEN)
MEMBER1 (PROD_SET-0)
MEMBER1 (PROD_SET-0)
MEMBER2 (PROD_SET-0)
 


By using REPLACEMIRRVOL, the repaired LDEV 33 will be initialized as mirror for LDEV 32 :

volutil: REPLACEMIRRVOL PROD_SET:MEMBER2 33

You can verify this with SHOWSET

volutil: SHOWSET PROD_SET MIRROR

Volume Name
MEMBER1
MEMBER1
MEMBER2
MEMBER2		 Vol Status
MASTER
MASTER
MEMBER
MEMBER		 Mirr Status
NORMAL
NORMAL
REPAIR-SRCE
REPAIR-DEST		 Ldev
30
31
32
33		 Mirr Ldev
31
30
33
32


Split-Volume Backup

Another feature of Mirrored Disk/iX is the way it can make backups. It is only necessary to take (physically spoken) one disk’s content out of a mirrored pair. This is possible via the split-volume backup. To split a volume set, no user is allowed to stay logged on for this volume set: You can use the command :TELL @ LOGOFF FOR BACKUP. The volume set will become unavailable now for one half:

:VSCLOSE PROD_SET; SPLIT
:DSTAT ALL

LDEV-TYPE
30-079370
31-079370
32-079370
33-079370		 STATUS
LONER-SU
LONER-SB
LONER-SU
LONER-SB		 VOLUME (VOLUME SET - GEN)
MEMBER1 (PROD_SET-0)
MEMBER1 (PROD_SET-0)
MEMBER2 (PROD_SET-0)
MEMBER2 (PROD_SET-0)


Now make via VSOPEN both SETS (USER and BACKUP) available: :VSOPEN PROD_SET

PROD_SET SPLITT USER VOLUME MOUNTED ON LDEV 32 (AVR23)
PROD_SET SPLITT BACKUP VOLUME MOUNTED ON LDEV 33 (AVR24)

Now the PROD_SET is available for production but without the mirroring:

:TELL @ SYSTEM IS AVAILABLE NOW
:DSTAT ALL

LDEV-TYPE
30-079370
31-079370
32-079370
33-079370		 STATUS
MASTER-SU
MASTER-SB
MEMBER-SU
MEMBER-SB		 VOLUME (VOLUME SET - GEN)
MEMBER1 (PROD_SET-0)
MEMBER1 (PROD_SET-0)
MEMBER2 (PROD_SET-0)
MEMBER2 (PROD_SET-0)

Start the backup with the command: :FILE T;DEV=TAPE :STORE /; *T; SPLITVS=PROD_SET; SHOW. This backup is compatible to the “normal” STORE.

Having finished the backup, the split can be canceled:

volutil: JOINMIRRSET PROD_SET SOURCE=USER

SOURCE=USER tells the system that during the JOIN and the following REPAIR to synchronize the BACKUP with the USER split, so the on-line users are allowed to continue to work. The status in VOLUTIL during the synchronization will look like:

volutil: SHOWSET PROD_SET MIRROR

Volume Name
MEMBER1
MEMBER1
MEMBER2
MEMBER2		 Vol Status
MASTER
MASTER
MEMBER
MEMBER		 Mirr Status
REPAIR-SRCE
REPAIR-DEST
REPAIR-SRCE
REPAIR-DEST		 Ldev
30
31
32
33		 Mirr Ldev
31
30
33
32

At most, six mirrored pairs will become synchronized in the same period for performance reasons. That means as soon as one of the six pairs is finished the next waiting pair will be processed.

This is an optional way to make a backup — as I said earlier, Mirrored Disk/iX is fully transparent to all applications. We’re still using TurboStore/iX online to back up our mirrored volume sets, and didn’t encounter any problem because of Mirrored Disk/iX.

Summary

Mirrored Disk/iX is a MUST for mission critical systems to guarantee high availability of the data. In case of problems because of the physical disks, the data stays available. The automatic repair processes in Mirrored Disk/iX are transparent to the users. The procedures are quite easy — but you must know them! This article will help all HP 3000 System Managers to have this handy. Comments and questions are welcomed; just send me e-mail at aschmid4@csc.com.

Andreas Schmidt is a computer technology specialist working for CSC Ploenzke AG, Germany.

Copyright The 3000 NewsWire. All rights reserved.