Wednesday, August 6, 2014

Restore A Missing Physical Volume


Let's find out the step-by-step involved to recover a failed physical volume in Linux.

The setup has got the"testvg" which got "testlv" and corresponding Physical Volume (PV) is on '/dev/sdb' block device as shown below:

[root@RHEL4 archive]# vgs
  VG     #PV #LV #SN Attr   VSize VFree
  VolGroup00   1 2 0 wz--n-  19.88G 32.00M
  testvg   1   1 0 wz--n- 200.00M 0              ← targeted VG

[root@RHEL4 archive]# pvs
  PV     VG     Fmt  Attr PSize   PFree
  /dev/sda2  VolGroup00 lvm2 a- 19.88G 32.00M
  /dev/sdb   testvg lvm2 a-   200.00M 0    ← targeted PV

[root@RHEL4 archive]# lvs
  LV   VG     Attr   LSize Origin Snap%  Move Log Copy%
  LogVol00 VolGroup00 -wi-ao  18.88G
  LogVol01 VolGroup00 -wi-ao 992.00M
  testlv   testvg -wi-a- 200.00M                           ← targeted LV


..... Destroyed the /dev/sdb master table using the below command

[root@RHEL4 ~]# dd if=/dev/zero of=/dev/sdb bs=512 count=2
2+0 records in
2+0 records out

..... Later, when I tried to check for the current PV, LV and VG, it was not found.

[root@RHEL4 ~]# vgmknodes

[root@RHEL4 ~]# vgmknodes -v
Finding all logical volumes

[root@RHEL4 ~]# lvs
  LV   VG     Attr   LSize Origin Snap%  Move Log Copy%
  LogVol00 VolGroup00 -wi-ao  18.88G
  LogVol01 VolGroup00 -wi-ao 992.00M

[root@RHEL4 ~]# vgs
  VG     #PV #LV #SN Attr   VSize VFree
  VolGroup00   1 2 0 wz--n- 19.88G 32.00M

[root@RHEL4 ~]# lvs
  LV   VG     Attr   LSize Origin Snap%  Move Log Copy%
  LogVol00 VolGroup00 -wi-ao  18.88G
  LogVol01 VolGroup00 -wi-ao 992.00M

[root@RHEL4 ~]# pvs
  PV     VG     Fmt  Attr PSize  PFree
  /dev/sda2  VolGroup00 lvm2 a-   19.88G 32.00M

..... Unable to activate this failed VG

[root@RHEL4 ~]# vgchange -ay testvg
  Volume group "testvg" not found

..... Even running e2fsck failed on such device

[root@RHEL4 ~]# !e2fsck
e2fsck -f -y /dev/testvg/testlv
e2fsck 1.35 (28-Feb-2004)
e2fsck: No such file or directory while trying to open /dev/testvg/testlv

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>

...... Found that the entry to the corresponding VG is still there in the /etc/lvm/archive folder for restoring

[root@RHEL4 ~]# cd /etc/lvm/archive/
[root@RHEL4 archive]# ls
testvg_00000.vg  testvg_00002.vg VolGroup00_00000.vg
testvg_00001.vg  testvg_00003.vg
[root@RHEL4 archive]# cat testvg_00003.vg
# Generated by LVM2: Tue Oct  1 15:06:48 2013

contents = "Text Format Volume Group"
version = 1

description = "Created *before* executing 'vgs'"

creation_host = "RHEL4.4"   # Linux RHEL4.4 2.6.9-42.EL #1 Wed Jul 12 23:15:20 EDT 2006 x86_64
creation_time = 1380654408  # Tue Oct  1 15:06:48 2013

testvg {
     id = "x1TBig-s0D6-7Njv-U2oR-bolr-VZaP-o4Mkw5"
     seqno = 3
     status = ["RESIZEABLE", "READ", "WRITE"]
     extent_size = 8192          # 4 Megabytes
     max_lv = 0
     max_pv = 0

     physical_volumes {

             pv0 {
                     id = "4X3wIH-uuCy-F99r-H2mt-zklh-5ZCR-64lTqQ"
                        device = "/dev/sdb" # Hint only

                     status = ["ALLOCATABLE"]
                     pe_start = 384
                     pe_count = 50   # 200 Megabytes
             }
     }

     logical_volumes {

             testlv {
                     id = "GsaJe0-QCoO-DyeP-mqB3-UaPM-12XQ-4CeVvE"
                     status = ["READ", "WRITE", "VISIBLE"]
                     segment_count = 1

                     segment1 {
                             start_extent = 0
                             extent_count = 50   # 200 Megabytes

                             type = "striped"
                             stripe_count = 1    # linear

                             stripes = [
                                     "pv0", 0
                             ]
                     }
             }
     }
}

..... Trying to activate in-complete VGs

[root@RHEL4 archive]# vgchange -ay testvg --partial
  Partial mode. Incomplete volume groups will be activated read-only.
  Volume group "testvg" not found

..... At this stage, we would need to re-create the missing PV using device reference and UUID details from archive list as shown below.

[root@RHEL4 archive]# pvcreate --uuid "KWfeHu-cP6u-Qpdz-XzDD-iomY-uSKd-2d6V6G" --restorefile /etc/lvm/archive/testvg_00003.vg /dev/sdb
  Couldn't find device with uuid '4X3wIH-uuCy-F99r-H2mt-zklh-5ZCR-64lTqQ'.
  Physical volume "/dev/sdb" successfully created

..... Now, ran vgcfgrestore on the corresponding VG

[root@RHEL4 archive]# vgcfgrestore testvg
  Restored volume group testvg

.... Next, activate the volume group

[root@RHEL4 archive]# vgchange -ay testvg
  1 logical volume(s) in volume group "testvg" now active


..... Check if respective PV, VG and LV are present.

[root@RHEL4 archive]# vgs
  VG     #PV #LV #SN Attr   VSize VFree
  VolGroup00   1 2 0 wz--n-  19.88G 32.00M
  testvg   1   1 0 wz--n- 200.00M 0                  ← here is the restored VG

[root@RHEL4 archive]# pvs
  PV     VG     Fmt  Attr PSize   PFree
  /dev/sda2  VolGroup00 lvm2 a- 19.88G 32.00M
  /dev/sdb   testvg lvm2 a-   200.00M 0      ← restored PV

[root@RHEL4 archive]# lvs
  LV   VG     Attr   LSize Origin Snap%  Move Log Copy%
  LogVol00 VolGroup00 -wi-ao  18.88G
  LogVol01 VolGroup00 -wi-ao 992.00M
  testlv   testvg -wi-a- 200.00M                          ← this is the restored LV


..... Now, the LV can be mounted for further use .

[root@RHEL4 archive]# mount -a

[root@RHEL4 archive]# df
Filesystem       1K-blocks  Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                   19481284   2183572 16308120  12% /
/dev/sda1           101086 10275 85592  11% /boot
none                254384     0 254384   0% /dev/shm
/dev/mapper/testvg-testlv
                     198337 10048 178049   6% /test
[root@RHEL4 archive]#

PV has been restored successfully!!! 

No comments: