Since quite some time, I’m running LVM nearly everywhere – also on my notebook.
But when rebooting it today, Murphy attacked me very hard – the LVM didn’t initialize any more at all. Neither the PVs, nor the VGs + LVs were initialized.
At first I thought this might be caused by a faulty initrd, missing the LVM binaries, but after checking the initrd using a LiveCD, I found everything to be ok.
The surprise came, when trying to initialize the LVM using the LiveCD – it failed also here:
root@grml / # pvscan
Incorrect metadata area header checksum
Incorrect metadata area header checksum
WARNING: Volume Group vg-harddisk is not consistent
PV /dev/dm-0 VG vg-harddisk lvm2 [232.65 GB / 96.10 GB free]
Total: 1 [232.65 GB] / in use: 1 [232.65 GB] / in no VG: 0 [0 ]
root@grml / # vgchange -ay --ignorelockingfailure --partial
Partial mode. Incomplete volume groups will be activated read-only.
Incorrect metadata area header checksum
Incorrect metadata area header checksum
Incorrect metadata area header checksum
Volume group "vg-harddisk" inconsistent
Incorrect metadata area header checksum
Incorrect metadata area header checksum
WARNING: Inconsistent metadata found for VG vg-harddisk - updating to use version 13
Incorrect metadata area header checksum
Automatic metadata correction failed
Ouch, this looks bad!
What should I do now? I remember, there’s a tool called ‘vgcfgrestore’, which is able to restore your LVM by using the backup files which are created on every LVM operation. Yes, I do even have backups of my data – but somehow I managed to exclude /etc from my backup – the needed files are inside of the PV and the root-LV.
My first idea was, to use ext3grep to do the dirty job of extracting the files from filesystem, but it turned out, it’s not that easy running ext3grep on a raw PV without knowing where the superblock of the filesystem, nested into the PV, the VG and the LV, resides.
So I’ve googled a bit for tools, which help extracting files from raw devices and I found ‘foremost’ (later I found out, there’s a compatible successor/rewrite called ‘scalpel’).
Foremost is able to extract files from a raw device, based on pattern matching. It has several strategies for detecting files, but there was none yet to recover plain text files containing LVM metadata.
After playing a bit with foremost and it’s options, I’ve added this line to /etc/foremost.conf
NONE y 20000 #\sGenerated\sby\sLVM2\sversion #\sGenerated ASCII
The meaning of the elements in this line:
NONE – This is, were usually the file suffix (jpg, pdf, …) is placed. But in our case, it was just a plain text file without an extension, so NONE has to be used here.
y – This enables case-insensitivity when searching for a match, ‘n’ would disable it.
2000 – this specifies the maximum filesize we’re looking for. I don’t know whether these are bytes and what value would actually make sense here, so I’ve used the same value as it was used in some example code for detection of C++ source files.
#\sGenerated\sby\sLVM2\sversion – This is the string we expect to find in the header of the file. The sequence \s is used for a single whitespace.
#\sGenerated – This is the string we expect to find in the footer of the file. As the footer of an LVM metadata file ends with } I’ve tried to find a matching expression, but couldn’t find out how to notate a newline in the foremost syntax. So I found the hint on another website to just use the same value as for the header.
ASCII – Let me quote the foremost.conf here: The ASCII option will extract all ASCII printable characters before and after the keyword provided. Voilá, this explains also, why using the same value for the footer expression as in the header expression works.
Now I ran foremost using this options:
foremost -c /etc/foremost.conf -vd -i /dev/mapper/cryptroot
The result were a lot of strangely named files and when looking at some of them, I was first really disappointed. They were just messy and not usable at all.
But after waiting 3 Minutes more, foremost found some more files – and when I looked at them, they were really fine. Complete nice files without the slightest corruption.
I used grep + awk + sort to get the most recent files:
grep creation_time * | awk '{print $3}' | sort
I’ve compared now the current state of my LVM (used ‘lvmdump’ to retrieve the current information) with the latest file from the backup and I was surprised – they were exactly the same (except of the creation timestamp).
So the general structure must be still ok, it seems just the VG header was messed up somehow.
I’ve followed now the instructions here:
I’ve extracted the UUID of the PV from the backup file and ran this command:
pvcreate --uuid "Enteh9-OyyN-UovO-wC6I-zRqe-kIZK-VKyfRC" --restorefile lvm-metadata-backup /dev/mapper/cryptroot
But unfortunately it failed:
Incorrect metadata area header checksum
Incorrect metadata area header checksum
WARNING: Volume group vg-harddisk is not consistent
Can't initialize physical volume "/dev/mapper/cryptroot" of volume group "vg-harddisk" without -ff
But luckily, this tool was smart enough to tell me what was missing – I did it again using the -ff option and I had my PV back, but the VG was still missing – I couldn’t see anything when running ‘vgscan’.
Now I could finally use the ‘vgcfgrestore’ tool which I intended to use:
vgcfgrestore -f lvm-19 vg-harddisk
Restored volume group vg-harddisk
Now running ‘vgscan’ and ‘lvscan’ worked just fine and after 9 hours of struggling with this, I finally had my LVM back!
But don’t think you could just run ‘pvcreate’ + ‘vgcfgrestore’ and your LVM is there again – always make sure the current layout of your LVM matches the one from the latest backup.
So all the steps described above are still necessary.
Since quite some time, I’m running LVM nearly everywhere – also on my notebook. But when rebooting it today, Murphy attacked me very hard – the LVM didn’t initialize any...
