The other day, while doing some normal maintenance and removing a few files, I accidentally removed the only copy of a particularly important disk encryption key. I have no backups for most of the data on these drives¹, and while it would be possible to rebuild most of it, it would take significant effort. To make matters worse, while I theoretically had the drive space available to copy the data off, it would have taken at least 2 drives, multiple days, and I had about 12 hours left before leaving town for a week.

I briefly considered two options. First, the key was stored on an ext4 filesystem, and there are easy-to-use tools available which will attempt to recover recently deleted files from such a filesystem. However, there’s no way to tell if such a tool will work without rebooting and trying it out. I had the disk open, which means the volume key was in memory somewhere; I quickly realized that I would need to find a way to dump the key from memory.

dmsetup

In the beginning, there was dmcrypt. Then, someone made LUKS. And then, someone made LUKS 2. Then they all rested.

In the olden days (LUKS 1 and plain dmcrypt), it was trivial to retrieve the master encryption key (“volume key” in LUKS-speak): you could simply ask dmsetup table (as root) and it would dump everything you needed to know. In LUKS 2, the key is now stored within a kernel keyring, and is not designed to be exportable. Various posts online simply say that it’s not possible and to start copying data off. Other posts say that it might be possible, but “not sanely”. Nevertheless, as a fairly smart and occasionally insane fella, I decided to attempt it.

kmem

The first thing I thought of was /dev/kmem, and for a bit I looked into whether that would contain the relevant memory regions, but shortly I noticed that my kernel has CONFIG_DEVKMEM disabled, so I don’t have such an option. At this point I was about to begin copying the most important data to another drive, but then I realized: surely someone has made a kernel module that will create a full memory dump, for forensic or debugging purposes.

LiME

Sure enough, such a thing exists, and while it’s unmaintained, it still builds and runs just fine. All that was needed was a quick make; insmod ./lime-*.ko "path=/root/memdump format=raw" and suddenly I had a memory dump sitting before my eyes.

At this point, the only challenge remaining was finding the keys within the memory dump; I figured I could find a tool to simply scan for anything that looked like a key, and indeed it was that simple. I found an article detailing the usage of a tool called findaes, which does exactly what it says on the tin, and applied it to my memory dump. The volume keys were some of the first keys that it spit out (and logically so, since it’s mounted - and the keys are loaded - during early boot).

Then it was a simple matter of converting the keys from the hex output of findaes, into binary, and then seeing if it worked. I used Perl to quickly turn it into binary (perl -ne 's/ //g; while (/(..)/g) { print chr hex $1; }' >mayberealkey), and then it was ready to test. The last thing to decide was how to test it; since I needed to replace the key anyway, I went ahead and did that. A quick cryptsetup luksAddKey --volume-key-file mayberealkey /dev/nvme0n1 later and all was well in the world again.

At this point all I had to do was run a cryptsetup reencrypt to rotate the volume key (since I had saved it to disk unprotected, in the memory dump) and go get some sleep before my flight!