The Beaver bites (recovering from a failing hard disk)

Last week I had a hell of a time when the Kingston SSD in my laptop started failing, badly. Poky is well known as a destroyer of disks but it seems cheap SSD’s are even more susceptible to it’s toothy maw.

At first it was just (extremely noticeable) performance degradation, but minutes after telling my manager about it and asking for a new disk it was starting to look more serious: directory listing (file manager, fancy filename completion in my editor and good old ls) was failing on my home directory.

I figured I had to get as much data off the drive as possible before I shut the machine off, rsync wouldn’t work as it needs to get directory listings. Fortunately Roger pointed me at ddrescue, which enabled me to carve a partition on an external drive and copy do a direct clone of the failing drive. ddrescue works by following the British mantra of “Keep calm and carry on”, that is if ddrescue finds some data it can’t copy it gives up and moves on copying as much data as possible rather than exiting because corrupt/unrecoverable data was found.

I managed to recover most of my data with ddrescue and while it was going I had to set up another machine to use while I was waiting for the replacement disk. This gave me opportunity to try out a backup solution and various tactics to make my data easier to recover.

The biggest problem, email. Having all my mail filtering and account configuration trapped in Evolution meant that I rely on it more than I am comfortable with. Based on recommendations and usage in the office I decided to try offlineimap and imapfilter.

Offlineimap is a Python program that will download your mail and store it in the maildir format, meaning that you can use any of a range of mail programs to read your mail. Imapfilter is a Lua program that will connect to an Imap server and move mail around on the server. You write simple Lua scripts whereby you instantiate an Imap object then write bits of code to move messages around based on the mail headers:

msgs = imapacc.INBOX:is_unseen() *
imapacc.INBOX:contain_field(“X-mailing-list”, “linux-fsdevel@vger.kernel.org”)
imapacc.INBOX:move_messages(imapacc[‘INBOX/linux-fsdevel’], msgs)

Brilliant! Now my email is stored in a Mail folder in my home directory and imapfilter connects to the mail servers directly and sorts my messages into appropriate mail boxes.

For backup Ross recommended duplicity, another Python program. This program uses librsync and GnuPG to create GPG encrypted incremental backups.

After the first run duplicity will only backup what has changed, and all of your backups are stored in GPG encrypted tar archives. The first run didn’t take too long to backup my 20GB home directory and I expect future backups will be even swifter.

Now that I have my new disk installed and my data in a more recoverable format with a simple backup solution I thought I’d try and address the cause of the disk destruction.

Blunting the teeth (trying to stop Poky so easily destroying disks)

Poky, and OE, are known as destroyers of disks, for understandable reasons. Building an operating system means a lot of disk activity, to increase the lifespan of the disk it would be good to reduce the amount of disk activity.

Now I’m not expecting my new disk to fail as rapidly, it’s an old school platter-based drive, but as I do all of my development on a single disk machine (laptop) the simplest solution; to use a separate disk for builds, is impractical.

I decided to create a separate partition for Poky builds (simple thanks to LVM) and use a filesystem setup which would hopefully reduce the amount of writes.

I chose to use an Ext4 filesystem without a journal. With this set up I get the modern filesystem goodness (extents and delayed allocation should equate to less disk activity) and also fewer writes through not using a journal (as it’s a partition for temprorary build data, so I’ll not be upset if I lose any/all of it). I strongly considered Xfs and Btrfs, I decided against Xfs after reading about it’s slow directory creation and deletion (Poky does a lot of this) but will likely try Btrfs soon, though I’m more inclined to play around with that on a set up that can benefit from its snapshot support.

As far as I can tell you can’t disable journalling on an existing ext4 filesystem, you need to create the filesystem without a journal. Fortunately I was working with a new disk where I’d left plenty of unused space. I created a new logical volume to house the partition, then created an ext4 partition in that logical volume using the default filesystem features (from /etc/mke2fs.conf) only, with the journal option disabled.

mkfs.ext4 /dev/mapper/vg_scimitar-lv_srv -L srv -O ^has_journal,extents,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize

Et voila, hopefully that will increase the longevity of the disk.

While losing several days of coding time to this was initially a little frustrating it was fun and productive to play around with storage, backup and mail solutions to make my working environment a little more pleasant and my data more secure.