Category Archives: Linux

Talking about Poky and Yocto

I just returned from an excellent week in the Pacific North West of America where I attended Intel’s Open Source Technology Summit (OSTS), our internal open source conference.

The view over the laptop was amazing

Whilst there I presented a talk on Building Using Poky; the Yocto Project Build System the slides for which I’ve just uploaded to my Talks page. This talk was a high level overview of Poky – with the aim of presenting the system to the audience in such a way that they could begin to understand how it works and how to start getting things done with it. That and to introduce the GUI I’ve been working on.

I also uploaded the slides from last years OSTS talk, Using Poky to Develop for Embedded Systems, the information there is a little out of date in places but could still be useful to folks.

The Beaver bites (recovering from a failing hard disk)

Last week I had a hell of a time when the Kingston SSD in my laptop started failing, badly. Poky is well known as a destroyer of disks but it seems cheap SSD’s are even more susceptible to it’s toothy maw.

At first it was just (extremely noticeable) performance degradation, but minutes after telling my manager about it and asking for a new disk it was starting to look more serious: directory listing (file manager, fancy filename completion in my editor and good old ls) was failing on my home directory.

I figured I had to get as much data off the drive as possible before I shut the machine off, rsync wouldn’t work as it needs to get directory listings. Fortunately Roger pointed me at ddrescue, which enabled me to carve a partition on an external drive and copy do a direct clone of the failing drive. ddrescue works by following the British mantra of “Keep calm and carry on”, that is if ddrescue finds some data it can’t copy it gives up and moves on copying as much data as possible rather than exiting because corrupt/unrecoverable data was found.

I managed to recover most of my data with ddrescue and while it was going I had to set up another machine to use while I was waiting for the replacement disk. This gave me opportunity to try out a backup solution and various tactics to make my data easier to recover.

The biggest problem, email. Having all my mail filtering and account configuration trapped in Evolution meant that I rely on it more than I am comfortable with. Based on recommendations and usage in the office I decided to try offlineimap and imapfilter.

Offlineimap is a Python program that will download your mail and store it in the maildir format, meaning that you can use any of a range of mail programs to read your mail. Imapfilter is a Lua program that will connect to an Imap server and move mail around on the server. You write simple Lua scripts whereby you instantiate an Imap object then write bits of code to move messages around based on the mail headers:

msgs = imapacc.INBOX:is_unseen() *
imapacc.INBOX:contain_field(“X-mailing-list”, “linux-fsdevel@vger.kernel.org”)
imapacc.INBOX:move_messages(imapacc[‘INBOX/linux-fsdevel’], msgs)

Brilliant! Now my email is stored in a Mail folder in my home directory and imapfilter connects to the mail servers directly and sorts my messages into appropriate mail boxes.

For backup Ross recommended duplicity, another Python program. This program uses librsync and GnuPG to create GPG encrypted incremental backups.

After the first run duplicity will only backup what has changed, and all of your backups are stored in GPG encrypted tar archives. The first run didn’t take too long to backup my 20GB home directory and I expect future backups will be even swifter.

Now that I have my new disk installed and my data in a more recoverable format with a simple backup solution I thought I’d try and address the cause of the disk destruction.

Blunting the teeth (trying to stop Poky so easily destroying disks)

Poky, and OE, are known as destroyers of disks, for understandable reasons. Building an operating system means a lot of disk activity, to increase the lifespan of the disk it would be good to reduce the amount of disk activity.

Now I’m not expecting my new disk to fail as rapidly, it’s an old school platter-based drive, but as I do all of my development on a single disk machine (laptop) the simplest solution; to use a separate disk for builds, is impractical.

I decided to create a separate partition for Poky builds (simple thanks to LVM) and use a filesystem setup which would hopefully reduce the amount of writes.

I chose to use an Ext4 filesystem without a journal. With this set up I get the modern filesystem goodness (extents and delayed allocation should equate to less disk activity) and also fewer writes through not using a journal (as it’s a partition for temprorary build data, so I’ll not be upset if I lose any/all of it). I strongly considered Xfs and Btrfs, I decided against Xfs after reading about it’s slow directory creation and deletion (Poky does a lot of this) but will likely try Btrfs soon, though I’m more inclined to play around with that on a set up that can benefit from its snapshot support.

As far as I can tell you can’t disable journalling on an existing ext4 filesystem, you need to create the filesystem without a journal. Fortunately I was working with a new disk where I’d left plenty of unused space. I created a new logical volume to house the partition, then created an ext4 partition in that logical volume using the default filesystem features (from /etc/mke2fs.conf) only, with the journal option disabled.

mkfs.ext4 /dev/mapper/vg_scimitar-lv_srv -L srv -O ^has_journal,extents,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize

Et voila, hopefully that will increase the longevity of the disk.

While losing several days of coding time to this was initially a little frustrating it was fun and productive to play around with storage, backup and mail solutions to make my working environment a little more pleasant and my data more secure.

All the toys!

I’ve acquired a bunch of new gadgets of late, I’m very excited about them all so I decided to promote them here.

Reading

A Sony PRS 600 E-Book reader was gifted to me this Christmas, and what a fantastic piece of kit it is.

I’ve been enjoying several public domain texts from Feedbooks as well as a bunch of geeky PDF’s.

The PDF handling is not as bad as I expected, at standard zoom it’s just about readable in good light although much more pleasant to read at M(edium) or L(arge) text size, despite the slightly weird line wrapping you end up with.
Unfortunately tables and diagrams are completely destroyed when the text size is increased so I tend to set it back to S(mall|tandard) text size for those bits and sometimes use the Zoom function to make text more legible in low light.

The touch function is fun and sometimes even handy for turning pages and looking up words as well as highlighting and annotating the more academic reading materials.

I’ve been using Calibre to manage the device. While the UI is not to my taste (photo-realistic icons, busy interface, etc) the functionality is excellent and extremely stable. I should really donate some money to the project.

Listening

The iPod Shuffle just wasn’t cutting it as a portable audio player once I moved to London and had lengthy commutes to fill my ears on. I tried using my Android phone but the short battery life made this a chore, so I decided to pick up a new audio player.

I wanted something relatively cheap which had to have a screen (yes Apple, a screen!), a reasonable amount of storage (more than 2GB) and allow me to put music on it by simply copying files. For extra points it would play Ogg Vorbis and Flac audio.

The Sandisk Sansa Clip fits all of my requirements. With 8GB storage, support for all the formats I care about (and more) and a built in FM tuner it’s a steal at less than fifty notes. I’ve only had it a couple of days but battery life seems reasonable and it charges in a few hours over USB.

Watching

Long have I been meaning to build a PVR machine, and this Acer (Veriton N260G) computer was too cheap to pass up on.

Pair it with a Hauppage WinTV-NOVA-T DVB-T tuner and an install of MythTV and I have a low power PVR running with little cost or effort.

I’ve a few niggles to iron out on the software side but this seems like a more than capable machine at less than two tonne. I’ve not tried watching live TV while recording or trying to shove high def through it but both of these are on my agenda. For now it’s happily (and very quietly) humming away recording things for me.

MythTV is a bit of a pig though, if I was building a media PC without PVR functionality I’d be running Boxee or XBMC and would have had it running in less than an hour from no-OS to full system. Alas that was not the case with the MythTV set up…