Category Archives: Work

Cooking on Gas

It’s been a while since I wrote anything original and interesting here and even longer since I wrote about work. Let’s try and rectify that.

The following is a re-post from the Yocto Project Blog, though I originally wrote it for this blog.

For the last 18 months or so I’ve been working full-time on the Yocto Project, where I’ve touched various parts of the system. I’m really proud of what our team has achieved – we have an amazing tool that people want to use! Despite that, there are still some common tasks that users can’t achieve as easily as we’d like…

A gas hob

Untitled By sunshinecity on Flickr

Our system, Poky, has traditionally assumed a certain familiarity (to varying degrees) with Open Source software, Linux and building software. However the more competent our build system becomes the more we attract the interest of potential users who aren’t a typical UNIX greybeard.

The Yocto Project has been making various efforts to improve the usability of the project at all levels. We have an excellent technical writer on our team who works (laregely underappreciated) on improving the documentation for our project, we have a small but dedicated team of people making application development for Poky built images easier – much of this work focused around enabling as much of the development as possible to happen in the premier Open Source IDE; Eclipse and a project-wide focus on making things simpler and easier to understand.

In line with this effort I’ve spent most of my development time over the last cycle working on a GUI for the Poky build system called Hob (the name ties in with the baking theme – a hob is what we call the top cooking surface on an oven in England).

The aim of Hob is to enable a user to perform common tasks graphically, we focused primarily on enabling you to generate a custom image for the initial release but have several plans for enhancements over the coming months.

By way of introduction I spent some time producing a video to demonstrate use of the Hob (I’ll avoid ranting about the state of Linux video editing tools), it’s available to watch on YouTube and Vimeo.

The challenge of the Hob was turning the traditionally batch-run BitBake program into something more interactive, unfortunately I underestimated how much effort this would involve such that what I was able to deliver in the Yocto 1.1 time-frame isn’t quite as functional as originally designed.

For Hob the problem is that there is only so much information we can know about the how a package will be built by looking at the metadata. For example: we can’t know which of the build dependencies will actually get linked into a resulting binary, so we can’t do accurate image contents prediction by operating purely on the build system metadata. To that end this initial version of Hob comes with a caveat, the reported image contents are only an estimate.

As well established by Brooks, this version of Hob ended up being sort of a Pilot System (“write one to throw-away”) and we’ve learned a lot from developing this tool. Fortunately I followed best practices as already implemented in the BitBake code base to modularise as much of the Hob GUI as possible. Further, as Hob is actually a BitBake GUI, rather than some kind of wrapper program, many of the improvements I made are in the core BitBake libraries and much of the code will live on as we develop Hob in future releases.

For the next release we’ll certainly be improving Hob, and whilst nothing is set in stone at this stage, we are contemplating switching to a two-phase build – where the custom image generation is done once packages have been built. If we go this route we can be more accurate about dependencies and the size of the resulting image. Our initial feedback on Hob leads us to believe that as many people care about customising the image after the build as care about customising how it’s built. With a two-phase mechanism we’ll be able to build GUI’s that work well for both scenarios and can be used seamlessly together in tandem.

Talking about Poky and Yocto

I just returned from an excellent week in the Pacific North West of America where I attended Intel’s Open Source Technology Summit (OSTS), our internal open source conference.

The view over the laptop was amazing

Whilst there I presented a talk on Building Using Poky; the Yocto Project Build System the slides for which I’ve just uploaded to my Talks page. This talk was a high level overview of Poky – with the aim of presenting the system to the audience in such a way that they could begin to understand how it works and how to start getting things done with it. That and to introduce the GUI I’ve been working on.

I also uploaded the slides from last years OSTS talk, Using Poky to Develop for Embedded Systems, the information there is a little out of date in places but could still be useful to folks.

make frustrated

November saw the release of Fedora 14 which includes the new and improved GNU make 3.82, this includes some fixes for long standing bugs in the parser – and therefore breaks dozens of projects’ Makefiles.

Common signs of the new, strict, make are the following output at make time:

Makefile:246: *** mixed implicit and normal rules. Stop.

This means you have both explicit and pattern targets in your make rule, this caused issues if explicit targets where used in the rule *after* pattern targets make could misparse the entire rule – of course, make had been doing this for long enough that everyone knew to avoid doing that…

Makefile:923: *** missing separator (did you mean TAB instead of 8 spaces?).  Stop.

Make has always required tabs, rather than spaces, for indentation. But in previous versions of Make it was permissible to use 8 spaces instead of a tab. This is no longer allowed in make 3.82!

Whilst the Makefiles of most projects have been updated we where bitten by this pretty hard where our stable release of Poky Laverne 4.0, released October 26th, can’t be built on Fedora 14. Ouch!

So this week, I donned my distribution engineer hat and set about resolving these issues, I had to hack the Makefiles of some dozen or so projects (and even did some guerilla pushes to Gnome git).

The need to do this has left me extremely frustrated, these bugs have existed in the parser for long enough that they had been widely relied upon in many projects – to ‘fix’ the parser and therefore break the Makefiles of so many projects feels like a pretty severe move to me. Could this not have been a warning in make 3.82 and turned into an error later?

This is the second case of upstream introducing regressions that have hit us in Poky recently, the first being an regression in Python 2.7’s xmlrpclib which changed the way some private methods worked (parse_response() would no longer accept a file-like object) breaking our XML-RPC server…

Unfortunately this bug was not fixed for Python 2.7.1, but has since been fixed upstream.

Whilst I’ve ranted here I do really appreciate both of these open source projects and my knowledge with both Makefiles and Python programming has increased whilst debugging and working around these issues for Poky. There’s still a part of me that wishes I hadn’t had to do it, though!