Kernel Hell and what to do about it

I've been in kernel hell with my home system for the past couple of days. What I want to build is a custom kernel that'll do xen, vserver, vmware, selinux, support both my wireless chipsets and support my video chipset. Ideally it should be built the Debian/Ubuntu way, so it just works on my Ubuntu Hardy Heron system.

So far I've had various combinations of four or five out of six working at once.

I'm not a kernel hacker, but I have a PhD in computer science, so I should be able to at least make progress on this, and the fact that I can't is very frustrating. At work I grabbed a kernel off a co-worker, but it wasn't built the Debian/Ubuntu way.

Standing back and looking at the problem, there seem to be two separate contributing factors:

1. There are a huge number of organically-grown structural layers. I count git, the kernel build scripts, make, Linus's release system, the Debian kernel building system and the ubuntu kernel building system. I won't deny that each of these service a purpose, but that's is different points a which each of the six different things I'm trying to make work can begin their explanation of how to make them work and six different places for things to go wrong.
2. There are about many Linux distributions and each of the things I'm trying to get working caters to a different set of them.

In many ways the distribution kernel packagers are victims of their own success, most Ubuntu, debian and RedHat kernels just work because they're packagers keep adding more and more features and more and more drivers to the default kernels. With the default kernels working for so many people, fewer and fewer people build their own kernels and the pool of knowledge shrinks. The depth of that knowledge increases too, with the each evolution of the collective build system.

Wouldn't it be great if someone (ideally under the auspices of the OSDL) stepped in and said "This is insane, we need a system to allow users to build their own kernels from a set of <git repository, tag> pairs and a set of flags (a la the current kernel config system). It would download the git repositories and sync to the tags and then compile to the set of flags. Each platform can build their own GUI and their own backend so it works with their widget set and their low level architecture, but here's a prototype."

The system would take the set of repositories and tags in those repositories and download the sources with git, merge the results, use the flags to configure the build and build the kernel. Of course, sometimes the build won't work (in which case the system sends a copy of the config and the last N lines of output to a central server) and sometimes it will (in which case the system sends a copy of the config and an md5 checksum of the kernel to a central server and optionally uploads the kernel to a local repository), but more than anything it'll make it easy and safe for regular users to compile their own kernels. The system would supplant "building kernels the Debian way" or "building kernels the RedHat way" and enable those projects working at the kernel level to provide meaningful support and help to their users on distributions other than slackware.

Potential benefits I can see are:

1. increasing the number of crash-tolerant users willing to test the latest kernel features (better testing of new kernels and new features, which is something that's frequently asked for on lkml)
2. easing the path of new device drivers (users get to use their shiny new hardware on linux faster)
3. increasing the feedback from users to developers, in terms of which features people are using/interested in (better, more responsive, kernel development)
4. reduce the reliance on linux packagers to release kernels in which an impossible-to-test number of features work flawlessly (less stressed debian/ubuntu/redhat kernel packages)
5. ease the path to advanced kernel use such as virtualisation

You know the great thing about that list? Everyone who would need to cooperate gets some benefit, which means that it might just happen...

Macrons and URLs

Macrons are allowed in the path part of URLs, but not currently in the machine-name part (or at least, not yet), so http://extensions.services.openoffice.org/project/māori-papakupu is good, but http://www.taiuru.Māori.nz/ is not (use http://www.taiuru.maori.nz/).

A review of how lots of programs handle macrons is at http://research.elabs.govt.nz/macron-support-in-open-source-web-applications/.

Exporting firefox 3.0 history to selenium

In the new firefox 3.0 they've completely changed he way history is recorded, using a SQL engine to record it (details here).

I wrote a quik hack to export the history as a series of selenium tests:

sqlite3 .mozilla/firefox/98we5tz3.default/places.sqlite 'select * from moz_places' | awk -F\| '{print "<tr><td>open</td><td>"$2"</td><td></td></tr>" }'

sqlite3 locks the file, so you'll need to close firefox (or take a copy of the file) first. Cut and paste the results into an empty selenium test.

Obviously, your profile will have a different name.

OpenID - Everybody Wants To Go To Heaven, Nobody Wants To Die

I've been looking recently at OpenId a little more closely. It's a great system, that I've been using for a number of years.

My first free openid provider (http://stuartyeates.videntity.org/) went belly up for reasons I'm still not too sure about, but the life expectancy of internet startups and free services has never been that long. I have a new identity (http://stuartyeates.myopenid.com/) which does me just fine.

One thing I've been noticing is that while a host of the internet services that I use want to be my identity provider, very few of them want to let me login using an openid provided by another identity provider. From a business point of view I can completely understand this: they want to know everything about the user so they can (a) off better, more integrated services and (b) see more customised advertising. I have no problem with (a), but (b) is one of the reasons I'm eying up openid in the first place.

They don't understand that to get to openid heaven, they're going to have to die. By giving away the user-authentication-and-give-us-your-personal-information step, they can drive significantly more logins and significantly deeper interaction with their website and with their content. Sure, they can't necessarily get access to the users email address and whatever fake personal details they submitted at sign up, but I've yet to see this used particularly well anyway. The personal details certainly don't seem to be used to anywhere near the same effect as analysis of user behaviour.

What would be really good is an openid identity provider with (a) shared no information with any of the large advertising groups or dominant internet companies, (b) really clear information about privacy expectations (and I'm not talking here about a page of legalese titled "privacy policy," but discussion and disclosure of things which jurisdictions user data is stored in, steps taken to avoid collection of user data, etc) and (c) a clear sustainability model to prevent it being bought-out for the user data and to ensure it's continuity (I can imagine paying a subscription for it).

I can imagine that this is the kind of thing that google or yahoo might sponsor in their efforts to promote their next generation of web authentication standards. The existance of such an outfit would hugely boost the reputation of such a standard and lacking the huge advertising and existing lockin it would hardly challenge their own identity providers while answering a whole range of independence, privacy and monopoly questions.

Tinkering with ohloh

I've been tinkering with www.ohloh.net, which is probably best described as a web 2.0 freshmeat. Rather than tracking manually-updated releases it relies on automatically detected updates to version control repositories, RSS and geo-urls. It relies on wiki like editing rather than the strict ownership rules of freshmeat. ohloh does automatic language detection and source code analysis based on the version control repository and attributes individual commits to specific developers and their ohloh user account.

I've added and am actively curating a group of go/baduk projects. The overall goal is to encourage reuse and reduce the willingness of hackers to rewrite go/baduk systems from scratch.

My next step on the technical side is to write some GRDDL (Gleaning Resource Descriptions from Dialects of Languages) to transform the XML returned by the API into RDF, which I can them import into simal.

My next step on the social side is to mention what I'm doing in some of the go/baduk mailing lists, but I want to wait until I've got something concrete to provide that Sensei's Library (the current repository of information about go/baduk programs) hasn't already got.

Happiness is tests passing

All tests passed
Originally uploaded by Stuart Yeates


2 tests failing

I've just (re-)attained the happy state of the unit tests passing in the key package of jgogears. I've invested a surprising amount of time and energy in the unit tests, mainly as a form of requirements analysis, and I'm really pleased with the result.

Sadly, the core package that contains much of the code has some outstanding, longstanding failures which are going to be challenging to fix. They are represent failures in jgogears' ability to round-trip board states between GNU-Go ASCII and SGF files and will require a bug-for-bug reimplementation of the GNU-Go ASCII board printer.

jgogears has reached the stage where it now plays games that are obviously meant to be go, but is not yet a serious contender.

Resetting vmware networking

I had a hardware glitch (apparently chained failure of on-board network card, auto-negotiation on switch, auto-negotiation on larger switch) which left networking on my ubuntu edgy box somewhat randomised. Ubuntu recovered after a reboot, successfully finding the new network card (but calling it eth11). vmware-server didn't recover, and no amount of fiddling with the magic of vmware-config-network.pl would cause vmware to forget all it knew about networking and look only at the current state of play.

In an attempt to erase it's memory I removed /etc/vmware/locations but this caused vmware-config-network.pl not to run at all.

Eventually I worked out that the trick was to remove everything in /etc/vmware/locations other than the lines near the start telling it where to find directories and executables and rerun vmware-config-network.pl selecting all the defaults.

Suddenly everything worked.