The truth about the Linux kernel

Arch Linux maintainer of the –beyond Linux kernel version announces that he won’t be a maintainer of that package anymore:

I’m sick of the kernel. It’s crap. Every release breaks something new, and every release is as poorly tested as the last. (not you tpowa [another maintainer], I mean upstream — you do a good job with what you’re given by them). The stable tree, isn’t stable. I want something that works and won’t break.

James, hat off to you!

That’s what I like! People who are not afraid to tell the truth. THESE are the people who can bring real value to a community. And the reality of the matter is, that no, the Linux main kernel is NOT properly tested before release. Major bugs slip with each release. The Linux kernel is patched up pretty quickly overall, but the point remains, you have things that didn’t work before and they now are fixed, and things that did work before but now aren’t. I am running Ubuntu and Arch on several machines, and each one has major bugs, ranging from ACPI to USB to Bluetooth initialization etc. Bugs that didn’t necessarily exist in the previous release.

The Linux kernel, and 99% of the open source applications that I have tried, only beta test, they do not do PROPER testing with pro tools and professional testers who are able to write specialized code and scripts to find bugs. Instead, the OSS community only releases versions deemed as “betas”, and then they are relying on average Joe for bug reports. I am sorry, but there is a full stage of testing that is completely skipped by these OSS projects. And that’s pro-testing, in house. And in the case of the kernel, they need at least 50-60 different computer hardware and at least 10 pro testers to test the kernel BEFORE they release an -RC version to the world.

But OSDL seems to mostly be about the “face” of the Linux kernel rather than actual work on it. Thank God that Red Hat actually *does* some of that work in-house (so their RHEL-branded kernels are not too bad). Ubuntu only does beta-testing, not pro testing — and that’s a major minus point off them as far as I am concerned. I am not sure what Novell does.

I remember Be’s testing lab, they had 4-5 pro testers in there and a room full of machines to test with (sometimes with scripts running overnight). And remember, Be’s kernel was tinee-tiny compared to the thousands (millions?) of lines of the Linux kernel code, and yet they were doing as much as in-house testing as they could before releasing betas to their beta testers. Think how much work the Linux kernel would require before they shove us a new version every 5 months!

I have been saying that stuff to my friends all along, but I am glad that someone from the Linux community itself has stepped up and told the truth. Linus himself must ask OSDL to hire pro-testers, or he should quit. That would be the right thing to do if he cares about the quality of the kernel.


Bruce wrote on May 11th, 2007 at 2:24 AM PST:

“using SPECIALIZED professional scripts/software and pro testers.”

Ooooh… “SPECIALIZED”… “pro tools”…

Adjectives aren’t impressive. Until someone seriously competes with the uptimes and reliability produced by the likes of VMS, talking about your tools is just talk. What is testing “fully”? Testing ’til they feel like releasing it. Which is exactly what the Linux kernel folks do. There is no “fully”, there is how much will you spend (time or money), and how much loss you are willing to accept.

This is the admin speaking...
Eugenia wrote on May 11th, 2007 at 2:45 AM PST:

Sorry Bruce, but Linux has proved time and again that it’s more buggy than the kernels that do get proper testing. That’s my problem. Not if Apache runs on top of Linux and has an 99.99% uptime, but when the usb bus does not wake up correctly, when the bluetooth hardware is not initialized correctly, when hardware drivers crap up after a wake up, when…… You get the picture. Linux under controlled situations on a server works well. It is on a desktop and the needs that a modern desktop mandate that it fails at. And that’s not because it doesn’t have the features, but it is simply put, buggy. Buggier than its competition at least.

roybatty wrote on May 11th, 2007 at 3:51 AM PST:

You go girl. Too bad you’re a leftist.

This is the admin speaking...
Eugenia wrote on May 11th, 2007 at 4:30 AM PST:

>99% of free software developers can’t afford testing.

OSDL can.

Tom Dison wrote on May 11th, 2007 at 4:47 AM PST:

You are partially correct.

I don’t think it necessarily has anything to do with Open Source vs. Proprietary. I think it has more to do with the size of the Project and the amount of money involved, and the company mindset behind the project. I am a software developer. I have workd for 9 years now, mostly in the Microsoft-based closed-source developement world. So far, even though I am often working on projects that sell for many thousands of dollars (some more than a million), none of them have used any automated testing. In fact, the quality control has been abysmal – code and burn I call it. I think you only see this kind of automated testing in very large companies, like Sun, Microsoft, IBM, etc. Perhaps small companies can do this if they are working on a very “niche” project where this kind of quality testing is actually part of the product (such as embedded applications).

The state of software development across the board is pretty poor. In fact, many of the tools for quality software development – Ant, Nunit – actually come from the free software world. Even though they are available, most of the small-to-medium sized proprietary software development companies do not use them. As I mentioned earlier, often this software is being developed for Fortune 500 companies, and it is very expensive. Go figure!

Having said all that, I would agree that it is time for companies/organizations like IBM, Redhat, OSDL, Canonical to step up and implement some better QA/QI in the Linux world. Just don’t forget, a lot of this automation is actually fairly new to even the Microsoft world. You would be surprised how buggy a lot of Windows code was. I think half-way through the Vista process, Microsft scrapped the way it was developing Windows and went to a more automated, modular system.

Bottom line, users must expect a whole lot more quality from their software – closed and open – or things will not improve. Don’t accept slapped-together spaghetti code. Just remember, you will pay more for it (although paying more doesn’t guarantee it!). More importantly, it may take longer and may not have the latest/greatest “kitchen sink” features.


Moulinneuf wrote on May 11th, 2007 at 6:06 AM PST:

Eugenia … nice rant , Hope you feel better now.

l3v1 wrote on May 11th, 2007 at 8:21 AM PST:

“That’s what I like! People who are not afraid to tell the truth”

Whadda ?! You make it sound like it someone would hold a gun to the man’s face not to say a bad word. Like you expecting people to complain and jump happily, like, hey finally someone sees the light I see. Anyway, it feels like, every Linux-related outcry must be the truth manifesting itself at last. Right.

“Linus himself must ask OSDL to hire pro-testers, or he should quit.”

Well said. I hope you also punched your fist in the table while saying that with a serious face :)

Anyway, what I saw was “I simply no longer have the time, or motivation to develop beyond. Uni and life are taking up more of my time, and I’d rather spend what time i have at the computer pursuing things of interest.” which sounds perfectly reasonable to me. Many/most FOSS projects probably need a lot of taking care of, and the kernel isn’t a particularly small piece of software either. But, as always, I still prefer frequent releases with new stuff to rare stable releases with small improvements. Well, shoot me.

mystilleef wrote on May 11th, 2007 at 11:37 AM PST:

99% of free software developers can’t afford testing. They rely on the goodwill of the public and community to make software better. I’ll give a personal story.

I released some software a few days ago thinking it was the most polished work I had ever done. I had been testing for months on my box. I thought it was “bug free”. So I made an alpha release about 3 days ago.

5 mins later – Bug that it couldn’t build on some weird system.

1 hour later – Bug when using an old version of some library.

2 hours later – Bug on AMD64 boxes with some libraries

5 hours later – Bug that it hangs on some weird URLs

A fews hour earlier – Bug in some third party library.

And then a few bugs about users using the wrong libraries and different permutations and combinations of weird libraries and users not even having the right libraries installed, gentoo users compiling with weird flags and on and on.

Yes, I have test scripts. I do unit testing. I thought I had made provisions for almost all nasty possibilities. But reality beckons differently. Most __clients__/__users__ cannot afford to pay me or software vendors the kind of money that is needed to create NASA(SPACE SHUTTLE)-like quality code. Except of course they are the US government.

So there you have it. I along with many free software developers just don’t have the resources to create bug free code. We rely on the community to help make the software bug free. And it’s a reasonable trade off.

This is not an excuse for the kernel developers. With the kind of funding and resources they have, it is clear they need way better testing facilities and processes. I used to think OSDL had some, but I may be wrong. But at the same time, the kernel is an exceedingly complex system and the linux kernel is tortured in ways no other kernel is.

I am only addressing the “99% of free/open software” don’t do testing part of your blog. I personally believe 99% of __all__ software (free or not) don’t do proper testing. But can you blame them? Can we afford to pay for properly tested “bugfree” software?

Some friends and I were having a discussion about games on Linux. They were talking about how Linux and free software doesn’t have games. I pointed them out to this.

Gentoo Games FPS Category

And that’s just under the games-fps section. Gentoo has 17 games-* (strategies, rps, action, genre etc) categories.

Their rebuttal. “Yeah, but 99% of them suck”.

My comeback. “99% of proprietary games I’ve played suck, so what’s your point? Every once in a while you hear of games like Halo and Thief, but that’s once in a green moon.”

We live in a world that is 99% mediocre and 1% genius. :-)

Okay, I’m ranting now. Good day.

spannerotoole wrote on May 11th, 2007 at 12:36 PM PST:

Didn’t Microsoft do this joe blogs testing with VISTA re Longhorn RC1 and RC2?
Weren’t these released to the public and some members of the public replied back with whats broke and what they would like anyone who doesn’t like linux because of this not tested startegy should also look at mac OSX as it is based around a similar concept. IE if OpenBSD is corrupt then so will mac OSX be corrupt as it uses that Kernel as its core.

This is the admin speaking...
Eugenia wrote on May 11th, 2007 at 12:41 PM PST:

You are very wrong. Both Microsoft *and* Apple *and* Sun *and* IBM test their kernels and OSes fully internally, using SPECIALIZED professional scripts/software and pro testers, BEFORE they send betas to their testers. Of course, while they have betas out, they also continue testing too. These scripts produce a lot of bug-finding, with much better documentation for reproduction by their engineers, than beta testers do. Beta testers do have their value in the testing procedure, but they are not the only source that a project/company should base its bug finding on. In most cases, finding a bug is not enough. Full reproduction/analysis is required, and 99% of the beta testers can’t offer that.

Apple’s kernel is not 100% the same as the one in BSD. Apple has a big kernel team which has changed a huge deal of the original forked code. Apple’s code is not the same as the original forked code anymore. A lot of changing and testing since then has happened. Don’t fool yourself.

KCorax wrote on May 12th, 2007 at 6:29 AM PST:

I believe testing with ‘pro’ tools looks sth like this and certainly ran over a room of diverse machines for each hardware combination they ‘officially’ support. If OSDL can’t sustain that then the HW makers of these architectures should.

People who don’t know squat about mission critical dev should keep abstain from making emotionally charged comments. Btw in Microsoft for every developer position they hire a mirroring ‘developer in test’ and they still have problems.

I strongly disagree however with a core developer on the kernel *fucking* openly saying this. I don’t expect a developer to understand politics but I think OSDL should fucking censor sth like that. I can see the ms evangelists screaming the quote over rooftops, in context with their still-on ‘get the facts’ campaign.

This is the admin speaking...
Eugenia wrote on May 12th, 2007 at 6:53 AM PST:

> I strongly disagree however with a core developer on the kernel *fucking* openly saying this.

The developer is not an OSDL employee, but a kernel maintainer of a distro. So, he doesn’t answer to OSDL, and he SHOULD BE FREE to make such statements. It is his right, it is his blood and sweat that he put into maintaining/fixing the kernel for 2 years now. He should have the right to express his opinion on the state of the kernel, just as he did. In fact, without telling the truth out there, things might never get fixed. I find his openess very positive for the project. Now, if MS uses it for an ad, that’s completely beside the point. It is like saying that you don’t want people to talk behind your back that you didn’t have a bath for a month, but on the other hand you still don’t go and have a bath!

> I think it has more to do with the size of the Project and the amount of money involved, and the company mindset behind the project.

OSDL took responsibility of a kernel that is extremely popular and much-used. For that reason alone, they should hire pro testers. If they don’t have the money, they should release control of the kernel to a company/organization that does. It’s about *responsibility*.

KCorax wrote on May 12th, 2007 at 8:04 AM PST:

I believe that a developer who chooses to participate in a project like that is motivated by more things than the money which is obviously not there. There is a vision with OSS, which at this point no one has the right to trash.

This isn’t about ethics, it’s about sustainability. A kernel is obviously a hell of a place to work in, but we shouldn’t discourage those young developers looking to get in.

You don’t bitch publicly in a project like that regardless of weather you are righteous or qualified in doing so. It’s unprofessional and in this case a PR disaster. You mail some people privately and move out with a ‘thx for all the fish’ notice.

This is the admin speaking...
Eugenia wrote on May 12th, 2007 at 9:26 AM PST:

KCorax, this is where we disagree. At that point exactly.

Your goal is to do “damage control”, while I am all about telling the truth to the community. I am sorry, but I don’t think that the way you see things in this case is a very objective one, or the right thing to do or even… fair! Younger developers who would want to hack on the kernel must KNOW where they are getting into. Because otherwise, their expectations will be crashed. Besides, by saying that something sucks, eventually would make people wanna fix it.

I believe that what James did was the right thing, he told the truth, and BECAUSE of that reason, things will get better. Not worse.

You seem to be afraid of what Microsoft will do or say. I don’t. Let them say whatever they want. As I said earlier, if you don’t want others to discuss your hygiene, go have a bath. Do not hide the truth under 3 winter coats.

This is the admin speaking...
Eugenia wrote on May 13th, 2007 at 6:59 AM PST:

The Linux kernel does not require any NDAs, it is an open platform. Emphasis on the word “open”.

Trent J. Townsend wrote on May 13th, 2007 at 8:11 AM PST:

Mind boggling some of the responses you get from people after hearing that Linux is indeed as crappy as most other software on the planet.

I do agree with the idea that openness with its failings can only help to make things better and am absolutely bewildered at posts from people who say that the Arch Linux maintainer had no right to speak of such things in public.

Freedom of speech anyone? Its a freaking kernel for crying out loud ;^)

Trent J. Townsend wrote on May 13th, 2007 at 11:05 AM PST:

Eugenia beat me to it KCorax… The minute a project requires NDAs is the moment it ceases to be an open project.

Last I checked, core Linux development required no such thing…

KCorax wrote on May 13th, 2007 at 11:08 AM PST:

Obviously so. That was a comment as to weather limiting free speech in the context of team work is ethical, desired or possible.

Open applies to code not dev methodology. There’s plenty of software that is open where developers stay mum about problems, or find less destructive ways to say just what they wanted to. See Redhat, Novell, JBoss etc.

This is the admin speaking...
Eugenia wrote on May 13th, 2007 at 11:43 AM PST:

>limiting free speech in the context of team work is ethical, desired or possible.

There are times where it is ethical and desirable. But I don’t think that this time qualifies as such.

KCorax wrote on May 13th, 2007 at 11:38 AM PST:

OMFG Trent, so now I’m a fascist ? Have you heard of NDAs ? What happens with free speech there ?

Comments are closed as this blog post is now archived.

Lines, paragraphs break automatically. HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

The URI to TrackBack this blog entry is this. And here is the RSS 2.0 for comments on this post.