There's a nine-week release because every nine weeks there's a new release going out, right? So there's Linus does a release this point in time and then the merge window is considered open and then for two weeks all the maintainers send Linus all the stuff they've had pending from the last release. We have two weeks to add all new features and then he does release candidate one. From there on it's bug fixes only for the next seven weeks. So it's bug fixes only, bug fixes only, bug fixes, it's regression fixes, we'll revert things, no new features.

Do I understand correctly that in the case of Linux is this a thing where every nine weeks there will be a release? It's time based. So we have that two week window of merging all the new features to Linus that have been in our tree and accepted already and proven to work. And the window is short nine weeks. We used to have three year long development cycles. And the problem there is even if you have six month development cycles, there's that fear of you have a feature, I want to take your feature, but it's not quite ready. Do I want to wait and things like that? But if you know that you can get your feature in in nine weeks from now and it's just not ready, it's not ready. The pressure is off me as a maintainer to take your new feature until it's ready.

Linux is the world's most widely used operating system thanks to powering most Android devices, servers, smart TVs, and embedded systems. But how is it actually built? Today, we sat down with Greg Kroah-Hartman, a Linux kernel maintainer for 13 years, who is one of the three Linux Foundation fellows. In today's conversation, we cover details on how widespread Linux is and why mobile versions of Linux have three times the lines of code as a server versions. What exactly it takes to get a change accept to the Linux kernel and merged by Linus Torvalds himself. How Linux manages to have 4,000 contributors per year yet have no product managers or project managers and many more details. If you're a software engineer, you will use Linux directly or indirectly. And this episode will help you understand why it's so widespread and how it's a lot easier to contribute than most people would assume.

I think as software engineers, we know Linux is important in the sense of it's it's running on most web servers that that we use and run. It's it's a desktop OS that some people use and it's of course, you know, powering a fork of it is powering Android. But what is there to know about Linux? How how big is this thing? How complex is this thing?

Well, it's yeah, we took it's an operating system. So, it's a kernel. We took over the world without anybody noticing. I joke it's Android devices or 4 billion Android Linux users out there or and they don't realize it. That's everything else is a rounding error which doesn't make the server people happy with me but it's true. It's in everything.

It's in all the embedded devices. It's in the air conditioning units, the car electric charging ports, uh satellites, runs international space station. Really? Yeah. Yeah. Air traffic control for Europe and probably the US. All the financial markets. I don't know of any place that hasn't taken over. The number the top five selling laptops for the past 15, 10, 15 years. Chromebooks. Those are all Linux based. Not Apple, but the Chromebooks are. Oh, iPhones. So, every 5G modem out there is running a copy of Linux. Really? Yeah. Wow. Wow. So now with Apple doing their new ship, I don't know if it's the new one, but Qualcomm, all the 5G modems, probably the 4G, I'm not sure, but I know all the 5G modems have Linux inside it.

In terms of the the kernel itself like how large is it? And I I know you know for for different devices it'll be split differently for for serverside Linux for for a an Android device it'll use different parts of the kernel. How how big is this in terms of contributors lines of code?

I know lines of code is not a great measure but lines of code is interesting. So we have just under 40 million lines of code right now. That's a lot. That's all the kernel that's the kernel. core part is like 5% of that that everybody runs and then everybody the rest of it is hardware support different drivers different devices different architectures different chips. so your laptop runs about two two and a half million lines of code your server runs about one and a half million servers are really easy those are very simple things your phone runs about 4 million your phone so are the most complex pieces of CPU and interaction out there they're just crazy complex.

why why is Like can we just pause for a second? So so like again lines of code we know is not a perfect measure of complexity but but in in this sense comparing it between the two of them with the same code base is somewhat so you said roughly give or take a server is one and a half million a phone is 4 million like three times the lines of code for a phone. Why the difference even though I I would I would think that the server you know does all this mission critical stuff.

A server is really simple CPU and a network card and a storage and storage that's it. So SOC on a phone has you have power control, you have clocks, you have five different buses on there talking to different types of devices. You have battery control, you have talking to your modem, you have another version of Linux in the modem. You got USB out the back. You got USB bypass to talk to the audio side. You have audio drivers. You have a zillion different clocks and fives and all sorts of stuff in there. The SOS and it's a eight core machine. It's there's eight processors and nothing. Those are not trivial things. And sometimes those processors are different sizes. So you have big and little sizes which add the complexity just for some control for some power management but they all run the same core of the Linux but it's the drivers and the devices and things like that.

So your Pixel phone I I look at Pixel phone Google ships a core kernel that all Android devices pick not hardware specific just says ARM 64. Pixel has 300 other drivers they add to get the Pixel phone working. I mean some of these are tiny. This is for this tiny chip. this this but your phone is is really one of the most complex beasts out there for software.

Is it safe to say that you know the complexity and you know the lines of code will to some extent scale with that has to do with the hardware the capabilities and and you know not about you know like how mission critical because of course it need the phone needs to be be stable the server needs to be stable my TV needs to be stable so you know that's just kind of a given right yeah oh and all TVs for the past 15 years are all running Linux so that's oh so my Samsung TV is running Linux Oh yeah oh yeah Samsung my your Samsung my Samsung washer and dryer are running Linux. so Um your Samsung watch is running Linux. So um Samsung has their own yeah distro all works really nice.

Yeah it's all due down to the complexity of the hardware. So the kernel controls the hardware. The job of Linux is to make all the hardware look agnostic to programs. So you can write the same user space program and run it on the same on different hardware and it does it just works. hard a kernel's job is to manage memory and devices in a common way and provide that to user space.

kernel versus user space so there's an idea um chips have a protected mode and a not protected mode in a very simplified way there's different levels of protection so The protected mode is where the operating system runs the kernel. Yeah. And that is where we share all the resources. It's one flat address space. And we are not isolating processes. So a user space process then runs on top of that and we isolate them and they they all individually think they have the whole machine but they don't. Yeah. So it's multitasking. You can run multiple programs at the same time. And the kernel is there to give you memory to give access to storage to give access in a common way to give access to the network in a common way to give or provide the pipes to go around the network stack in the user space. We provide a way to make it so that user space can talk to the kernel in an agnostic way and it'll their stuff will just work because it all the graphics work the same interface. We talk to keyboards all the same way things like that.

So it's a commonality of providing a a shim layer above the hardware and then for example drivers do they always live in the user space? So yeah no all our drivers live in the kernel. So the kernel and drivers are all Linux is not a micro kernel architecture it's a monolithic. So the the code is all in the same address space. So a bug in any one of them has a chance to take any part of the kernel down.

So Linux ships all the drivers for all the architecture in one big tarball. We that's 40 million lines of code. Other operating systems try and go out there and um had split things off. So the core of Windows is their kernel and then you can put drivers additional on the top. Um we tie everything together in one big giant blob. Theirs is still monolithic. any driver and theirs will can crash the kernel within reason. Um, in that way we can refactor the way the interfaces between drivers and the kernel are. Uh, Linux drivers are on average one-third smaller than other operating system drivers because we can see the commonalities if you send oh three different drivers for three kind of same hardware. Well, let's combine them all make it smaller and refactor things and make it easier and oh, let's change this API. And this has to do with the open source approach, right?

that you see it like so we have we see all this common code and we can refactor it and we can make it better and cleaner and we're not tied to any fixed interface. Our fixed interface is between the user space and the kernel. We will not break that. That's our guarantee. We've guaranteed it for a long long time. And so we always want you to be able to upgrade your kernel and not feel worried that your old programs are going to crash. So you should always be able to upgrade. That's our guarantee to you. If it does break then it's our fault and we'll regress. There are some exceptions. There's some gray areas. There's some really low-level parts between the user space and kernel that we kind of work around and we argue about these all the time, but we never try and break user space on purpose. That's our number. That's our only really rule of of kernel development. Don't break user space on purpose.

Can we look at a specific example of how development actually flows through uh with a specific patch? Before I show a specific example, so say so we had 4,000 developers last year. So, they make a change. So those 4,000 developers will send an email to a maintainer and a maintainer maintains a subset of the kernel. Every part of the kernel is owned by somebody and then you are one of these maintainer. So then yeah I maintain some drivers and things like that but then those maintainers send things off up the tree to a subsystem maintainer. So like USB serial then will get sent to USB and then USB will go to to Lenus. So it's kind of a pyramid scheme that way.

So here is a change. So this was uh written by somebody named Chester. um he made a change to the USB serial driver. Uh it's an option. The chip is called option. Um these are a USB to serial devices. Um they're in modems. They're in lots of different things. There's a ton of different ones and there's no standard for these types of devices. So you had to add a custom device ID for every single one that you want to use. It's just the way they work.

Um so here's a patch and here's the description of it. Um this is just an email. The description here is there's the subject line. Yeah. USB serial option adding whatever adding that device and then here's so good part about uh hardest part about writing a kernel change is the description of what's going on really yeah I mean the the code is easy it's the description explaining it is hard right you don't explain usually what it's doing you have to explain why you're doing it.

for something as simple as this it's like it's really easy so this person says uh this driver is part of a cat 6 modem uh the product ID is shared across multiple modems it gives some a a little dump of what it looks like in the device. And then there's some more information. There's a signed off by line. And signed off by is what we created a long time ago. Um that shows that I have the proper authorship of this and ownership and I give it to this project under the license by which the project is run. So it's saying I'm licensed this thing under the GPL.

And then way down below is probably the oneline patch. This is all just so this is all the description. This person is giving context on like here's what needs to do about the model the different you know specifications or what and here's the change. So somebody changed this removed that line the red is removed the green is add. Yeah. So somebody added and had to reformat the lines based on some new ones they added. Um and that's it. So they add a few new device IDs and then there's a device ID and then we see a hex a few hex numbers. Those are like some IDs here. So for USB um USB devices have a product and a uh vendor and a product ID. That's how they're vendor and then they have products and then there's some subvice and subproduct IDs. Got it. Got it. So that's what this is for. Okay. So they're saying we're just adding support for some the driver already works for these chips this chip but we just have a new ID because a new vendor came along and they wanted to put their own vendor ID on it. Very common.

So it sounds like this change is as simple as it get in terms of the code changed but still the description was very extensive right so very des extensive. This should be in the patch itself, not the PR. Ah, so a PR would be so say you have 10 changes you want to make. So a PR would be the patch zero out of 10. This this is the commit. This is the commit itself. Which is a big problem of why I don't like the GitHub model is because people don't put the changes in the GitHub in the git commit. No, because the git repo Well, no. So there's a problem when you commit the when you're looking at the repo later and you look at the change, you don't see the pull, you can't see the pull request information. Yeah. And it's gone. And that's a big problem I feel with the GitHub model.

So, he sent it to the Yeah. So, the owner to this is Johan. Uh there's a script we have that says take any patch and give me the people who are responsible for this and the mailing list. So Johan and me um picked this because us and sends it to the USB list and copies a bunch of other people that I guess they worked with and that have changed this driver in the past and and then the the copy is also done with the tool as you said. It kind of looks to who who touched this code or or who might all automated. Yep. All automated. He sent it and this the mailing list has two copies of it. That's just because it went to two different mailing lists. But then they said, um, oops, I messed up. There's an email from the person instantly after he sent it off. Oh, and said, I I messed up. So they go and change the comment and then they resend it and you just send a new version. So here they sent a version two patch. If you can see that it says version two, right? Oh, there. Version two. Got it. And then here's the same information. And then there should be some comments about what changed between the two versions. Yes. Changes in version two from the previous one. And there's a link back to the first one. Nice. Very nice.

First they said um oh he Chester wrote hey please please please apply this after two weeks after a week he said because after a week or after two weeks it's nice hey what happened to this what's going on um Johan said you submit this patch during the merge window um I'll talk about how we do our development model but there's a twoe merge window for when we do releases that Lenus takes all the changes from all the maintainers that have been in their development trees we can't add new changes at that point in time so there's a two kind of blackout for new development, but this is where all the stuff is flowing into Lenus for the next release.

So then Johan has it. Yeah. Johan applied it to his tree because he then wrote saying, "Hey," and Johan is very nice here. He said, "You kind of didn't do the comments in the proper format. I fixed it up for you." Oh, nice. So for driveby changes like that, we want to make it really easy and make it We're not mean people. I I mean I mean I mean clearly this this feels like it's a person who is unlikely to become a regular contributor. They're getting their work done, right? They're they're adding they have a device that they have to ship. Yeah, pretty much. But we want to be friendly and open and easy to everybody because everybody submits their first patch at one time, right?

part of the hierarchy is the human aspect. So I if I take code from you as a maintainer um I'm now responsible for it because my name's on it. So, if it's a simple one-off or it's a simple driver that nobody cares about except you, great. I know you're the only one that's going to be affected by it, it's fine. But if it's the core part of the kernel and I take changes from you now, I'm responsible for it if you disappear. So, I have to trust that either you're going to be here or that I understand it good enough that I can maintain it.

So, part of Linux development trust or issue or model is trust. And it's trust in human interaction. like I will take stuff from people if they whatever they send me cuz I trust not that they got it right but they'll be there to fix it when they get it wrong because we all get it wrong and that's that's the part so that's that's the trust model we have and that we've been burned in the past by some major features were landed in the networking core subsystem a long time ago and then once they landed and were merged and taken the email address behind it disappeared and the network developers had to took six months to unwind the mess.

Johan then makes a pull request to me. Mhm. So this is an output of the get make pull request. I don't know what the actual command and it this is what a pull request from get and this is because Johan is a subsystem maintainer and it maintains the USB to serial drivers. There's a bunch of drivers for this types of things. And then he sends it off to me the USB maintainer. Got it. and he says take this patch or pull from this tree at this tag and it's a signed tag. So it's signed with his GBG key so I can verify that it's really him. Yeah. When I pull from it.

But he sent me a pull request and I a pull request is that I don't actually review the changes in it. I'm not reviewing each individual patch through email. I'm trusting that he sent me four patches here and that they're good. Yeah. And I have known Yan and I know that he will be there if something goes wrong. Yeah. Some subsystems that I don't necessarily trust as well. Um, I will make you send them an email and I'll actually review them and then I'll review them. And then when I review them, I add my signed off by to it.

So Johan sends it to me and then I take it and I put it in my tree. I pulled it and pushed it out. Yes. And there's my email that says that. So now I'm responsible. It's in my tree. Yeah. So now the um since this is a device ID, these can go to Lenus at any time. We can add bug fixes or new device IDs. So then a few days later, I send this change off to Lenus. So I send Lenus. I said, "Hey, Lenus, take all these following changes, these changes, and here's a whole bunch of USB fixes." So, I summarize it all. I say, "These are all the things in here." And these are going to be like a few dozen of of patches, something like that. And but here's the list of the patches down below. And here's of them. Here's the diff of them to make sure that this diff matches what he pulls from. This is signed with Mikey. Lenus takes this and he puts in his in his tree. So then it got picked up. All this whole process took about a week and a half.

So Lenus does a release every nine weeks. Yep. Bug fixes come in during those nine weeks or the last release. You're running the last release, right? You want those bug fixes. A long time ago, we realize that people don't want to wait 8 weeks. So let's create a model of we have a development tree and we have a stable tree. So when Lenus does a release, I fork off Lenus' branch and I say this is a stable branch. So if 6.4 four. I do 6.4.1.2.3.4.5.

I do a release every week and what I do is the patches have to be in Lenus' tree first. We can't diverge. So if it's in Lenus' tree first and a bug fix and it meets this criteria, I put in the stable tree and I do a release. And so we do new releases every week for that. When Lenus does a new release, then I throw that stable tree away and I make a new stable tree.

People want to make a device. You want to make it something that's going to last a long time. So, what we come up with the idea is long-term stable trees. And there I pick one kernel a year and I maintain it for to start with two years, sometimes six years. So your Android phone is running off a kernel that's five years old, but it's still getting bug fixes back to it. So I I maintain like four st long-term stable trees at the same time and we backport all these fixes to all the different branches and then we pick one a year and we maintain these.

So, Linux has almost always been paid to be worked on. So, I started keeping the numbers back in what 2006 or something. And at that point in time, 80% of the people that contributed were being paid to do it full-time for their employer. And their employers want people who know how to do Linux because they want to solve their problems. They want Linux to It's much cheaper to pay a few engineers to add a few new features than it is to write your own operating system. That's the beauty of Linux.

So, we've been doing it. And the joke used to be you get three changes into the kernel, you get a job. It's not really a joke. Um, as long as they aren't spelling fixes, but um some people do spelling fixes, which is great. We have people that do janitorial work to the colonel. They sweep the tree for common problems and they just clean stuff up and keep code alive and keep make sure it's fresh proper coding style. We have coding style issues. We have people just fixing spelling mistakes and comments which is great because you got to start somewhere. In fact, spelling mistakes and comments is a great place to start because it it makes you get the workflow down. You figure out how to make a patch. You figure out how to send an email.

you're now a full-time fulltime kernel maintainer. What does your kind of dayto-day or week to week look like? Cuz I'm I'm going to assume it's it's going to be a little bit different than most developers who, you know, like write code, review code, do those kind of things. So, yeah. I mean, I been working for Linux Foundation for what, 13 years now. Or if you're a maintainer, a maintainer is, and the networking maintainer said this the best. Um, we're like editors. We used to be a writer. All we do is critique other people's stuff now. But because we're a writer, we have a little side project. So, we do have little things that we do dabble in stuff.

So, my day-to-day is I read other people's stuff. Like I said, I get a thousand emails a day to do something. You're not excitating. No. Yeah. You you get a thousand emails a day. It's Wow. So I don't have a lot of it just file off and I do and it's like oh this this like I subscribe to a number of kernel subsystem mailing lists to see what's going on. Yeah. And I don't have to do something with all of those. Yeah. But some of them you need to do something. Yeah. Some of these I do need to do something with and some like so I'll so say for USB is one of the subsystems I retain. I showed them all off to a mailbox and then once a week I'll go through them all and say okay let's review all these. And so I'll look at my inbox, I'll have 200 USB emails to patches to go through and other people review them and other stuff like that.

we don't have grand proposals sent to the colonel list. We don't say, "Hey, wouldn't it be great if you did this?" I don't want to see that. I want to see code that works. And I love it. As proof, then code that works matters is because um you've taken the time, you've proved that this can be done. Yeah. Now, not necessarily that it's done right or done the best way, but it could be done. Yeah. And that's now you have the skin in the game, and now I'm willing to work with you and let's go on that.

So we have 25,000 lines of Rust in the kernel already. Oh, we do. Okay, awesome. Yeah. Um so most of that is just bindings. There's no real functionality. Um in the latest release, um if the kernel crashes, it'll put up a QR code. You can take a picture of it to get the crash dump. That code was written in Rest. Oh, nice. Um that's in Rest.

A couple years ago, they came to us and said, "We think we're ready to do this. Do you want it?" And we said, "Yeah, let's try this experiment. You're willing to do the work? Who am I to tell no to?" Um, I mean, it's Linux. Yeah. I mean, it's it's it's now the problem with Linux and Rust is it would be easier to write a core piece of Linux and Rust than it would be to write a driver. A driver is consumed from everywhere in the kernel. So you want to talk locking, you want to talk input and output, you want to talk talk to the driver model, talk to the USB port, all this stuff. Drivers have to can be really tiny because they take resources from the rest of the kernel. In Rust, you need to have a binding between the C code and the Rust code.

There's an intermediate layer. The C the kernel in C has these very opinionated model ideas of how it handles objects and how it does memory and how it it has its memory model. Rust has its very opinionated model of how it does this type. Same idea. This meshing is tough. This meshing is also the most crazy complex Rust code you've ever seen. So from a new Rust developer like me, I can barely read the bindings, but I trust other people are doing it. So yes, so the trick is we now need to write a binding for every different part of the kernel in order to write a rest code scope, a rush driver.

So I've seen every single kernel bug for the past 18 years. Half of them will be fixed with Rust. It's just it's just going to be fixed with Rust. It's the stupid oneoff bugs. It's the I oops, I overwrote an array and I didn't realize it by one. Oops, I um forgot to clean up this error path. Yeah, I forgot to unlock this lock. It's I It's stupid little things like that. There's logic bugs. Of course, you can write logic bugs in Rust. You'll always have those, right? But I think yes I think Rust needs to come in because it should be easier to write drivers in this stuff. We have a lot of issues with lifetime rules of when you yank out a device. Devices are dynamic and dealing with these reference counting of things like that is very tricky to get right. Rust is forcing us to actually document our C code better and it's cleaning up. So if Rust disappeared tomorrow, I've had to clean up code in the driver core that's like, oh yeah, I guess we can do things better and safer in the C code in order to make Rust easier.

So I think we've hit the tipping point where you'll start seeing new stuff in there and we need to do that. I mean there's mandates from governments that you can't use memory unsafe languages like C and products. Yeah. And if I want to see Linux to succeed, which I do, we're going to have to change. And I can say going forward, if you want to write in rest, you can write in rest. Now, that being said, we still have 40 million lines of C code.

And I used to have I had a manager at IBM every year come to me and said, "Hey, is Linux done yet?" I was like, "No." It took me 10 years to finally come up with the answer of um it'll be done when you stop making new hardware. And when they stop making new hardware or having different work classes, then we'll stop. But we're one of the few projects that keep having to add new features because of new hardware. We're not doing it just because I mean Lin has been working for all of us for 20 years. We're doing it to support new hardware to support new use models to support things. We don't add things for fun generally. We add it to solve a problem that somebody had.

will LLMs have any impact on on how development is done? No, no, not there's not. I mean, they're all trained on Linux kernel code. So, you write out another driver, but LMS are great for writing um boilerplate code and things like that. In Linux drivers, you don't have much boilerplate code because we've stemmed that down into the core and made that work better. Um LLMs are used to find bugs and find the bugs fixes to match that we should be taking. So, we but again, we've have published papers on that for eight years. Um there's been lots of research on that. Um, so we we've been using that for a while.

What's the most memorable patch that you've contributed to in Linux? So, this is going to be about people again. early 2000s um we were starting to get Microsoft was saying Linux is a cancer. We're all worried about Linux. Um we started getting some really really good patches for some hardware that we really didn't know that well that was showing some really good stuff and it was like this is really good and we're like where did you get this information? How did you know this stuff? And the person wrote back and said here's how I found this. Here's how I tested it. How they did this? like okay all right we took this and over time we took all these patches over time and then we have this conference once a year for all the maintainers and you get invited to it and we're like oh let's give them an invite because it was really good and they came and they showed up and he he showed up and he's like a sorry I had to bring my mom because he was in high school he was 17 years old none of us knew and he contributed and it was like okay great and it turns out um he later went on to MIT And now he's a professor at Stanford. Wow. Yeah. All you see is an email address.

What's your favorite programming language? It's still C. Still C. I mean I've been doing C for what 30 years every day. So yeah C. Yeah.

what's a book or two that you would recommend reading? The old code complete book was a really good one. That was a really good one. It taught me that um coding style matters. It doesn't matter what the coding style is. It's just a a spe a generic a set coding style matters because our brains work on patterns and as programmers we're reading patterns and when the patterns the same the metadata goes away and we can see the logic easier. Code complete is aged a little bit weirdly. If you look at the first book it has a lot more C examples and whatnot but it it talks about the basics behind programming all that stuff and that was a really really good book. Um, on the flip side, another really fun one, um, that's really tiny, programming pearls and like bit fiddling and, um, cute little algorithms and neat stuff like that, which surprisingly we still do today.