Chris Lattner created some of the most influential programming language and compiler technologies of the past 20 years. He's the creator of LLVM, used by languages like Swift, Rust, and C++. He created the Swift programming language, worked on TensorFlow, and now works on the Mojo programming language.

In this conversation, we cover the original story of LLVM and how Chris managed to convince Apple to move all major Apple dev tools over to support this new technology, how Chris created Swift at Apple, including how he worked on this new language in secrecy for a year and a half, and why Mojo is a language Chris expects to help build efficient AI programs easier and faster, how Chris uses AI tools, and what productivity improvements he sees as a very experienced programmer, and many more.

If you'd like to understand how a truly standout software engineer like Chris thinks and gets things done, and how he's designing a language that could be a very important part of AI engineering, then this episode is for you.

So, Chris, welcome to the podcast.

Well, thank you for having me. It is so nice to meet you in person because I feel like the work that you did has had an impact personally on me at Adooper. We migrated to Swift and of course a lot of the software we use runs on things that you've built. So can we rewind time back all the way to the early 2000s and for some of us who have not really been around there can you describe like what was the industry like in terms of compilers, languages and what was the status quo back then?

Yeah. So I mean I was involved in the kind of early days of Linux and so I started maybe not the really early days but I started working with Linux in the mid '90s and this is when Java was coming on the scene and a lot of things were changing in the industry as that played out. GCC was the compiler that standardized a lot of software and a lot of people were using it to build Linux based software and this is how I got to know it. GCC is a great thing. Before GCC, a lot of people don't know this, every hardware vendor was making their own compiler and it was a gigantic mess for even just C code because none of these compilers were compatible with each other.

And the reason they did that, just so I understand, is like you have C code, it needed to translate to assembly for a specific hardware and then the hardware vendor, you know, they did the mapping and figuring out like what custom instructions or something like that. Was that a reason?

Yeah. So basically back in particularly like the late 80s and then the early 90s there was a lot more diversity in terms of like instruction sets and chip makers were innovating and you had HP building things and you had Intel was making a lot different systems back then and there's all kinds of different stuff going on and so you had risk computers that were being invented and so the challenge at the time was that everybody had to build their own compiler and so because everybody had to build their own compiler they all wanted C and some C++ was emerging and C was a standard and so there was a spec that said this is what the code is supposed to look like but as we know for many standards and many specifications they're never complete and so what ended up happening is each of those compilers was a gigantic mess because they had different bugs they had different misfeatures they lacked certain capabilities and so software of the day was built with systems like autoconf which was this really weird macro processing thingy that tried to work around some of the limitations and it was a gigantic nightmare.

So GCC came out on the scene kind of in the '90s really and cleaned up that mess. It became the standardized thing that people could plug into and a lot of chipmakers embraced it and it was free software and so that led to the rise of Linux. Linux adopted it and that really kind of brought the world forward. Now GCC was a good thing. It really did help standardize the world and I think a lot of free software and open source pays a debt of gratitude towards GCC but also it was a really old thing and so I made fun of it at the time. It was over 20 years old and its architecture was, you know, very very old school in many different ways. It was also not built to be a modular design. It had, you know, was designed to do one thing, take C code in and then put out assembly code for a given chip. And so there's a lot of things you couldn't do like JIT compilation and you couldn't at the time even optimize across files within one.

And JIT is just in time compilation, right?

Just in time compilation. Exactly. And so there's a lot of things that people wanted to do that you couldn't do with GCC. And so in around 2000 that's when I was in university and I was studying compilers and I said oh okay well wouldn't it be interesting to build something new in the space and so I started working on LLVM. LLVM is a it started out as a code generation system so you could target multiple different architectures but really for me it was a learning process it was about saying okay compilers are cool don't let anybody tell you otherwise compilers are cool.

Even today right.

Even today. And so, but I didn't really understand anything about it and so I wanted to learn by building and so across my university project I built this thing up more and more and more. We then open sourced it. It got a little bit of a community. Later I went to Apple which really helped foster and fund a lot of the development and that early starting point was coming from you know GCC and open source technologies are really good but there's a lot of things we can't do. And so that's where I kind of fell down this rabbit hole.

And like when you started and and open source what did you open source? What could it do? Yeah. And then you know what was the reaction to to this early version of LLVM?

Oh, so LLVM back in the day and this is super funny because many people look back on successful systems and they assume that everything was obvious when when actually every step along the way is challenging and you have to earn any success you get. Very very rarely do you like luck into it. And so when I first started working at LLVM again it was a research project at a university. There's a lot of research projects at a university that don't go anywhere.

I'd argue most of them probably won't go anywhere.

Exactly. And so when and the default assumption is that LLVM also would not go anywhere. That's a safe assumption for any project. And so we used it internally. And so my adviser Vicram Adve encouraged us to continue building it and then we taught it to a couple of classes. And so we had a few people using it internally and we got some like use case with it. But at that point it was really just optimization and code generation. And so it plugged into GCC used the parser and the to actually parse C code for example and so it was just about code generation and it was useful for compiler people trying to learn things but it wasn't very useful for general application developers really.

When we open sourced it then we got I don't know two or three people that were interested in it and it was mostly other compiler nerds that you know are delightful but there was no major community there was no major reason for people to contribute and what I did was I said okay well actually just treat the world like the rest of the research group and just have open development, encourage people if they they come by and they want to help and do something awesome. If they have questions, I'll answer them and just treat people with respect. And what happened over time is slowly the community grew. We got in individual people that were interested in different things. Some people were interested in esoteric programming languages and and they were interested in compiler technology for that reason. other people were interested in performance and different people had different interests and so very very very slowly it kind of grew.

What was the big difference between GCC and LLVM right like of course modularity was one thing but but also you mentioned capabilities that you wanted to do that GCC was just either couldn't do or was really difficult to do what were those capabilities?

Yeah so so originally it was just in time compilation so runtime code generation.

That was a thing with GC couldn't do they could do only compilation.

They could only do a kind of batch upfront traditional Unix style code generation. At the time, GCC could not optimize across files in your project. And so if you had a C, if you had a function in one C file, it could not inline it into another C file. So things like that. LLVM was also written in C++, which was highly controversial at the time because...

GCC was all C and uh, Richard Stallman was very opposed to C++. And so actually there's a parallel universe because in 2005 I had joined Apple. We decided okay well let's see if maybe LLVM and GCC should merge and actually proposed to the GCC community hey we've got this interesting technology this seems very complimentary it could lift the architecture of GCC maybe we should do this and uh it didn't go anywhere because primarily was written in C++ but it was also not invented here there's a bunch of other you know very serious very experienced people and they're like okay kid what do you why would we listen to you and so it did not go anywhere and thankfully so um it made meant that LLVM had to grow up on its own.

And then did LLVM really start to take off when you got to move to Apple to now work on it full-time? I assume there must have been a at least a small team like investing in this. Is that how it went or something?

Yeah. So, so the story of getting to Apple. So, I when I was graduating, I was looking for jobs in compilers and jobs that ideally would let me work on LLVM and continue the work because it was still an advanced research project. It was 5 years in, the community grown quite a bit. I was doing regular releases every six months and so was really putting a lot of energy into advocating for the work and showing momentum and velocity. I caught the eye of a VP at Apple and there was a couple of people that were working on adding a Power PC back end to LLVM because that's what Apple was doing. And so I got to know them and they said, "Hey, come like we are frustrated with GCC. That is the foundation of all of our all of our compiler technology."

Because GCC compiled Objective C for example. Right.

That's right. Exactly. And so and it was again an older architecture. It was very difficult to work with. They weren't getting the performance they wanted. It was very difficult to add features. The community was also kind of annoyed with Apple for a variety of reasons. And so basically management said, "Come like work on LVM." And I said, "Yes, sign me up." And so when I started they did give me one other engineer to work with, but really they didn't give me a whole lot of guidance. And so I was like, "Okay."

You pretty much...

Cool. I will I will I will go implement PowerPC support and integrate with the Mac OS back in the day and things like this. And then there became a time when they said, "Okay, cool. You're working on this." And my manager said, "It's great that you're working on this cool technology, but at some point we need to make sure that it's actually relevant to Apple."

Was it not relevant?

Well, it wasn't used in any products yet.

Oh, I see you added support, but it wasn't used just yet.

Just yet. Yeah, exactly. And it wasn't as good as GCC in a number of different ways. And so I basically got the vibe that it's like, okay, well after a year if Apple's not using some product, then we'll ask you to start doing something else cuz we're not just going to pay you to work on an open source project actually have impact on our company. And so when I got that memo suddenly and my manager helped me a lot, he was amazing. Went around shopping for okay, well, what are the the near-term impact that we could have on the business?

And so from there, the first use case was actually for graphics. And so OpenGL was doing just in time compilation to do some graphics stuff. And so we were able to do something very small that actually had value. And suddenly aha this is not just a science project. This is actually interesting. Now it's still missing tons of features. But that enabled the development of the next set of features, the next set of features, the next set of features.

And it was now in a product that Apple actually used and shipped and it was you know generating business value if you want to put it like that.

A small amount but at least non zero.

Non zero.

Exactly. And so and so what I did over the course of many years across Apple is then say okay cool we'll keep investing the technology keep building an open technology and an open team so community but make sure to deliver more and more and more and more value to Apple and as that happened suddenly what ended up happening is I end up replacing all of the developer tools compiler code generation debugger technologies within Apple and that...

One one step at a time.

Exactly. And so over the it took about 5 years to the point where we built a new C++ parser clang and Objective C front end and all this kind of stuff. We got to about 2010 that's when Apple was releasing the first 64-bit iPhone. And suddenly this was made possible by LLVM by all the technology we had built and all this kind of stuff. And it was a it was a epic moment in the industry because everybody is convinced that 64-bit phones were stupid. That was not a thing. and 64 bits doesn't make sense. Why are you going to have more than 4 GB of RAM in a phone? Right. Hilarious.

Wow. Back in the day. But that's how people thought, right?

That's how they thought. And they're like, "It's inefficient. It can never work." And so it actually the first 64-bit iPhone, it was the iPhone 5S shipped in production before ARM got their chips back to test. And so we had built the entire software stack. We enabled the entire operating system, the entire tool chain, all this kind of stuff internally. And we were collaborating with ARM. But they had no idea how far ahead we were. And then suddenly we're shipping it in production and the whole world's heads explode because how good the performance was and how all the capabilities we enabled and it was it was quite fun.

So I'm having just trouble putting two things together and you can help me out with with what I'm missing on on one end you know you made it seem like okay like you know initially when you got to Apple for a year it was a experimental project then you just went slowly with one team and after the other but now next thing we know fast forward to four or five years later it's actually powering like like the core of Apple. How did it go from like you know the small projects to actually getting into like the core of of the business? Did it help that you started with developer tools and more developers were convinced like how did you get through to you know like probably the highest level kind of architects and decision makers who in the end probably I assume they had to take a risk on technology that was like somewhat new compared to like at least GCC that's been around for like 25 years at this point.

Well, so it's a combination of having top down support. So people that wanted me to be successful and that's because they're frustrated with GCC and so they wanted a new technology. So that was very helpful. A combination of bottom-up success and it wasn't one thing. It was like every 6 months we'd have a new thing that worked well and and go. I became known as somebody who'd get things done and so I got more scope and responsibility and so my you know I went from being an engineer to a manager to a second line manager to eventually a senior director over the course of number of years.

Um, and so it wasn't one thing. It was just a lot of hard work and it was a lot of fun because we were able to, you know, a great thing about Apple back in that time was that you could have a lot of impact because the team was growing, but also if you could get stuff done, you had an amazing platform in which to work because the iPhone came into existence and a whole bunch of other technologies were made possible. And Objective C was very Apple specific, but we could move it. That was huge. We added a whole bunch of features to Objective C. Um and so as we built into this like there's a lot of opportunity but it was hard one right it wasn't in in any given moment it wasn't guaranteed but as my team at Apple was making progress we were doing all this in the open then suddenly other people started seeing value and like Cray built a supercomputer and used LLVM for it and Google eventually adopted it 2010 and started using it within Google for some small things and then kind of a parallel project within Google of another hero within Google like built a team and add a lot of impact to Google. Therefore, justifying more people to work on LLVM within Google and different tooling and different projects and different things that all happened eventually, you know, Intel or ARM or these companies ended up canceling their internal compilers and switching to LLVM.

And so then suddenly the vast majority of their engineering teams all working on a thing and then what you get is you get value that compounds and you get a today LLVM has a community of thousands of people that come together and so it's amazing to see that.

It's incredible. It it it feels a little bit like when you're starting to roll snow falls and you're starting to roll this small thing and then you know you do it for long enough and most people don't do it but then you you could have a snowman and you're like oh where did you get all that snow from? Just incredible.

Well, so I mean it's funny because people always imagine success, right? And so you you start from a starting point and you say I want to be the president of the United States or something, right? But but and some people get very motivated by that and it's very powerful. Uh what I really love is doing all the work, taking each of the steps, doing the thankless thing, figuring out that really obscure thing that you know if you get the architecture right, it makes the whole thing compound and scale and compose the right way and it allows you to solve problems that nobody else can solve. But what that means is most of the time I'm working on things people don't understand. And so I'm fine with that. I've normalized to this. This is what I expect at this point. I don't expect people to understand me. But what happens is that these projects over time they grow and they get to the point where suddenly it clicks and suddenly people start to understand and suddenly you can explain it because it maps into their value system in a way that you know they can measure.

And and with them, let's get to Swift. The story of Swift as you as you shared many times is how you started to work on it in secret. Can you tell me like what led you to start experimenting with a new language? what you saw the problems being with Objective C especially because now we solved the compiler problem like you know the compilation was probably as as good as it it could have gone and how did you pull off this kind of what is secret mean right?

Yeah well so so the backstory on Swift is that uh so I joined Apple in 2005 and I built and basically replumbed their entire C++ and C and Objective C tool chain and by 2010 at WWDC which is their summer conference um we had launched a full stack replacement and it is working and building a full stack fully integrated C++ compiler is no small feat. C++ was even then was a very complicated language and it was a very big technical challenge but it was also very demotivating because C++ at least to me because C++ is just a beast of a language and so you couldn't come out of that and not wonder could we do something better right and as you said we built a lot of this fundamental technology could build pretty much anything we wanted at that point.

And so I just started again nights and weekends didn't ask permission just started fiddling around seeing what could be done that would be better and to me what I did was I actually said okay let's go look at a lot of existing languages so let's go look at both the new ones um let's look at Java or let's look at C++ things but let's also look at functional languages like OCaml and Haskell and things like this let's also look at esoteric languages like the the really weird other things let's look at TypeScript and Dart and other things that were kind of on the scene and happening and Go and languages like that and so it was really about saying like well what do I like and how do I see success?

And what I defined success as with Swift was making it easy to use and scalable. And so I saw a lot of languages JavaScript is probably the best example of this where you know you start with something really simple and you know a weekend hack with JavaScript and then and then it gets adoption and then you get developers and then developers bring their tools to other domains and so you know JavaScript was designed for onclick handlers but now people are running web servers in it.

Yep. Well, and so we see it at all all too much.

Exactly. And so what my my observation was is that success meant that you would be naturally brought into other domains and so think beyond just the the first win but think towards more generality and scale. And what I realized is if you take something like Objective C which I was deeply familiar with for example you know this dichotomy between objects and C. So Objective C had this small talk inspired object model and then it had C for performance. It was actually a really beautiful combination because you can get all the way down to the metal with C. Uh, but you had high level APIs. And what I realized is that they didn't have to be two languages.

So, so you could you could take the good parts.

And pull it together into one system that could actually scale and would be way easier to teach, way more memory safe, way more modern, good type inference, fix some of the syntax problems because Objective C was a little bit of a a mashup between two different worlds. And the funny thing about that is that you fast forward all the way to today. That's exactly what Python is. Python is a beautiful language for objects built on top of C and C++ and Rust and it's got this two world problem going on within it. And so it's very interesting to me just again how history rhymes with itself and there's so many different things going on. But the initial the initial ideas of swift was really about saying how do we build a scalable language and how do we pull together the best ideas shamelessly you know borrowing ideas from different places.

When you have a black canvas or when you were thinking like how do you go about it do you kind of like I don't a meta program like kind and write down what you love to see. Uh you also of course built compilers you start to immediately think of of how that would work like can you give us a little bit of a a view of like what what how do you go from this blank canvas to like okay here's what I think the language will be.

Well so that's a good question for me in general because I start from a lot of blank canvases across my career right so LVM blank canvas, MLIR later blank canvas, mojo swift like many of the systems I build are blank canvas projects and so for me it's it's a couple of things first of all doing that kind of analysis the reflection the deciding what is success ultimately look like? Then do the survey. Go look at what's out there. What's good? What's bad? One of the things I really strongly believe is that you can take a a system that you don't like, but buried within a system, even if you don't like it, there are often good ideas. And so you can take the ideas that are good and ignore all the rest.

And then when I start building a new system I expect to design and redesign and iterate and learn and grow. And what I try to do is I try to go through proof points and validate specific assumptions and say okay I'm not going to solve all the world's problems. I'll keep this part and this part and this part super simple. It's like you know the two circles that make your owl just leave them as circles and then uh and then invest in proving this piece and if I can get this piece done and I can understand that it works and I can gain validation and confidence in that. I can know the interfaces going in and coming out of it well now I can go out into the next system and build that out and build this thing out so that we can understand it and learn as we go. but every step along the way being unafraid to go massively change and flip over the table and try try again.

And so you started to experiment with Swift uh like the kind of weekends and on the side in 2010ish. How did it take until 2014 for a first release like what really goes into building out a language? Because again I I think we can all I can kind of emphasize like this is cool like you have these ideas of course you have a lot of experience you try out things but then like as a as a software engineer I shouldn't ask this but I'm going to ask what took so long.

Oh well so it turns out building programming languages is not easy uh takes about four years I guess.

Can you give us a bit of like emphasis of like you know like what like makes it so hard?

Yeah. Well, so so let me let me just frame up how it went just so you know because uh the first year and a half it was literally just me working on it.

Oh, okay.

So in nights and weekends and I had a day job and I was managing a big team and a lot of stuff going on.

Hold on. So at this point, yeah, you were already like pretty second level manager or something and had a team of 40 people and running a lot of a lot of very interesting and very fun stuff.

This is passion project. Yeah. Yeah. I'm also a fairly good programmer so that helps. But after a year and a half got to the point where I told management about it. I said, "Hey, I'm working on this thing. What do you think?" They're like, "Uh, yeah. Uh, no. Why would we want a new language? Objective C is what made the iPhone successful."

Right. Right. And so I'm like, "No, what?" I'm like, "Well, how about I just have like one or two of the people in my team like work on this part-time?" They're like, "Uh, I guess you can tell them about it, but we can't commit resources really, you know?" And so, and so, but you know, again, I was respected and doing good things. And so I was given a little bit more rope than some other people would. And I got to the point of saying like, okay, let's build up some demos. And then it was about 3 years in and most of the feedback I was given was, okay, well, if you don't like Objective C, go make Objective C better. You own Objective C. Also,

It's a natural reaction. I mean, as as much as engineer, I don't want to hear it. If I put a little bit of a business hat on, it's what you would...

Makes total sense. That's what the entire community is using. It's very low risk. It makes your existing investment even bigger. And so what I did was I said, "Okay, well, uh, across that four-year journey was not just building the thing full out with like a team of people that knew what they're doing. That's the discovery, the research, and also the socialization process with executive leadership." And so, um, what I realized, I said, "Okay, well, I want to have memory safety. Objective C does not have memory safety."

That was one of the biggest things, retain and release, and it was completely manual, and it was just a kind of a nightmare. And so we invented ARC, so a way of doing reference counting automatically. That made Objective-C way more teachable and safer. It also pulled it much closer to Swift. We then did modules. We then did literals. And in Objective-C, we added all these features that what it was doing is was pulling the lived experience of doing Objective C programming closer to what Swift would be. So that ultimately when Swift came out, it would be less of an abrupt leap and it was still a pretty big leap.

1.0 you know came out and you know the feedback I can just talk from you know like what what I remember from there there's two things people were super excited that Apple is releasing a new language and iPhone was was huge 2014 was was the time where like you knew that you could make a build a big business on iPhone. However, when you tried it out, it was just not good. Xcode didn't have refactoring support. Debugging was a nightmare. We should launch.

[laughter] Yeah. So it had to be 1.0 to launch it because Apple doesn't launch 0.5s basically. So it if we wanted to launch it had to be called 1.0. So that was one aspect of it. But the other aspect is we wanted to get it out there. And at the point that we launched it, only about 250 people at Apple knew about it at all. Most of those people were in my team and then some executives in marketing and other people like this. And so we knew we couldn't build a good programming language in in a vacuum with an NDA that you had to sign to get access to it, right? You have to actually get it out there. You have to get usage experience.

And so the learning for me is don't call something 1.0 until it's actually ready and it's validated and you have a lot of usage experience and things like this. We bring a lot of these lessons to Mojo by the way.

But also it's about making sure that we communicate clearly with the community. And so one of the things I think we did well is we said look uh internally to Apple we said look we we need to launch this we know it won't be perfect and so we're going to tell the community that you can adopt it and you can build apps and submit them to the store but we will break your source code.

Yep.

And this is because we know that we have to change the language when we get usage experience. And so we we told people like we will help you but but but expect code breakage. And I'm glad we did that.

I I remember that that was really clear. People were upset about it, but they knew like when we had the discussion of should we move over to Swift or not and uh at at Uber uh the rewrite happened in 2016 where Swift was at 1.2 and the the platform team made the decision after evaluation that they will take a risk and it was a big risk and there's now stories about it. I have a blog post there some other people with their experiences but they did it because they knew that Apple is committed. I think the open source part also helped build a lot of trust and I almost feel that that was one of the last times that I saw like the Apple community really unite against uh like iOS developers specifically.

Well well this was a fun fact in 2016 I remember middle of 2016 or maybe the a little bit earlier Uber was one of the big mobile first companies built on iOS grew up on iOS lived and breathe iOS and of course Android and the mobile platform team ran a survey in sometime early 2016, 2 years after Swift was out. This is Swift 1.2 saying we are planning to rewrite the app. The rewrite was a thing just because they needed needed to for architecture.

And they ran a survey would you prefer, you know, as an iOS engineer, Objective C or Swift? And the results were pretty much 50/50. So and these were again very experienced engineers. they they knew and the two camps were violently disagreeing with another and and I don't I still don't know why Uber went with uh with swift someone was a tiebreaker in the end but back then this is how foot on foot it was.

Well so but I've learned a lot and again you reflect forward because this impacts exactly what I'm working on now. same thing which is that...

With Mojo?

Experts don't like change.

This is counterintuitive.

Well I mean if you think about it like if you're if you're an expert objective C programmer I don't know if you were but if you're an expert Objective C programmer you've been doing it for five or 10 years you've learned all the baggage tricks all the weird cases you know all the arcane thing you are the person that people turn to for help comes out now your expertise is invalidated you're on day one just like everybody else you are not the expert You don't really want a new thing. You want to be the king of the hill of the old thing. And you don't want the new thing to be successful.

And so now some people are intellectually flexible and will switch over and there are early adopters and things like this or people that like new things. And so it's not true of everybody, but it is actually sensible human behavior to protect your your prior investment. And for a lot of people, you have to have a really good reason to switch over. And so in the case of Swift, you know, Swift UI was like a reason that moved some of the longtail people.

Now let me tell you the thing that I find most that I'm most proud of in this journey. So when I started 2010 and it was just a toy project like playing around with in spare time not expecting to go anywhere. I was kind of implicitly coming from the assumption that we could do something that was better than C++ and Objective C and that was more easy to use and things like this, but didn't really know how it would go. After we launched, I would have people that stopped me in the streets and said, "Thank you, Chris." Like, because of Swift and because of not just me, but the whole team working on this, I was able to become an app developer. Before Swift, I tried building apps with Objective C, but it I could never figure out the pointers and the square brackets and it would crash and I couldn't like I'm not smart enough to build an app.

But now with Swift, it became easy enough that even I could do it. And now I am an app developer instead of a web developer. And I've become the expert in my company. And now I have learned and grown in my career. And so now thanks to this new technology like I progressed. And if not for this enabling technology, it never would have happened.

And that's the exact same thing happening with GPUs. Right now we have the exact same thing with CUDA and C++ and all these tools that are gatekeeping so many developers away from this emerging kind of compute and kind of technology. And it's the same exact thing. And so this is what really makes me passionate about working on this kind of technology because I believe in the power of programmers. I believe in the human potential of people that want to create things. And that's fundamentally what why I love software is that you can create anything that you can imagine. And and to me, that's what's so powerful about working in this space.

Looking back at it, uh how do you feel on how the language has evolved? What what are parts that you're really kind of happy on how it lived up? What are parts where you're you're you're thinking that maybe if you would have stayed longer, maybe you would have like tried to have it in different ways, if there's any.

I can talk to the mistakes that I made, right? Because and they're learnings that I could bring forward and that way it's my fault and it's not me saying other people weren't able to do it. Right. Fair. I like that. But like early Swift, it was, you know, I'll just say it bluntly, I didn't know what I was doing, right? So I'd never done it before. I'd built a C++ compiler which had a spec.

Well, it was the first language you ever built.

It was the first language I had ever built, right? And so a lot of the ideas going into it were new. A lot of the like the type-checking algorithms were exciting, but I didn't really know what I was doing. I'm also not a math guy, and so if I actually understood math, then maybe it would have been obvious. But things in Swift like expression too complex to type check in reasonable amount of time. That's my fault, right? And there there's specific features that cause that to happen and it's very difficult to fix.

Another mistake is that we were very ambitious in terms of adding features that we wanted to see in the language and we added them faster than the architecture could keep up. And so particularly if you talk about Swift one, but even today what ended up happening is the internals of the compiler and the design didn't really line up right the right way. And so you get a lot of things that work kind of 80% or 90% and they didn't quite fit together and then as you start adding more and more and more stuff it doesn't really line up right and it just gets more and more complicated. C++ has the same thing going by the way. Swift's not unique this way but but as a consequence of that like you get an emergent complexity that then developers can feel and swift today people make fun of too many keywords and things like this. Well, it's because the origin story was really not, in my opinion, didn't put enough energy into like really refactoring, really redesigning, really making sure the core was great. Instead, it was okay, let's get to app development as quickly as possible. And so, as a consequence of that, what happened is then, you know, I and many other good people put more stuff onto it and more stuff onto it and more stuff onto it and the language got more complicated. And so, you know, partially that's maybe in any individual decision, but also it's because some of those original things didn't line up the right way and so the complexity kind of compounded.

So, as we get to what you're doing today, you're after Apple, you worked at a few interesting places, Tesla, Google Brain, SciFive. Can you talk about what you've learned at each of these places and kind of how it's, you know, because I think it'll be interesting to look back on how it just all is leading to what you're doing now.

Yeah, totally. So, so the way way I frame it up is that, you know, 1.0 of my career was at Apple and building developer tools and understanding CPUs and I built Open CL, which is a GPU thing also. and compilers for the GPU and like all this kind of stuff. And so learning a lot about that hardware software boundary and developers and developer tools and Xcode and like that whole thing, right? 2016 I fell in love with AI. And the reason was is that about 2016 the photos app had the ability to tell you is this a cat or is this a dog? And it was a very like it was inception v1 or some really ancient AI model was in there. I'm like I own developer tools. I know a lot about programming. I have no idea how to write an algorithm to detect a cat in a picture. How the heck does that work? So I fell down this rabbit hole in 2016 and this is what leads me to 2.0 of my journey which is a very AI focused like go figure all this stuff out.

And so Tesla is doing applied with self-driving cars. That's where I learned Caffe and tensorflow and I realized tensorflow was a good thing but it could be a way better thing and so that led me to joining Google and helping scale TPUs there. I ended up owning the bottom part of TensorFlow, all the CPU, GPU, TPU, plus a bunch of other weird internal ASICs uh within TensorFlow, but notably I scaled the software platform for TPUs and across years uh learned both, hey, it's actually really hard to make AI software without CUDA. TPUs don't have CUDA.

CUDA, which is Nvidia's uh way to program kernels, right?

Yep. Yep. So, so basically the entire AI ecosystem back then but also today ends up being very heavily biased by Nvidia and by their API and software and language toolkit called CUDA. It's about 20 years old and so and it's awesome and it's very mature and there's a lot of good things about it but it's also 20 years old and so there's time and space for new things but when you're building an ASIC so TPUs are very large large scale data center training and inference accelerator very fancy very very uh frontier particularly back in 2017 you don't have software and so you have to create everything from scratch and then you have to get TensorFlow and PyTorch to talk to it and and nobody really understood how that worked.

And so across the years I learned so much and I'm so thankful for my experience at Google because I learned about the algorithms, I learned about AI, the frontier applications like you know the attentions all you need paper was invented in that time frame on TPUs and so... The origin story of modular really came down to my co-founder and I who were buddies. as we got to know each other previously at Google, but really saying like, you know, we need something like LVM, but for AI.

But for AI.

Like we need the thing that people can implement support for their chip into it and then get an AI software stack because, you know, if you're building CPUs 20 years ago, you didn't want to have to build a whole C++ compiler and C compiler and code generator and kernel and web browser. You just want to do a little bit of work and then get software. And that doesn't exist for AI right now.

And then for for for those of us who are software engineers but we don't build AI software we we we just you know we use the models we we might invoke the API or we might invoke the trained one. Can you explain to us if you know we moved into the the world of actually building AI applications running them directly on on GPUs may that be Nvidia or now there's AMD or or Google TPUs. What software would we use today? like what is the status quo or what was at least the status quo when you started modular and then what is your vision which I I sense there's a bunch of similarities with like GCC and LLVM here.

Yep. Absolutely. So um so before GCC every chip maker had to build their own software say that's literally the shape of AI software today because there is no thing to plug into besides modular. Uh everybody has to build their own vertical software say and so when I was at Google we built a thing called XLA that's the thingy that we built kind of like CUDA but for Google TPUs.

If you look at Apple they have you know metal and MLX and uh their their whole stack. If you go look at AMD, they have Rockm, their whole stack. Everybody has to have their own verticalized stack and they share very low code.

Oh, so this is why I I I was just looking at uh I talked with some folks at Anthropic and look at their job sites and I saw that they're hiring both CUDA engineers but also Google TPU kernel engineers.

And Trinium which is the AWS.

And and Trinium and so they're writing the same models three times over and over again.

Depending on the hardware they do.

And so what we bet on was we bet on a two-level stack. We bet on Mojo a programming language, a brand new programming language. And as we all know, making a new programming language is doomed and like never works and you cannot get it adopted like all all those things.

Well, you have a good track record though.

Yeah, it's it's actually just a hard problem. But but you have to be smart about it and we can talk about that. And then um on top of that saying like let's take this programming language and design it so it can solve these problems and scale across hardware etc. But then take these compiler algorithms and put them into the language so that you as a mojo programmer get the power of compilers without having to be a compiler engineer. And so a lot of what makes Mojo really special is it takes power out of the compiler and gives it to you as a developer. Because it turns out a lot of people can write code, they can't necessarily write compilers. And so what this does is it breaks open the talent problem. And so now we can have a lot more people building these very highly adaptive, highly scalable, highly composable systems in ways that they couldn't do before.

And if if if I want to get started with Mojo outside of just the basics which are so like using some of these more advanced features, is is it the best way to actually like rent a may that be a GPU or or or or like some some these days is is pretty easy and and go and try to implement an algorithm a program that actually needs high performance and and start to play and understand these additional you kind of the ways that I can control a compiler or or what would your suggestion be for someone who It's like, okay, Leon believes that this is the future and this is like cool to to to experiment with on their job. They don't currently have a problem like this. How would you go about it on on the weekends, right?

Yeah. So, um so the I can talk about what makes Mojo cool. Mojo is a member of the Python family. So it's super familiar, easy to learn. And by the way, with the AI coding, it's easiest time ever to learn a programming language. Just like you mentioned, one of the simplest easy things to do with Mojo is to extend an existing Python module.

So if you ever been coding up Python, building and building building, it gets big and big and big and big and suddenly gets slow. Well, historically your option is to go rewrite a big chunk of it in Rust or C++ and then you have to use bindings and it's real kind of a pain in the butt to be able to do this kind of work. Mojo is the most beautiful way to extend Python you can imagine. No bindings just like Swift and Objective C. They they natively spoke to each other.

Oh yeah. No bindings because because Rust is is a popular choice to extend Python but with a binding.

Yeah. But you need bindings, right? And so it's very mechanical and you can get it wrong. And so Mojo does that without the bindings, but also integrated with uh pip packages and the build systems all integrated and it's just super beautiful and awesome and you can stay on CPU. You don't need anything. It's all free like go nuts and go do this.

One question I I've been meaning to ask uh because you built several languages, this comes up and and I asked folks like what is what are things they'd love to hear from you? And this this this was repeated quite a bit. Do you think there is any logic in having a new language that is designed to make it easy for LLMs to code with and like attributes for example better pattern matching or other things that might be the strength of of LLMs and if so like what would these patterns be? This is really weird like I I would I would have not thought we would talk about things like this but but now we should.

Absolutely. I'm happy to talk about it. And so I often sometimes get asked uh so given the AI is writing all the code, why are you building a programming language, right? Which is another way of asking the same question maybe a little bit more aggro. And so to me, I don't think that optimizing for the LLM is the right thing to do at all.

Because go back to what we were just talking about. The important thing is reading the code. Always has been, by the way, right? Code is read more often than it's written. AI has made it way easier to crank out the code than it ever has been. And I don't think we're going back to where we were. So writing the code is actually not the the key thing to me. It's about reading it.

And so what I care about with Mojo and with these new emerging systems we're building is is really a combination of two things. One is expressivity. And so can you express the full power of the hardware? Yes or no? Like if the answer is no, then you will never be able to get it. And so JavaScript will never be a good way to write a CUDA kernel or GPU kernel. it just will not because it can't express the things you need to express, right? No, no slide against it. It's a good system, right? But but if if that's the goal, then you have to be able to do that now.

Assembly code can express the full power of the hardware and that's not what I'm advocating for, right? The the other thing that goes with it is you need readability, right? And so you need this combination, this intersection between expressivity, the so can you express the important thing in our case performance um and then can you understand the code? Can you build abstractions? Can you build in scalable systems? And that intersection I think is the key thing. And so this is where you take Python syntax for example. Python's super easy to read. It's actually very standard. A lot of people know it. Let's embrace that. Let's go all in on that. That's a good thing. And so Mojo embraces Python and this the syntax of Python at least and says, "Okay, well now let's fix the problems with Python, which is basically the entire implementation. Let's replace the Python interpreter and all that kind of stuff. let's keep this good part and make it so we can express that full power of the hardware."

And with that, I think the LLMs will continue to get better and better at dealing with any weirdness. And what I've seen is that the LLMs are an amazing way to learn coding a new language.

Yeah. Now to close with you have mentioned that compilers are cool multiple times here.

I I'm slightly biased I admit.

Absolutely. But I I'm getting I'm getting convinced that compilers are cool. For software engineers who have not built or worked with compilers. What are some kind of like like pointers you would give them of like, hey, I'd like to get into like understanding compilers, maybe building one. Should I try to come up with a language? Um like what are some things that you point people to?

Yeah. So, so I'll tell you why why I fell in love with compilers. So, if I go back to university, I was taking the standard computer science program of uh basic programming and then data structures and then like an operating system class and guey class and all this kind of stuff. The thing I love about compilers is that you built project one. And so in this case, it was what's called the lexer, the thing that tokenizes the source code. Um, and you had to use some data structures. So I'm like, oh, okay, cool. I'm using the things I learned, not just how to write code, but what a what a tree is and things like this. And then you get to the second part and you build on top of it. And so now you build a parser, the thing that decides the syntax of the language using the tokenizer. And then you build the type checker. And you build on the first two. And then you build the next thing and you build on those things.

And so what I loved about compiler as a class was that I got to build and learn and iterate and then if I made a mistake I had to go back and fix it cuz you're building higher and higher and higher. And so I think that compilers particularly in the university setting reflects more of like real software development in a way that a lot of other classes I I experienced did not because a lot of the other classes ended up being build build a thing turn it in throw it away build another thing turn it in throw it away and so that's that's why I fell in love with it.

Uh today if you want to learn compilers it's easier than ever you can go to LLVM. There's a great tutorial called the kaleidoscope tutorial that I wrote wonderful 15 years ago or something. It's been a long time ago. The Rust community has a lot of compiler lovely compiler nerds in it and so a lot of compiler based technologies built in Rust these days which is also super cool. Um and so there's uh there's books and lessons and things like this. I don't literally think everybody needs to become a compiler engineer but I think that compilers don't get the credit they deserve and it turns out there's a lot of really good jobs in compilers and so if it happens to be that it's a interesting place and interesting set of technologies then I really do encourage people to do it because there we need more more folks in this field.

Wonderful. Chris, this has been really, really interesting. Learned a lot. Thank you.

Yeah. Well, thank you for having me. I hope it was useful.