Host: How do you think about Python ecosystem today? The Rust ecosystem and the Go ecosystem.

Armin: The Python ecosystem is a lot of infrastructure, a lot of provisioning machines. Rust, I think if you work with binary data, if you build a load balancer, you build a database. Go in particular, I think it's just a good language for building web services and really kind of only web services.

Host: Speaking about AI agent coding, how are you using them?

Armin: I had Claude build me my perfect control system to get my logs and visualize what's going on in production and I would never have done this before just because like it wouldn't have worked.

Host: Why have you become so much more positive about these AI coding tools?

Armin: So the biggest thing is that...

Host: Armin Ronacher is a widely known open source contributor and the creator of Flask, a popular Python web framework. He was also engineer number one at Sentry and is now building his own startup making heavy use of AI tools. Today with Armen, we cover why AI coding tools are making the choice of programming languages more important and not less. Python versus Go versus Rust versus TypeScript and which languages to use for startups and why. What Armin learned about error handling after 10 years of Sentry and why type-safe languages like TypeScript don't seem to reduce errors and many more. If you're interested in understanding more about the strengths and weaknesses of programming languages, how LLMs are changing how startups are built or want to know more about error handling, this episode is for you. This podcast episode is presented by Statsig, the unified platform for flags, analytics, experiments, and more. Check out the show notes to learn more about them and our other season sponsor. Let's jump in. So, Armin, welcome to the podcast.

Armin: Hi, happy to be here.

Host: So, let's talk a bit about programming languages. You've been very deep into Python for many, many years, well over a decade, and now you've touched on other languages. But with Python, how have you seen Python itself change? And can you give us, for those of us who are not as in-depth in Python, give us a bit more detail about the two to three migration drama? Which I think if you work with anyone who worked with Python, you've heard the moaning. I was at Uber when this happened and there was a lot of back and forth, a lot of delaying. It seemed very rare for across any languages to see what happened from Python 2 to Python 3, which seemed like breaking changes, lots of disagreements, some people just wanting, a lot of very competent and good engineers wanting to stay on the kind of two which is lower than three. So what happened there?

Armin: Yeah. So, fun fact, this year was the Python documentary which Guido invited me to be part of, which was very nice. And that documentary had a segment on it on that Python 2/3 migration and that basically made me go back in time just to remember all of it. Because like in many ways, my memory was also incredibly fuzzy and hazy. Like I also remember certain parts more than others. So if you really want to go into the deeps of like Python, Python 3, like look at the Python documentary has a whole segment on it.

Armin: But I think like in retrospective, I think you can't do it anymore. It was just at this moment in time, it didn't kill Python. But I think it could have, right? If people didn't put a lot of energy in to actually make that migration work, it could have been quite problematic for the language because if you go back to Python 3 and 3.1 and 3.2, there were so many missteps in the original library designs and decision to be made.

Armin: And so as someone who maybe hasn't like participated in this originally, you have to consider that Python started out with just basic strings like C. Eventually, it gained to be unicode strings as an option. So, you did a U in front of the string and it became a unicode string. And Python 3's biggest motivation of existence was to be a lot more strict about strings and move everything to unicode. What wasn't quite anticipated is that that migration was a little bit more nuanced in practice than designed, right? There was a very simplistic view of what Unicode looks in the real world. and that just it didn't look like that.

Armin: And I actually think that Python's messy remigration also had a really positive impact on other ecosystems like Ubuntu for instance because there were like two things sort of happening simultaneously. Python 3 got a little bit more lenient to working with non-unicode data but also a bunch of the situations where Python made it very hard to work with unicode was actually all kinds of configuration issues. So like I remember not every Linux had UTF8 everywhere. It was still very common for you to connect to university network and not find a UTF8 Linux which had all kinds of really funny things on a file system. Right? sort of these things converged a little bit. So the unicode story got better.

Armin: Um but you had millions of lines of Python 2 code that couldn't work on Python 3. And the initial assumption was you can just you just migrate once and then your Python 3 and you're good. And that's not what happened. We had to maintain libraries simultaneously for Python 2 and Python 3 for many years. And I for instance I remember meeting at the Python language summit in Florence where I proposed to bring back the U prefix on unicode strings so you can have them optionally because I noticed that if they were there I could write code that supports both Python 2 and Python 3. And I still remember there was a huge push back like why would you even want to maintain a library for both two and three? You should just move to Python 3 and then make everybody move like that. But that just doesn't it doesn't work like that.

Armin: So what really made the Python 3 migration ultimately work was a lot of effort put in by a lot of people and just a more realistic look into that you have to it took 10 years right or more even. You have to just support both versions in one form or another. That that is what made it work. But I think the Python 3 migration also left a pretty important data point for future languages to approach this better. And I remember Rust early on specifically pointed at the Python 3 migration to demonstrate why it wants something like an edition system or I forgot what it was called originally but basically a very targeted opting into new features or opting out of all features so that you can have code bases even within the same project from different versions.

Host: Nice. And speaking of the differences in programming languages, you mentioned Python has been your favorite language for a very, very long time. Later you helped introduce Rust into Sentry partially for performance reasons as well. And now you're also playing with Go. How would you compare these languages? And when you think of a language, what is your mental model of like what is this language good for, not good for? What would I use it for?

Armin: I wrote a bunch of blog posts that are not quite about languages, but they are sort of about my general split brain in a way because I have two programmers in myself. One is I like building cool open source software that hopefully thousands of people use and put a lot of handcrafted efforts into it. It's like the the Swiss watchmaking of source code writing and there is like I care a lot about the language. I care about a lot of the API and everything.

Armin: But then when you build a company, when you build a product, none of that matters. Like the is a is a write once run many times code, but not a write once write many times against your code, right? And so what makes Rust amazing, I think, for for crafting really cool open source code also makes it a sub-optimal programming language for a startup because there's much more friction in it, right? It's a much more precise language. It's a great lift up compared to C++ which is in many ways what the alternative would have looked at at Sentry for binary file processing but it's a much less capable language for a startup when it comes to rapid iteration.

Host: Can can you give examples of this friction? What

Armin: compile times? What like Rust compiles so incredibly slow.

Armin: That's a huge factor. Um, you write a lot more code compared to Python. You spend a lot of time thinking about types. I love types. I think they're great. Um, but certain things are incredibly hard to express with types. And I don't even have to think about it in Python, right? working with dynamic like the and I think one thing to consider here is like the dynamic and static languages sort of have aligned a little bit more over the years. I remember the dynamic keyword landing in C# was a great thing because it showed people they can have a statically compiled language with like opt-in dynamic runtime typing for instance and Typescript took that entire idea and sort of applied it to JavaScript. So you have you can have both worlds in a way.

Armin: Um but but Rust is like very straightforward like there's just static types and if your problem is a dynamic type problem then you're basically going to create your own dynamic type wrapper left and right. It is also that the borrow checker as great as it is is a huge um problem because it doesn't allow to express certain things that other programmers expect there to be like adjacent borrows. So you cannot have a struct that borrows into itself coming from C++ that's like why the hell like why does it do that hey here's my problem I want to solve it and the compiler says no it's like no you're wrong it should work so so there that is like you can run you can program yourself in the corner where it's like incredibly hard to come out.

Armin: um so I think from that perspective there are a lot of problems which are not rust shaped there are a lot of problems which are in fact rust shaped but but a lot of them are not and so one of The reasons I'm using go right now for the new company I'm working on is because it is a very pragmatic language and if pragmatism is what you're after then why not and and you can expect it to hang around and it's not super sexy and and it like worst case even if Google stopped maintaining it you can imagine there will be dozens of other people who want to keep it alive even modern Java is actually an awesome language like it has virtual threats and makes me happy like I don't have to deal with promises all the time um so the But it I have I think a much more pragmatic view when it comes to what a company should build its stuff with because the focus is not so much the code, it is the product that you're building.

Host: So you're you're now doing a a startup. Is it is it a few people? Is is it uh yourself?

Armin: So at the moment it's me and a co-founder uh but we just started hiring and and and also like it's it's it's me and a co-founder a bunch of Claudes and a bunch of Codexes and everything. So like the world has definitely changed in the sense that there's an army of interns that also now writes code and that has also changed the way I'm looking at things.

Host: So, so now it's it's you and your co-founder with the startup. You've got an army of Claudes and and Codexes and you're expecting to to grow the team to hire a as as you've settled on on the the technology you you said in this case this is a lot of it will will be go but what does a programming language need to give you and and for example like why was uh Python is is very flexible. Okay, it's a bit slow but but you said it was a good choice. What what made you what would you say like yeah go is is this good middle ground between Python and and Rust and so on.

Armin: Yeah. So right now I think if you build any company at all you're going to end up with Python in there if you want it or not. It will be impossible for you build a company that doesn't have Python code in it. Do you want to build your main service with it? Probably not. But if you do anything with ML, if you do anything with data processing, Python is going to be in there and and and you can write nice code in Python too, right? So it's like that's going to be there.

Armin: Good luck not having JavaScript in a company. And if you have JavaScript, you have TypeScript. So so those languages are going to be there. Another question is like how much of that code that you write overall is in those languages. And I actually felt that I don't want to build backend services in either one of those languages. And not even for performance reasons, but just for ecosystem, like what the ecosystems are good at.

Armin: Um, in my particular case right now, a big part of what what I'm doing at the company is parsing emails. And, um, that's just not something JavaScript is amazing at. Um, that's something actually Python is quite good at, but I think at scale is is it is like is is Python a good choice here? And I looked at it and I made the decision that I wouldn't. And from the experience of of of sort of doing Rust in that space at Sentry, I felt that like I can totally imagine that at scale like Rust will make its way back in. But at the moment, I don't think that that's the right trade-off.

Host: And so when you say at scale, this is just thinking about when you will have to process huge amounts of of data, lot lots of processes running and so on. Are we're talking specifically about the performance capabilities of the language.

Armin: I think it's like at scale of a company and that is not necessarily measured by how much data goes through it. If you scale a company either by the number of people working on it or the complexity of problem or the complexity of like how much data goes through it, you have to make tradeoffs and sometimes that means introducing a new language either because of performance reasons or because you integrate into some ecosystem that's particularly strong in some certain environment and then all of a sudden the language is here.

Armin: Why did Rust exist at Sentry? Well, it didn't just exist because I wanted to write Rust. It also was like, well, now we have a problem with performance. How are we going to solve it? And on the one hand, it was like, well, we could probably write a Go service here, but I didn't want to maintain another service. So, Rust was a pragmatic choice of embedding it in Python. And then later on when we did native symbol processing, the alternative to Rust basically would have been C++. And from my experience of working with C++, we use C++ on that project first. We just had a lot of crashes and I just didn't feel like I want to maintain that and so then there was no choice of using go because there was no go ecosystem. So unless I would have built everything from zero, it wouldn't have worked. But Rust was growing and rust had the same problem in the compiler. So the rust ecosystem for debug files was really good. And there was another company also working on it which was Mozilla at the time. So there was already two shared interests in this and and that was a really pragmatic choice even though the language was Rust.

Armin: Then maybe we overdid it a little bit by putting Rust into places where it wasn't the most optimal choice like the ingestion system. I think at this point probably wouldn't make that decision again but it would have been non-pragmatic at that moment to say like you know what we're going to do we're going to go into corner for like 9 months and we're going to build this all in this now. Um because for the the size of the team at the time the problem was too big right and so we had to reuse something. So that's one of the reasons I think why I can imagine that my company is going to again end up with adopting a language further down the line that it kind of initially said no to just because the situation makes it necessary and it doesn't necessarily have to mean that performance is the reason. It cannot just be ecosystem stability or or anything like that.

Host: And speaking about ecosystem stability ecosystems, how would you how do you think about Python ecosystem today, the Rust ecosystem and the Go ecosystem?

Armin: Python ecosystem is a lot of infrastructure, a lot of like provisioning machines. So that's what I'm using it for right now. I using Pulumi with Python to get the infrastructure up. Um it is really well entrenched in machine learning. So that's I have a little data pipeline going on that uses Python. It is also pretty good still at at at bringing services, web services to some of the things Python is good at. Would I necessarily write then like the main application logic in it? Maybe, right? Because like if if if I'm doing a whole AI first company and all it does is inference. I'm just waiting on a network layer anyway. So maybe maybe like putting some things together with Python is great. Um particularly also if you start to do aic things where you want to have code execution on demand, maybe maybe that's a good choice. Um so I think Python is a great future in it. I just don't think it's the most natural language to pick for a service where you think like it's going to be a lot of that like higher throughput. Also, I think like at this point, Go is a is a is a is an easier language to write than Python for a bunch of engineers because Python's increased in complexity and Go didn't quite as much. Um, so I find it to just be a better trade-off for complexity, too.

Armin: So Rust I think like if you have if you work with binary data if you maybe uh if if you have a you build a load balancer you build a database well maybe databases now people talk about Zig and stuff but like if you if you build like a a like a rust shaped problem which I think is largely defined as you single binary to distribute for some reason or another um it you you can't deal with the garbage collector you want very predictable performance um you really care what data layout although I dare say also is good but Rust is the benefit that if if you also then work in something that requires concurrency and where you're afraid of mismanaging memory um and crashes like modules that they might load into some other high security environment like a browser seems like a pretty good choice um working with again like dwarf files like working with just data in general I I love Rust.

Armin: and I think it's it's it's a such a pragmatic language also for if you extend Python. So if you already have a Python problem and now you have a performance problem or you have an ecosystem integration problem and Rust happens to fit good into that. Writing extension modules in Python for Python in Rust beats everything else out there, right? It's a really good choice for that.

Armin: Go in particular, I think it's just a good language for building web services and really kind of only web services. Maybe some command line tools. One of the other reasons why Rust I think is impossible to get rid of is Web Assembly never really took off but the use cases of Web Assembly are impossible to get rid of at this point. There's so many things you actually have to bring into the browser and Rust can do that. I tried it with Go. It's not a great experience because of the garbage collector and some of the complications that come with running Go in the browser. So you wouldn't you wouldn't do that but with Rust you can, right? So is it I think like you you kind of have to approach this from a perspective of like what am I doing here like what's the problem that I have and then you said like I want to have not more than three four technologies in the beginning what they use it.

Host: and then finally I guess the the the the one that we didn't talk about which is popular JavaScript typescript ecosystem

Armin: well in the browser you can't get rid of it. um on the server I think there's I'm very conflicted on this because I actually think that it's a pretty good language environment at this point. Particularly like hypothetically if if someone were to make the JavaScript 2 like the Python 3 of it and get rid of some of the naughty stuff. Like I don't think anyone should be doing this because I've seen the migration. But um there's a lot of good stuff in the language now. That's just also some really old dumb stuff.

Armin: Why am I not building backend services with with TypeScript right now? Actually, it's because of the npm ecosystem. Like that ruins it for me. I'm a low dependency kind of guy. I want to control my and I it just I feel like it's impossible for me for me to build productive in the JavaScript ecosystem with under 500 dependencies and that makes me uneasy on the browser. I can live with it because like I don't have much choice but on the server do I really have to do it? Do I have to deal with the fear of of of all of it and managing it? So it's like that that honestly ruins it for me more than anything else.

Armin: And so maybe maybe that will change as that ecosystem matures in other ways. But that really I think is is the biggest reason. Um and it it doesn't have enough benefits for me on the server that I I wouldn't just say like there's this idea that you have a unified codebase and every time I'm trying to make a unified codebase work, I just realize that the browser and that the server actually is sufficiently different that it's very hard to actually have that unified codebase. And so then it just be explicit like do some open API shenanigans, some code generation and it actually feels much better.

Host: Yeah. The the biggest argument I I keep hearing from developers for TypeScript is the unified codebase that you can have a React and TypeScript and then on you use let's say Node because that's or Express or whatever uh on on the back end and now you have the same language and people can contribute which I think was very very compelling especially before we had AI tools because now you didn't have to learn a new language.

Armin: Now it it might change. I I said this recently in a in a sort of talk I gave at a meetup which is I think right now the floor is raising the ceiling not really in the sense that the expectations that everybody has in everything is much higher than it was before but also because the expectations are higher and more tooling to enable it and so code generation for instance right now is in a much better state than it was even just two to three years ago you can even buy offtheshelf open API to SDK generators now right there like I don't even know what the companies are called Stainless for instance is one of them which just generates all the APIs for all the AI cloud providers right now.

Host: Yeah. For for SDKs. Yeah. They're one of them. Yeah.

Armin: That was a huge cost of doing anything at century is like how do you keep these SDKs around is now it's like well I have an open API shenanigan I somehow magically on GitHub an SDK appears. Is is it really so important for me to have a unified codebase if the code generation is so damn good? Like all of a sudden it doesn't matter quite as much anymore. Um, so I think that there's definitely some value to it, but you also have to imagine that if you have the unified codebase, then the the boundaries are sort of accidentally very undefined. Particular with RSC, I noticed this a lot where like it's very hard to actually reason your head around what's on the server, what's not. So actually having this extra boundary for me is almost like a feature.

Host: Speaking about uh AI agent coding, one thing that was very interesting that you mentioned is there's two of you in the startup right now. uh and you have just a lot of kind of you know like these AI interns running around Claude, Codex, how are are you using them what are you seeing for them and by the way can you also share what you're building with them?

Armin: Yeah. So, we scoped out the email to be a fascinating space to work in. And so, uh, you can kind of imagine what the kind of problems are that you have when you work with emails in one form or another. And maybe maybe a little bit of a of of of why we went into email is because one of the things that is very evident now of the biggest shift in at least my lifetime. what's capable of with computers is that natural language processing is now available at scale in many languages um for for for a reasonable amount of money and and that is a good reason to go into email because email is all about natural language and there's a lot of good data in it but it has been very hard to work with the data in the past.

Armin: if you read my blog you know that my opinion on AI stuff for programming was incredibly negative up until February March where it slowly moved around and then in May I was like holy nothing is going to be the same anymore. And one big reason why I'm actually so bullish on this is because when you think into and actually think it doesn't really work that well for teams today. I I think maybe maybe they're ways and I'm going to see as I'm scaling out this company like how it's going to work at scale. But one of the reasons why I think it will actually be really good is because if I if I look over the years of all the stuff that I wish I would have been able to do, just didn't have the time for it and I had to make this explicit decision not to do it. um that I have too many cases of this, right? Like I remember we we were were reworking the grouping algorithm at Sentry.

Host: The group grooving algorithm, what is that?

Armin: It's like making sure that two errors which are very very similar are grouped into the same group. But it's like it's it's a data problem at scale, but it's also one very specific data. And so we we spend like three weeks building just a basic visualizer for ourselves to figure out if the new algorithm works better than the old one.

Armin: But I know for a fact that I get this tool in 30 minutes on the side from Claude. It's around 5,000 lines of code and it looks better and has like pretty UI and everything, right? So the these sort of there are many projects that didn't go anywhere because the leg work needed to build yourself this bespoke tooling to even be able to pull this project off like a migration system or something like this. It's just too much, right?

Armin: And so that's I think one of the big changes I'm noticing for myself is that I have so many better tools around now. Um that just make it much better to work with. I have uh like I'm at the m I I moved from Terraform to Pulumi for one of the services just to see how that goes and and I I had Claude build me like my perfect control system to get my logs and um and visualize what's going on in production and and I would never have done this before just because like it wouldn't have worked.

Armin: Another big question is like how well does it work on a main codebase, right? My co-founder is not a technical co-founder. Um but that doesn't stop him from using Claude or Codex now. And it's actually great because it means that even someone who maybe you normally like in in certain stages of a company there's a bunch of stuff where you wish you could do something but but it actually requires maybe more coding skills than otherwise right and so there's an entire codebase where I basically don't look at which actually validates a core part of the feature that we're working on and it's just vibes and it works. It's not pretty, but it doesn't it's not necessary because like the whole point of it is figure out like is this the experience that we want and and this is great.

Armin: And then the thing is like how much of the code that I'm writing which I'm feel like this is going to be a foundation of what we're doing is is is at this point agentic and like I guarantee you it's more than 80%. And it's like there's very there's very deliberate code which is not written this way or like where where like really we put a lot of effort into it. But most of the code that exists in this codebase is not so important. It's a bunch of API endpoints. It's a bunch of open API specifications. It's a bunch of run-of-the-mill pieces of code. They should look nice. They should pass all the tests. They they should follow exactly the architecture that they want. But if you were to compress them down and say like what's the actual information contained within them is very little right and so with the right approach you're actually very productive with this.

Host: what made you so negative or or like just like pretty negative up to around February or so the things that you were just really skeptical about and then why have you become so much more positive about like these AI coding tools or agentic coding right

Armin: the biggest thing is that it it actually now starts doing work that I hated doing but I know I had to do so that that is a like as an example yesterday I I really had to figure out why one of like the endpoints that I have work running there in production didn't work quite as it should and it was it was not a problem of the error in the code it was a problem of AWS permissioning and like

Armin: there was like there was a chain of three things that I did wrong that all resulted in it couldn't do what it was supposed to be doing right like first there was an IM permissioning error and then there was a a whitelisting error on a different system and then the final up was um I forgot what it was. I think but but the final up was also something with maybe that was actually logic bug but the point is it was one error and one error and another error and you couldn't see them all at once.

Host: No.

Armin: And I was also busy doing something else and I was looking like I know I need to debug this. Um, but it's if I if I'm going to do this now, it's going to take me at least two hours. But it didn't matter because because Claude was doing a lot of it. Like I still had to copy paste a bunch of things from the logs, but it it knows so much because it has world knowledge access. It can combine a bunch of things together. And I could make progress on the thing I was doing while it was also fixing and debugging this production issue. And that is a significant improvement.

Armin: The same thing with one of the things I hated the most was creating repro cases but I know that every time I got a good repro case my my I enjoyed it so much more right and now I can say claude make me the repro case for this this is roughly what it looks like let's not try to fix it I figure I was like how do we get this one function that can keep calling in a loop that is what we're trying to figure out here right so I was not able to get that experience out of cursor even with the agentic mode prior compared to to to to May, April. But that was really what changed my opinion on it. And and then as I started going deeper and deeper into, okay, now that I know that I can do this and I can do this and I can do this, I also got incredibly adventurous in in in giving it more things to do and and now I can and I have a new sense of where it could go just by giving it the things I don't want to do, but staying in control of the things I do want to do. Um and and that was that was the big shift for me was was recognizing that it can do that in a way.

Host: And so far what are you seeing in what these agentic tools change about software engineering and what are the things that don't change especially you know you you've been writing code for like what 20 years or so. So, so system architecture and like overall complexity and and and like how you build things in a way that something stays maintainable as it scales into a larger and larger company. There's a lot of experience in it and I also didn't have all of that experience at any point in time. So I feel like I this is one of the things where I really like as time went on I learn a lot more which gives me a benefit over my prior self that didn't have that that is going to continue to be true.

Armin: Now there's a there's an argument to be made that maybe Claude is is like systems like this will will help you and maybe can even bootstrap these sort of things around it. If you fully delegate everything that you're doing to a machine, then the person that doesn't do that has an edge on you, right? Because like there's going to be some innovation that is not in these things yet, right? So like right now what these systems spit out is sort of our best understanding of how stuff worked up to this point, right? And so having something that maybe at scale is not in these corpuses yet is where some of the edge comes from.

Armin: It also creates a better culture in a company if you're not completely delegating everything to a machine, right? It's like because like what what's the point if it's just a bunch of like this like this is one of the reason why I want to hire more people now. It's just because Claude is not a human and it's just there's a there's an energy to to a well-running company and to to to motivated people that just a computer can't replace. So I think that is that is a big part that that sticks with you as an engineer is like understanding how to create the systems. Understanding when the code that it produces is is more right versus more wrong. Um so I don't think much of this has changed. I feel the same programming with claude even if I physically write less code. It's just that I I don't know the the the physical act of punching in the keyboard is is not quite the same anymore.

Host: We talked a lot about programming languages and you have very specific you know you have your opinions on them but with AI and Claude, Codex and all these more and more capable agents being here how important how do you think the importance of programming languages will change do you think it'll be limited to a few people or or just fewer and fewer people like yourself and others who actually have spent a lot of time understanding the pros and cons of programming languages thinking well beyond because it sounds like you know you can just tell AI to like solve it in whatever give it an existing codebase it it'll just do it right so there is an argument to be made that programming languages might not be that important anymore what is your take

Armin: well I think I I will be consistent with the prior thing that I said which is I think I have a lot of respect for programming languages because of the trade-offs that are in there and with AI now I think you might have to start looking into different kind of trade-offs the reason another reason why I end up with Go in my startup is because I just noticed how good go code scales for AI because the abstractions are very thin. It understands the code better. So clearly there's some there's there's a I did I did the measurement, right? It's like I I made it write a certain type of program in different languages 10 times and see like how often did it pass and I just noticed that it did so much better on Go than it did on Python. Um and much better than at Rust.

Armin: Clearly languages matter because the quality of what the agent is going to do will matter. And then is is any of the language they have right now is that the perfect language for both a human and a computer combined work environment. I don't know maybe maybe found the pinnacle of programming languages but maybe that's also exactly the moment in time where someone will come and say like hey I actually have a brilliant idea of what the better trade-off is here because the cost of doing this kind of stuff is going down with AI but the cost of this and this stuff is going up because of AI. So let me rebalance this in a different way.

Armin: Um, so I actually I think that I think that programming language will continue to matter a lot particularly the trade-offs that the programming language implies on the runtime environment matters a lot more. So I don't think that's going to change. I think it's just going to everything accelerates. Everything is going so much quicker right now or it feels to go so much quicker because the the fundamentals are shifting around and and this is not just with now everybody wants to build all the new things. It's also like you feel like if you if you're not working all the time now, you're missing out on some of the changes. And I don't think that's just on the product side. I think it's on the tech side, too.

Armin: I'm pretty sure a bunch of people who are feeling like I want to be the first person to build the programming language that works really well for agents and humans and and and but it's not just one person, probably a lot of them who are who are looking into this at the moment. Um it it just feels so like there is in fact a moment right now to build the perfect language uh or a better language that works in that environment because we are probably not going to get rid of a agentic decoding and what is the likelihood that all of the languages that we have right now are are the best ones probably not that high right so it will be the same and different in in many ways.

Armin: um and and one thing not to be underestimated is that the likelihood that computers are just going to program and No human is going to be in a loop. I don't think it's very high. I think the human will stay in the loop longer than we want and in many more cases than we want. And so you you cannot go down to say like well the optimal output is write bunch of assembly code because then nobody can review this anymore, right? And so if anything the trade-off will have to be like maybe higher level so that reviewing gets easier.

Host: Yeah. But you mentioned an interesting thing, which is how everyone seems to just be working all the time, which is this interesting contraction, how AI or these agents you can kick off and and you could you could in theory go to sleep and and have it just run in loops for you and so you could work less. But in fact, we're seeing working more and you and we now have this this thing which is very amusing to observe that AI startups in particular are increasing demanding or advertising this 996 from 9:00 a.m. to 9:00 p.m. 6 days a week especially in in a SF. They're putting it in the job adverse. They're they're posting it kind of proudly online that in the office at midnight or after midnight people are still there and working hard. Uh you you you share some thoughts on this as well on on your blog. It caused quite a bit of stir. But what is your take on on this? Because you did say, you know, like there is a lot of energy right now. There's a feeling of like not wanting to miss out. Clearly that's why why these people are doing it.

Armin: Yeah. So handful of thoughts, one of which is I actually have to credit Peter Steinberger quite a lot of uh pushing me into a decoding because he uh he worked at a company called PSPDF kit which he sold and then I I don't want to tell his story but a long story short at one point he was kind of stopped programming and then he fell in and said like hey I found this this computer which programs for me and he he said like he doesn't sleep anymore. more. He's like, I'm so addicted to it. You should try this in a way. Like he didn't quite say it like this, but like I'm I'm skeptical, right?

Host: But the computer that does your work, you cannot sleep because of this. That's kind of so amusing.

Armin: Yeah. It's like I I realized like it does something with your brain. If you if you do this a trendy coding thing because like in the beginning, I felt like any minute I'm not having it running and doing something, I'm just wasting time right now. It's like it was a huge part about this like it's almost like a drug, right? it it it has this instant gratification of something happened and and you can kick it off again and kick it off again. It's like this it is a slot machine, right? And so I think I attribute some of it to that and it took me a while actually to to come out of this like way of thinking because it's not very sustainable. Like there were a bunch of nights where I just I I I did through the night with with coding not because I felt like I'm incredibly productive on something using from a startup. It was just like just like this is blowing my mind. I just I just one more just one more prompt one more prompt. Um so I think like to some degree because if you're so AI close that probably also contributes.

Armin: The other one is I think everybody sees the change and wants to do as much as possible. I think the tricky bit with the 996 thing is that 996 defines a very very specific kind of work regimen which is you work 12 hours a day and you basically don't have a weekend. No. Um, I never did something like that even though I probably worked sometimes 80-hour weeks. And the reason for this is that I know that I cannot be productive all the time. If I if I work late at night, I will in the morning. I have three kids. I have a wife. I I family is very important for me. It's the most important thing. So, I will always arrange my work around that. You can work of intensity, but you can still um like do it in a way that is not crazy hours at crazy times.

Armin: And I also want to say there's a huge difference between if you are the founder or if you're very very early engineer with a really good compensation system where equity means something to you from most of the people uh who join a company later on or where the company has no realistic path of their equity being worth something. There's there's absolutely no point in putting a ton of effort into something where only one person benefits from. And there's there's just there's a lot of like subtlety to all of this. And what annoys me with the 996 point in particular is that they picked that word which if if you go back to why that naming even exists is because of basically there's no subtlety to it. It's like you work 12 hours a day, six six days a week, and you give up what what's what what is otherwise in your life. And I don't think that's I don't think it's a trade-off that anyone should make at any point in time because there there's more to life than working in a company. And and that is that is not the thing that you should be putting on your flag, which has such a negative connotation because of exactly what it did. um where people literally kill themselves over it, right? And like if if you want to be a high intensity work environment and say that but but but be be more because in practical terms you cannot be a company that that runs like that. You can only be a company where high energy people work and they build themselves up. But there's a limit to which you can do it and you have to be transparent about what the cost of this is.

Host: Now having worked at at Sentry which is all about error handling or at core is there you've probably seen tons of stats on on most common errors uh that that are coming up in different uh servers. What have you learned about error handling and how to build better software with fewer errors from a developer perspective?

Armin: So the biggest thing that I learned um and ironically hasn't really dramatically shifted despite the existence of tools like Sentry and their popularity is that many of the ways in which errors are worked with um just doesn't carry enough information or only carries useful information for debugging in debug builds. And that's problematic because the most interesting errors don't happen in debug, right?

Armin: Um now maybe this is changing a little bit with um with AI because like the cost of producing a reproducer from not perfect information might go down. So so that that maybe there's a counter move to that. I I really felt that as a language designer, as a VM creator, you put should put a lot more emphasis on on making sure that the errors can carry really useful information rather cheaply even in release and in production runs. And so not a lot of ecosystems don't do that, which is which is very sad.

Armin: And and I think that the fact that Python had such great introspectability without further like Python is a slow language. So if you already for all your debugging tools added, it doesn't get significantly slower. That actually was the reason Sentry existed because Sentry was able to show local variables for Python. it it had all this rich powerful data that for many languages even today we're not able to do because the runtime doesn't support it or or the the the runtime uh effect of doing that is too high.

Armin: and so that's the thing that I learned the most is like your your experience of of debugging issues is so much better if you have rich um rich information errors and unfortunately both language languages language creators uh runtimes often neglect errors. So they don't carry the right information and the application and library developers often don't think about errors at all or or they they captured them down in the wrong places and then the useful stack trace is gone. All that sort of stuff because you think they're exceptional. They're not happening all that often, but unfortunately every time they did happen, you know, you didn't have enough data. Um so like just in general like the design of errors is completely uh is not where it should be given how important they are.

Host: So that's one thing but Sentry works across a bunch of different languages almost all languages right these days or or at least the ones that that can that are widely used. Have you seen interesting patterns between the types of errors or the frequency of errors of of using languages? Basically, I'm trying to get to like how important it is is it to choose your language in terms of, you know, like having more correct programs?

Armin: Well, different languages crash in different ways and not just because of the language, but also what you build with it. And so for instance, JavaScript at scale like if you look at a lot of websites errors all the time the percentage of errors in JavaScript that actually are like meaningful is very very low.

Armin: and that makes sense because you you very few errors that you can actually cause in JavaScript and crash a browser tab for instance right so it's like you're you're it's like on error resume next you're hobbling along like something is broken but doesn't really like but like the the existence of an error in the in the console log doesn't imply that the website is broken but for instance like might like now the event listener might not fire anymore. So the the click is a dead click which is something that sentry needed a session replay product to find because the the actual error that you would find in sentry was insufficiently linked to user not being advanced on the page.

Armin: On the other hand, if you have a computer game it's written C++ and it crashes your session is over, right? And so the there are comparatively fewer crashes in queria games compared to all the nonsense going in JavaScript, but when they do happen, they're much more meaningful. And so it's very hard to say like error rates in different languages and so forth because like the impact that these errors have is very very different. So C++ code crashes in computer games shockingly little. like the the amount of traffic that Sentry gets from these is very very low, but the usefulness of each individual crash report in C++ is so much higher. So it's just it's it's a very complex topic at scale to say like how does something error because it really depends on like what does it take down.

Armin: Um obviously there are certain types of errors that you if you see them long enough you feel like they really shouldn't happen. To some degree there was a a large scale realization in the JavaScript ecosystem that type checkers could get rid of a whole bunch of class of errors because at the very least you have to explicitly check if this thing is nullable or not. But I I also didn't ever get the feeling that the adoption of Typescript would dramatically change anything about like centuries JavaScript error rates. If anything it like none of of that adoption had any meaningful impact on on how cry

Host: Yeah. Well, because because what what what what I would have expected, right? Like we know and I think it's pretty commonly assumed. I'm not sure if it's proven that type- safe languages will reduce certain kind of more obvious errors. TypeScript does this with JavaScript. I mean, it has a compiler, but but in the end, you you do get a a check before it all gets compiled to JavaScript. So, we should have fewer. That's what that's what I think. So you're saying that you didn't really see uh too much change?

Armin：是的，如果有什么影响的话，也是无法衡量的。然后，当然，你可能有其他频率函数发生，抵消了你捕捉到的一些错误的改进。也许这反而让你更大胆地去构建更复杂的代码。在 Sentry 的很多时候，是疯狂复杂性的时代。所以很多错误与“这个是否可空”无关，而是突然之间我正在与一个微服务通信，它们的版本不一致，我的类型检查根本帮不上忙，因为有人决定在网络层上它现在无论如何都是 null。

Armin：Sentry 多年来处理的许多软件的复杂性增加了。我认为你可能可以衡量这一点，特别是，我记得 React 生态系统中越来越多复杂东西的采用，极大地增加了错误的类型。比如，水合错误（hydration errors）在很多年里都不是问题，然后突然之间，出现了一整类新的错误，因为现在初始服务器渲染和 JavaScript 动态加载的视图稳定性之间出现了问题。这是一整类以前不存在的错误，突然之间就存在了。所以，仅仅因为我们改变的方式以及我们的应用程序变得多么复杂，就很难衡量这些事情。

语言设计中的权衡：性能与可调试性的永恒之战

Host：所以，即使在 Sentry，你们团队的重点也是完成任务。你测试它是否正常工作，然后当涉及到错误时，就像你说的，它通常是事后才想到的。但是对于编程语言，听起来很讽刺的是，在许多语言中，它也是一个事后才想到的东西。因为就像你说的，如果你更好地思考过，说：“嘿，在某个时候，程序会崩溃或出现错误。我能做什么？”然后你谈到了 context local 这种东西。

Armin：我不认为这是事后才想到的。我认为这是故意不做的，因为你必须想象，一门编程语言中有不同种类的力量在说“语言应该做什么”，其中一些力量显然是相互矛盾的。Context Local（上下文局部变量：一种在异步执行流中传递数据的机制）有什么问题？它们会使每个调用变慢。所以现在，所有想要更快语言的人都会说：“好吧，你现在得给出一个非常有力的论据，说明为什么应该这样做。”

Armin：我记得多年来在幕后隐藏的一场最大的斗争是关于原生平台的。编译器在某个时候做了一个决定，我们可以通过使用所谓的“栈寄存器”来获得一个额外的寄存器。你基本上会放弃最后一个寄存器来恢复堆栈帧，以换取一个额外的寄存器供其他用途，然后你必须使用一个非常复杂的 DWARF 解堆栈系统来恢复它。所以，你基本上是说，我们把能够在运行时遍历堆栈的复杂性，转移到通过一个嵌入在调试信息文件中的非常复杂的二进制程序来延迟完成。

Armin：但你通过这样做失去了什么？嗯，你失去了运行时分析的能力，因为大规模地这样做太慢了。所以，你不能做一个进程内分析器，而这被证明是非常有用的，因为你会发现通过追踪无法发现的信息。它也使得 Sentry 很难为原生代码做正确的堆栈报告，因为我可能无法恢复调试符号，因为它可能是某个人的二进制驱动程序，而且从来没有人给我 DWARF 文件。

Armin：我知道 Facebook 的应用会秘密上传安卓系统符号。你的设备会被随机分配是否上传，他们收集所有系统符号来提高错误报告的质量。像 Facebook 这样的规模可以这样做，但 Sentry 不能。我们不能让每个 Sentry SDK 都神奇地上传一堆文件，那会毁掉用户信任。但最终，我想那只是两年前的事，Debian 或 Red Hat 的某个人说：“好吧，这太荒谬了，我们现在要改变这个，我们要把栈指针带回来。”这是一场斗争。他们说：“我们对所有东西都进行了基准测试，我们只损失了 5% 的性能。”除了 Python，Python 解释器由于某种原因慢了 20%，所以他们没有对 Python 这样做。但这是一个标志位，重要的是这个标志位默认对所有东西都设置了。但这 5% 的性能差异，背后是长达一年的施压。

Armin：所以我认为这很难做，因为你有这些不同的利益在里面，有些人想要性能，有些人想要可调试性，他们可能生活在完全不同的世界里。也许那个非常关心性能的人从来不需要看任何崩溃，因为那不是他的责任范围。然后你有一个维护着一堆复杂服务的人，每个服务都有很小的崩溃概率，而且常常运行的不是他们自己的代码，然后一旦他们遇到崩溃，他们就没有需要的数据。

Host：这太有趣了，因为它提醒你设计一门语言有多难。因为你有这么多用户，就像你说的，刚刚遇到崩溃的人，他们实际上希望语言能支持他们以一种非常简单的方式恢复，但这默认会降低性能，要么通过性能，要么通过更高的内存使用。你不可能有免费的午餐。

Armin：是的，看起来午餐总是有代价的。但我现在对语言设计者有了更多的尊重。因为当你刚开始编程时，你会有这种天真的想法：“它必须这样工作。”然后随着时间的推移，你意识到，哦，你必须做出所有这些权衡，其中一些真的很难量化，你甚至可能不得不做出一些非常不受欢迎的决定，而且没有真正的对错，只有很多权衡。也许这也是我们在某种程度上有这么多语言的原因。但思考这些看似微小的默认设置对你能用它做什么，以及什么样的人会选择你的语言，有什么样的影响，是非常迷人的。我曾认为 Python 是构建 Web 服务最棒的语言，因为我可以检查每个进程，而且它不会变得更慢。但现在我知道，那也是 Python 从一开始就很慢的原因。

给初创公司早期工程师的建议

Host：作为初创公司的早期工程师，几乎是创始工程师，你学到了什么？对于现在早期加入一家快速发展的初创公司的工程师，你有什么建议？

Armin：嗯，我可以从一个十年前加入公司的人的角度给出建议。情况总是在变。我认为我的生活在很多方面是一条非常线性的道路，当机会出现时，我决定是否接受它。这就像秘书问题，但我并没有尝试太多次。对我来说，通过的标准是：“我认为这很棒吗？我想做这个吗？”然后我也许就会抓住机会去做。

Armin：对我个人而言，我不是一个很好的员工，因为我并不一定适合某个特定的模式。我在 Sentry 多年来的头衔，在很多方面我甚至都不认同。比如，某个时候上面写着软件工程师，但我却负责着一个有下属的办公室，有时我觉得自己像个发薪水的。因为一个小小的初创公司有很多这样的事情，现在有人也需要处理这些杂事，比如你需要雇人把家具搬进办公室。

Armin：如果你早期加入一家公司，一切都处于变化之中，没有什么是明确定义的。有些人喜欢这样，有些人不喜欢。我认为你必须愿意体验新事物。如果你加入一家草根初创公司，特别是如果创始人以前没有创办过公司，没有经历过公司从小到大的成长过程，因为即使是现在，我也再次意识到：“哦，我好像忘了我需要再做所有这些事情。”有些人喜欢这样，有些人不喜欢，你只需要意识到这样做的后果，并以某种方式为自己铺平道路，要么接受做任何出现的事情，要么非常有野心，刻意地朝着你想去的方向走，即使这可能会让你在某种形式上暂时受挫。

快问快答

Host：最后用一些快速问答来结束。我会问，然后你告诉我你想到的。在所有编程语言中，哪一门是你的最爱？我真的很想听听。

Armin：是 Python。有两个原因。一是因为它给了我现在的职业生涯，所以显然有很多情感上的依恋。但同时，它在很多方面是一门糟糕的编程语言，设计不佳等等，但它非常务实，我就是喜欢这一点。我想起了 Kell Henderson 多年来做的很多事情。他最初是 Flickr 的 CTO，然后是 Slack 的，一个 PHP 开发者。但他使用 PHP 的那种务实精神，在很多方面就像我一直喜欢用 Python 构建产品的方式。你并不真正关心它在那个意义上是好是坏，你关心的是你能用它做什么。而我能用 Python 做很多事，我为此感激它。

Host：那么，你喜欢使用的一个工具是什么？它做什么用？

Armin：我将以一个非程序员的身份回答这个问题。我喜欢使用的一个工具是螺丝刀。它用来拧螺丝。但我之所以如此喜欢它，是因为我多年来学到的一件事：我从未有过一把好的电动螺丝刀。然后当我们买公寓时，我买了一些非常好的工具，包括一把螺丝刀。它增加了我钻孔、组装家具和做所有事情的意愿。到目前为止，这可能是我现在最喜欢的东西，就是一把制造精良的螺丝刀。

Host：我想知道这里是否有一个可以应用于开发的隐喻。就像，当你拥有好的工具时，你实际上会更有动力、更愿意，也更大胆。嗯，谢谢这次聊天，真的很有趣。

Armin：是的，聊得很愉快。