从手动编码到AI辅助的软件开发历程与架构演进

Host: In today's preview: Larry and Sergey from Google. Somehow one of them reached out for an interview at Google. Google was three years old. Three years in, and I rejected them. You'd agree that's quite something.

Peter Mattis: I have always been a very prolific programmer. Looking at what was published on my GitHub, in my peak year, I probably wrote something like 100,000 lines of code a year.

Host: And that's before AI came along, right? This is you doing it manually by hand. Looking at this, I can only shake my head in awe. We also talk about the course on erasure coding they completed. That was a pretty big milestone: having nine shards of data where any five shards can be used to reconstruct them, meaning you could lose four copies and still recover their data. In your opinion, looking at software development four years ago before AI compared to software development today, should ambition grow?

Host: He created the GIMP image editor while studying in college, developed the original storage system that Gmail was based on, and has built many other large-scale, complex, and widely used systems. This is Peter Mattis, co-founder and CTO of Cockroach Labs. As Peter told me: "Over the past 30 years... but my current results are a little crazy. From writing throwaway vibes to high-quality, high-performance database-ready code with powerful coding models." Today, let's explore how he repeatedly leveled up throughout his career and how he built diverse systems. If you're interested in successfully using artificial intelligence to build production-ready systems, this episode is for you.

Host: In this episode—I really enjoyed spending time with them—the engineering team at Turbobuffer is doing something genuinely cool. They completely redesigned their storage architecture from first principles to make search faster, cheaper, and more reliable at massive scale. If you follow Turbo Buffer, their storage architecture was critical to their early success. Redesigning a winning architecture is no small feat: one business requirement is passing all unit tests, and the other is proving parity or superior performance and production reliability. But here is the most interesting part: Turbo Buffer is documenting all of this. Their new storage architecture, which they call T-Puff V3, tackles complex systemic challenges, and they are publicly sharing design decisions and benchmark results as they release them. They are keeping a development log, and the first post was published today. To follow the updates, visit turbobuffer.com/v3.

Host: Peter, welcome to the podcast.

Peter Mattis: Oh, I'm glad to be here.

Host: It's perfect. I'd love to dive in. How did you get started in technology? When did you realize computers were interesting?

Peter Mattis: I realized this back in elementary school and high school. Gaming was, for me as well as for many other software engineers, the gateway drug. Well, in my early years, my mother programmed at IBM. I didn't entirely understand what she was involved in, but we always had computers at home, like the Apple II Plus and Apple II GS. But then, when I entered college, I thought, "I'm not doing computers to make money. I'm going to be a mechanic," following my father.

Host: Majoring in mechanical engineering?

Peter Mattis: That specialty, yes. I had done computer stuff before. It felt really silly until I took a computer science course, and it was so easy. Meanwhile, everything else was completely overwhelming. I felt like: "I am not myself here. I don't belong in that industry. Please let me switch."

Host: And so you switched.

Peter Mattis: Then I switched, yeah.

Host: What was the first software you built in college where you thought, "Okay, this is real software that I'm proud of, a serious software project"? Must have been during your college days.

Peter Mattis: Well, it had to be college. I had this course with my roommate in college. What was it—a compiler course? I don't quite remember, it was 30 years ago. We were a bit bored with it. I had worked on high school journalism in my senior year and knew about computer graphics, and I wanted to make something like Adobe Photoshop. So we simply started experimenting, and we built what many people know today: it was called GIMP. With GIMP, I did a ton of work on the GTK graphics library. From there, it evolved dramatically. What's very interesting is that after college, I walked away from it. I was not very actively involved after that first year out of college, yet people still know me for it, which has been an interesting recurring theme throughout my career.

Host: When you started GIMP, thinking "I'm going to make something like Photoshop," were you being as stubborn as possible? That wasn't what you were learning in college classes, right? It was just you figuring out how to build a graphics rendering engine, drawings, data structures, and all that kind of stuff, right? You had to figure it all out.

Peter Mattis: Exactly. I remember trying to read whatever documents we could find back then. My roommate and I reviewed docs, and we were just trying to figure it all out. It's one of those things where, if you knew beforehand how much effort it would take, you never would have started. To take on that level of work, if you truly understood what it entailed, you wouldn't touch it. But once you start, it grows like a snowball, getting bigger and bigger, and at times you think, "This is truly amazing." But there's an amusing, little-known story. I might have mentioned it before, but it's simple: we reached a point where we decided we should release it publicly. Back then, there were newsgroups where people posted about various topics, including graphics. A couple of weeks before we planned to release the first version of GIMP, someone else showed up and said, "I'm working on this graphics program," and described what it could do—matching GIMP and claiming even more. We were just like, "Oh, this is terrible." But we decided to keep going and finish our work because it was fun. Then we released GIMP, and after that, complete radio silence from that other guy.

Peter Mattis: I took away a valuable lesson from that: someone else is always working on your idea. You shouldn't be discouraged or back off just because they announce it ahead of time. Nothing came of his announcement. In marketing, you often hear worries like: "What if they steal our thunder?" I never found out what actually happened to him, but the lesson stuck with me. If you read an announcement where someone claims, "I'm doing all of this," and you think, "Someone's already done it, I'll go do something else," then GIMP never happens.

Host: That's right. Especially looking at startups today, that's exactly how it feels. Whatever you want to do, explain it, right?

Peter Mattis: The advice I give people is: as unique as you think you are, in this world there are roughly a dozen people thinking the exact same thing. And probably a few of them are actively working on it. But most people never execute beyond the idea phase; they just can't bring it to life. So when you hear that someone else is working on your exact idea, they probably are. We're working on some very cool things at my current company, and I can guarantee competitors are working on them too. You just have to know what competition feels like, embrace that aspect, and enjoy it rather than fear it.

Peter Mattis: So, a bit after college, I was at another startup. Larry and Sergey from Google—I'm pretty sure it was Sergey—in fact, early versions of the Google logo were made using GIMP. So they knew of us. Somehow, one of them tracked me down and invited me for an interview at Google. This was back in 2001, when Google was about three years old. And I said no.

Host: You didn't have to say no! You said no?

Peter Mattis: Yes, I said no. Here was my calculus at the time: Google was down in Mountain View. I lived in San Francisco and didn't want the commute to work. So I went and worked at another startup for its first year. As it turned out, that company went nowhere. Then Google called me back and asked, "Hey, would you like to re-interview?" I replied, "Of course." They warned me: "We can't offer you the same stock options we offered before." I said, "Understood." I don't recall the exact math now, but I still remember the numbers from the initial offer. Obviously, it would have amounted to significantly more money if I had joined at the start. But I still did very well. I'm not one to complain, but whenever I recall this, I think, "Yeah, that turned out to be quite a story." Similarly, when Red Hat went public, they offered a Friends and Family IPO share allocation to many open source contributors. We were offered shares, and I made some money. While it wasn't a massive fortune, right after graduating college, it was a meaningful cushion. So I came out quite well, and working on GIMP made me realize: "Oh, I can actually afford things."

Interviewer: Tools like Photoshop and GIMP, and things like that—I was really into using this opportunity to make a positive impact. That was the essence of it: free software for everyone. You know, back then, who was I? Software at that time versus today was so different; free and open source is common today.

Guest: Yes, yes. And furthermore, that was where I learned how to program, becoming a good engineer and programmer now. And then I joined Google for the second time.

Interviewer: And they said, "Okay, what are you doing? Have you started working?"

Guest: Yes, yes. When I arrived there, they said something like, "Hey, you know..." It was in early 2002. It was April 1st, an auspicious day.

Interviewer: Well, you started on April 1st, 2002.

Guest: Yes, April 1st, 2002. And I arrived there, and they were like, "Yeah, well, you know, we're going to build an email service, 'Google Mail'." At that time, it wasn't called Gmail yet; internally, the codename was Caribou. Caribou.

Guest: Yes, so I started this work, and I was tasked with working on the message flow processing, storage, and indexing system. And I ended up working on this intensely during the first year and a half. In fact, I worked on it for maybe three years up until launch, which actually dropped on April 1st, 2004. Do you remember this? People thought it was an April Fools' joke.

Interviewer: Yes, I remember. Correct me if I'm wrong, but at launch time, Gmail came out offering 1GB of storage capacity—or something like virtually unlimited storage, starring 1GB. Back then, most webmail suppliers offered you only 10MB or even 4MB of free storage, like Hotmail or Yahoo Mail. And to pay for maybe 50MB or 100MB was super expensive. This launch showed someone could provide a free email service offering 20 to 50 or 100 times larger storage. We discussed this internally; it was a shocking and reverent campaign for the industry.

Guest: I think so. It was actually 4MB in the case of Hotmail or Yahoo Mail, and then you had this. And not only was there much more storage, but it was indexed very fast, so running a search returned results almost instantaneously.

Interviewer: But tell us internally: when the project started for Google Mail, how did the team address storage and speed? Because disk storage was still expensive back then, and we are talking about HDDs, definitely not SSDs if my memory serves me right. If you can take us back, what were the constraints, and how did you implement an innovation that had literally never been done before?

Guest: Yes. At that time at Google, there was the large-scale distributed file system named GFS, the Google File System. We looked at the numbers and thought, "Yes, we can develop it." They had a wealth of knowledge on how to execute search and retrieval from existing search systems. They had started creating a prototype, and then it was completely rewritten. That was actually exactly what I did: when I started, a prototype already existed, but then I completely rewrote it from the stream side. We decided from the beginning to implement message streams, which was quite innovative. This is a common phenomenon in mail systems today. And in the stream, the message flow required specific underlying data structures for storage, and you needed high read throughput. This is where B-trees were involved. I had already implemented B-trees about a dozen times before.

Interviewer: How were B-trees connected with the stream in email?

Guest: Well, for specific storage with thread IDs of incoming messages, you have to look them up. It used search indexing to match topics and associate the message identifier with the discussion thread. A lot of other things were happening under the hood because we were tracking the number of unread messages across discussion branches, and so on. Looking back, I can recall all these details now—it feels like ancient history. This was 2004. Here we are 22 years later. So from my memory, B-trees and inverted indexes were used there, and all this code was completely rewritten from scratch.

Interviewer: Offering this for free—surely you calculated the economics. Wasn't there a risk of terrible, massive losses?

Guest: Yes, yes, absolutely. At one point, the idea emerged: "Hey, maybe we can serve ads here." And this was considered one of those wild, crazy ideas. I wasn't personally involved in that part; that was Paul Buchheit, who later went on to build FriendFeed, do great things at Facebook, and eventually become a partner at Y Combinator. But one evening, he said, "No, I can take some of our existing advertising capabilities and integrate them right into the interface." It happened very quickly, and this grew into an unbelievable, massive business. So it comes down to understanding what is economically possible: "We know this costs a few dollars per user per year. How do we monetize it?" We didn't want to charge users for it. Eventually, there was pricing for Google Workspace, but at that time the question was: "Can we do this for free? Is it economically viable?" Taking that risk took real courage.

Interviewer: Do you remember the launch? The reason I ask is because I remember it was based on an invitation system. Not just anyone could get in. Was that done to control the anticipated demand? Again, offering something for free that was previously paid was obviously going to create huge demand. How did you think about monitoring demand and making decisions?

Guest: Honestly, this was teamwork. I didn't directly run that; I was primarily focused on handling the system load. The idea of "invite-only" served a dual role: to some extent, it throttled hyper-growth to manage capacity and prevent overload, but at the same time, it generated immense buzz and social heat. You know what it felt like when people were desperate for Gmail invites back then, right? People wanted them so badly.

Interviewer: And after dealing with Gmail, where did you move on to? What was the codebase and build collection like?

Guest: When I arrived at Google, it was prior to my arrival that they moved forward to Google2. At a certain point, we saw the consequences of Google2: Google2 was essentially driven by a single, monolithic Makefile. I think there were actually a few sub-makefiles, but it was really this huge, heavyweight Makefile. Someone approached me and said, "I think there's something we can do here." I was interested in assembly and build systems, so I laid the foundation for Google3. Many people were involved, but I was doing the initial design work on Google3. The initial insight for Google3 was: "Look, writing these makefiles is somehow terrible." I presented this concept based on declaring file collections. I decided to make it a simplified Python language, keeping the syntax clean. Subsequently, instead of a monolithic build file, you didn't have to write low-level build scripts like that anymore. Over time, this evolved internally into Blaze. Externally, Bazel was open-sourced from it, and Buck at Facebook came from people who worked on this.

Interviewer: Did it significantly improve convenience and maintenance? Was defining dependencies before like writing assembly language?

Guest: Yes, exactly. It was like dealing with dependencies in assembly language. Nobody wanted to write their dependencies in assembly language where a single mistake meant missing something. Instead, you express dependencies declaratively, and from this, the build system handles the direct compilation and dependency tree. This brought massive productivity improvements. In Google's massive repository, a clean codebase structure helps improve productivity, making builds demonstrably simpler, cleaner, and faster. So we experimented, created these build files, and people enthusiastically adopted them.

Interviewer: What did you focus on next?

Guest: Well, I mentioned GFS earlier. At a certain point, we realized GFS had severe limitations and bottlenecks in scalability. Because I had worked across the entire storage layer for Gmail, I had actually experimented with alternative storage systems—research that laid the groundwork. So we worked on this, and I was invited to join the founding team for Colossus, the successor to GFS. Colossus is still running today, having gone through multiple iterations at Google. It is the second-generation distributed file system.

Interviewer: So what exactly is Colossus? When you describe the system from an external architectural perspective, how should we think about it?

Guest: Yes. When talking about the system and thinking from the outside...

Colossus Engineer: みてください S3 のようなもの。BLOB ストレージの一種彼が持っていた物フラットネームスペース、S3 のようなもので、名前があり、しかし彼女はとても限定。えっと、でもこれはファイルシステムではない POSIX。だから、カタログには、完成すらしていなかった許可制度。その多くは後から追加されました。しかしファイルは保存先お近くの車。全体がある公園だよ、君のねそういう種類のサービスすべてのファイルが含まれています。彼らはそれを書き留めるか、今は SSD に搭載されています。クライアントはアクセス。そしてこれらすべてが複製するためもしそれらが起こるなら。

Colossus Engineer: いつなのかはわかりませんが、時々 S3 エンコーディングを受信しました消去、私たちはエンコードコロッサスにおける消去。それある種の大きな突破口。

Host: 何、何のエンコーディングですか？

Colossus Engineer: ハァ。私たちは使用済みリードのアルゴリズムソロモン。

Host: えっと、それで、聴衆が消去によって、何か思いつきますか？例えば「私は欲しい線がある」、ええと、そしてそのようなもの、ええと、RAID と呼ばれる。

Colossus Engineer: 私はあまり完全必須レプリカ。私はできる A プラスを取るなら、.そしてますます今はあると思います。方法によっては、そうですね。はい。And we had to, uh- 強制的にそれは働くことです分散型ファイルシステム？あなたは集中していましたね。保留、保留機会より効果的データストレージ。それは彼らよりも効果的ストレージ。つまり、GFS では、ええと、さまざまな費用があります。保存しない場所 2 部または 1 部データのコピーまたは 2 部、3 回保存になるだろう使用。そしてリード・ソロモン大幅にそれを減らす。私はそうではない最初から覚えている私たちが用途リード＝ソロモンだが、それは実質的に 2 倍。

Colossus Engineer: でもあなたはあなたもこれを受け取ります留保付きで。それで、少ないように見えるが、上記の予約について。だからこそ、私は思うのですが、世間知らずは世間知らずだあなたが言うとこうなります：「はい、私は私のデータは 3 つの複製場所」3 つのノード、3 つのマシン、物理的な機械、そしてあなたはこう言います。「1 つのコピー 1 つのコピー、コピー 1 つ。私は３箇所。完璧です。もしそのうちの 1 つが爆発する、私はすべてを持っている 1 つは 2 つのまま残る、完璧に。そしてあなたはアルゴリズムはあなたなのいいえ 3 倍のデータ、そしてデータ量は 2 倍に賢く分割するそれらを機械間で、または、もしかしたらあなたはそれらを取り上げて下げてください。例えば、それらは爆発するだろう、私は私のデータはすべてまだ残っています。分割されているため半分に。右。

Colossus Engineer: あなたも使用メンタルモデル、もしあなたがただ望むならこれを理解する高レベル、つまり次のように言うことができます。「私は 8 人欲しい。このデータのコピー。」または、私には無理だと思います。使用九。彼らはすでに公の場で発言したこれについて。持っていますか約 9 データの断片だが、これらのうちどれか 5 つ断片は彼らのために使う再建。それつまり、失う君ならできるデータ。多くの場合、これはむしろ、ほら、最初の 5 人断片は正確ですレプリカ、その他 4 部パリティ。私は少し詳細をいくつか忘れていました。以上です。

Colossus Engineer: でも、あなたはそれを証明できますこのアルゴリズムうまくいくか、それともうまくいかないか？はい？これは少し非常に重要な理由できるだけ早くうまくいくでしょう。うまくいくよ。そして、そんな感じ。私はわからない、私は私には全く理解できませんでした。すべての数学背後にいるのは私いつも後悔していたそれ以上はしなかった数学大学ですが、そうではありませんそれは必要だった。あなた、どのように読書と飛行、改良版どのように機能したか、そうですよね？これはそれは 1970 年代のいつ頃だったか何か関連のあるコミュニケーションネットワーク。だからあなたはこれを取ってあなたが知っている、これを使っていますか？経験はありますが、使ってみてそしてあなたはすべてをしなければならない優れたエンジニアリングこのため、これがシステム内で作業しましたデータストレージ。

Host: そして、あなたが作成する分散システムデータストレージ、コロッサスなど、まだ物事は、普通だよ、君保存したいデータは安定しているが、効果的な方法で？他に何か問題はありますか？それは必要だった解決する？

Colossus Engineer: 私は思う潜在的に重要シャーディングのようなもの再シャーディング、またはメタデータ、そういったこと。ええと、特に Google が GFS で働きたいこの制限があった拡張性。私あれが全てだったと思います。千人かもしれない車、そしてそれは何かどのような、どうやらこれは面白いそれだけでは足りないですよね？2004 年にしてはかなり多いように聞こえる年。そしてグーグル「いいえ、私たちはそれは必要だった最大 10,000 まで拡張可能機械」

Colossus Engineer: そしていくつかボトルネック。こんなのがあった GFS マスターノード、ちょっと窮地に立たされている。必要分散型メインノードメタデータストレージ for all objects. そしてシステムと呼ばれるビッグテーブル。コロッサス彼はメタデータ内部 Bigtable. そして、その一つは、私はそれを誇りに思っていますし、解決策の一つしかし実際は私たちは分散型分散型断片。これは一体何ですか？だった？Fragments of 64 メガバイト。そして各ファイル。はい。そして, repair work. そして That's it for short. したがって、最大の Bigtable なので、私たちは、Bigtable を動作させるにはコロッサスの頂上。

Host: つまり、円の概念、見てくださいベン図、お分かりですか？ええ、まあ、読み込み中ですよね？はい、読み込み中です。ああ、そうだね。はい。どれか動いていますか？のように、どこかが必要何かをモデル化するということですよね？

Colossus Engineer: はい、いいえ。つまり本当のところは当時働いていた、そしてそれ以来彼らは交換したんだよ、なぜならそれは始め方とそれを使うには、あなたが持っているもの、そして最後にあなたは取り除くこれだが、いくつか Bigtable 財団は、 Bigtable はコロッサスを使用しました。巨像があり、 Bigtable を使用し、そしていつものように大型テーブル（立ち見）コロッサスの頂上。私が言っているのはこのことです。そして、すべてうまくいった。何年も前から、だから私はそれらがいつのものかは分かりません。これを処分しました。彼らはこれを削除しました一瞬だけど、えーと…えっと、あれは…でも、私は…そう思うのですが…何が作れるのか進行中のハッキング長い間、これらはハッキングであり、あなたを助けています死者から蘇るポイントですよね？移動ど真ん中ですよね？はい？なぜなら、もし私たちがそうしなければならなかったんだ、ほら、このようなことを実施する Bigtable レイヤーは、始めれば、簡単だろう遅延時間、必要コロッサスの創造。

Host: 私がシステムの中で衝撃的、コロッサスと同様に、彼女が約束すること、あるいは少なくともこれ約束されるだろう、それは Google 内部では、しかし、分散ファイルシステムは外部の、彼らは約束する高スループット能力、高いアクセシビリティと低遅延。そして私にとってはいつもやや物議を醸す、例えば、そうですね、私は思うこれはとても簡単です建てる分散ファイルシステムシステム。私の限られた知識おそらくできるでしょう。何かをする私は高い容量、しかしレイテンシーが高く、なぜなら毎回 This is for all the lines. あなたすでに述べたもの方法だが、どのように例えば、同意する、低くなる方法遅延、高スループット能力、また起こっているファイルシステム？

Colossus Engineer: つまり、これらの分散型ファイルシステム遅延はそれほど大きくない低い。つまり一生懸命にコロッサスが作った円盤今のところ、何をしているのでしょうか S3、あなたは本当に遅延に気づきます。非常に生産的システム、Google、うーん、GCS、 Google の競合相手、を基盤として構築された帯域幅、読書は 20～30 ミリ秒、そしてそれは限られている遅らせるハードディスク。もしインストール先 SSD、レイテンシーは SSD ですが、実際にはそうではありません最もモダンな SSD のレイテンシは、私たちの産業、単にどれだけ速く機器になった。

Colossus Engineer: ハードドライブから読み込んでいますディスクは現在使用中です 5～10 ミリ秒、つまり、私は私は決して触らない。ハードドライブ。 NVMe 経由で SSD から読み込む 50 マイクロ秒かかります。 50 マイクロ秒。これはマイクロ秒、1 ミリ秒 1000 マイクロ秒。だから私たちは私たちが話しているのは大きな違いだ。さて、今はスタートアップ企業がありますまたはインフラストラクチャ企業が始めるそれを使え彼らができるという事実 NVMe レイヤーがあり、彼ら、彼ら、彼ら物を引き上げる、または予想通り、あるいは予想外だが、おっしゃる通り、例えば、物理的現実変更すれば、システムを作成する使用すべきここに。

Colossus Engineer: はい、もちろん、あなたもいいですか、それはディスクが SSD を使うとずっと速くなります。もっと早く。つまりそれはまさに狂気だ、どれくらい速いかゾーン内 Google センターの遅延 or Amazon. それははい、しかし私たちがコロッサスを創造し、それらが何だったか覚えている。ミリ秒データ交換、そしてゾーン。つまり、私はこれらのことを考えます。「本当に提供「うまくいった。」はい、時々私はソフトウェアはるかにより速く、そしていくつかのソフトウェア。時々私は多すぎる私たちは皆に頼っています私たちはそのようなことは一切しません。

Host: ナプキンのようにサイモン・エリクソンとターボ Buffer はこれについて話しているナプキン数学的解析、つまり、こう思ったでしょう。「さて、理論はこうだ。ハードウェアの制限ソフトウェア; SSD からの読み取りはシステムを構築する、とは異なりええと、人は気づくだろう、私たちは、これ。はい、はい。そして時には、例えば、考える必要がある人為的な遅延もありますが、多くの場合、レベルに対処するストレージシステム Turbo Popper では、彼らはそこで素晴らしいことをしている。もの。

Colossus Engineer: あなたには規模について考えてみよう車とスピードはるかに機械人間より速い感知。例えば、ある人が耐えることができる約 100 の遅延ミリ秒、またはあなたはゲームをプレイしています、もしかしたらあなたにはフレームレート約 4 ミリ秒ですが、車はたくさん欲しいそれよりも。サイモンは言う：「数学数字で、そして時には文字通りあなたを…ゾーン間の遅延、間の遅延地域、これ光速と fiber optics. 欲しいですかいくつか聞く驚きの事実？

Host: 聞くのが好きです。驚きの事実。

Colossus Engineer: 最速の方法荷物を送る弾丸を発射する彼を宇宙へ連れて行った。これは速度が真空中の光はより高い？はるかに速い。いや、正確には、まだ完成していない。真空。とんでもない。はい。だから、あなたはもっとカバーする距離。はい。

Host: ええと、実際には、どうやってやるの？

Speaker A: Send it over, exactly that.

Speaker B: That's right. After that, Starlink transmits it, sending it down on the opposite side. That is, you want to get it out of the atmosphere as quickly as possible, plus the digital conversion. All that takes time. If you do this well, transmission can end up being faster than optical cable where the speed slows down in the medium, right?

Speaker A: In any other medium, it's slower. I had discussions; I went deep into it. They talked about using satellites and microwaves, but they don't offer those to you.

Speaker B: But they probably will. Yes. Yes. So what they do is, high frequency is not actually big enough to transmit through space freely, so they use microwave beams.

Speaker A: Yes. But if you really want to be faster, you need to build something between them, like a vacuum tube to send it through... Maybe they will actually do that.

Speaker B: Maybe they might. Yes.

Speaker A: But I mean, in my opinion, performance today is just incredible, exactly because of how many layers you have to pay for performance. This rabbit hole goes extremely deep. You are almost certainly working on a multi-threaded system, right? And you think: "Oh, I need a multi-threaded program, so I need synchronization." To get the best performance, you just avoid synchronization. Part of this is lock-free programming, but part of it is layout so that locks aren't needed at all. You have to pay attention to the processor.

Speaker B: And there is a whole system of caches above the processor: L1, L2, L3, memory, and to disk. You need to work with these levels. If you optimize access to the cache, performance will be massively improved. Some people focus heavily on this all day. There are many other aspects, like this focus. You also get gradual degradation of software performance if you ignore it.

Speaker A: Let's talk a bit more about programming. You built several libraries, data structures that are quite brilliant. I mean, generally, people just think sorting algorithms are fascinating. Can you explain sorting simply, or is it much more complex?

Speaker B: I know. I know all of this. Yes. And you look at it and go: "No, no, but this is clever..." Yes, you reach these levels where you say: "Oh, wait. Someone invented a really clever idea here." So, one thing I worked on when I did some part-time work at Google: a colleague came up to me and said: "You know what? We use the STL map structure everywhere." And the STL map is a balanced binary tree—whether red-black trees or whichever balancing algorithm they teach computer science students at university. And he came to me saying: "I think we can do better, because look, here is actually an issue with cache. Every time you traverse a node, you cross into another cache line."

Speaker A: And he thought about using something else instead of pointer paths to do this, some wonderful data structure that everyone should pay attention to?

Speaker B: But for a moment I thought: "Actually, this is close to a B-tree." I had already implemented B-trees several times and understood how to build one that preserves almost all the semantics of STL map. It can do this cleanly, though when you insert data into a B-tree node, you may have to shift data around, so you don't get pointer stability. If that's acceptable for your use case, it actually packs much more data together.

Speaker A: So the thing is, in the real world, a B-tree works just like that to explain this?

Speaker B: Exactly. Because if you have a small list of elements, say 8 elements, the best and fastest way to access them is just a sorted set of elements, right? Literally, you have no tree pointers at all. A simple list, a very simple sorted array, and you do binary search. Often that is way better. Think about building it like this: start with one node of 8 elements. Oh, I need to insert a 9th element? I split it into two parts—one with 4, one with 5—and then a parent node points to them. Then you just apply this recursively. That is the B-tree algorithm in a nutshell.

Speaker A: In practice, almost no one implements it all the way through like that today, even though there are so many optimizations you can perform on B-trees?

Speaker B: There are a lot of optimizations you can perform on B-trees. But coming back to this: there was an existing map implementation, and your colleague looked at the code and said: "I think we can do better." From the outside, you don't know how these libraries or data structures were built; I used to assume, perhaps naively, that really smart people had sat down, evaluated the state of the art, implemented it, and there was no way it could be faster. I've had disputes in the past where people claimed: "This is definitively the fastest sorting system." But tell us a bit about what actually happens inside, and how people like you and your colleagues end up saying: "Oh, what if we tried something else?"

Speaker B: He was working on a huge internal system called Gaia, if I recall. In comparison, when logging in, you have an identifier you need to look up. It compares the user identifier and email metadata stored in STL maps in memory. You look at where memory goes, and this shows up prominently in profiles. And we asked: "Can we improve this?" That was the beginning. He and I started tackling this problem: "Let's build our own internal library." At that point, working on our own system, he contributed it back to the shared codebase.

Speaker A: That happens when managing layers. Everything seems fine, and suddenly there's a huge explosion in memory usage. You simply have to ask: "Why is this happening, and is there anything we can do about it?" Right?

Speaker B: Right. In the early stages, he realized that in a map mapping integer identifiers to something else, look at a red-black tree: each node stores pointers in addition to your data. You might have an integer identifier of 4 or 8 bytes, and then multiple pointers... right? You look at that and think: "Oh, this is a massive overhead." And with a B-tree, you get much better spatial locality, which makes it significantly faster.

Speaker A: But at the same time, that led to contributions to Go later on, right?

Speaker B: Yes. Well, that came later. Can we talk about that in a moment?

Speaker A: Yes. Okay, another thing I pay attention to is research into hash tables. Hash tables are the data structure when order doesn't matter. Over time, ways to create hash tables evolved: using an array of segments, things called hash chains, or re-hashing and probing down segments to find an empty slot where you belong. I remember reading about this new technique that originated from Google engineers, apparently in their Swiss office, which is why it's called Swiss Tables.

Speaker B: Yes, exactly. That seems to be why it got that name. I believe that's where the name came from. Later, when I was working with Go, Go had a built-in map structure. It's a hash table, an extremely optimized hash table, because the Go team has very talented engineers working on the runtime. Different people tried writing Swiss Tables implementations for Go. I looked at some of them and thought: "It's very interesting what they are doing with Swiss Tables—and I'll explain in a moment how it works—but looking at this, it's really hard to outperform the Go runtime." The runtime execution is already so good.

Speaker A: I continued pondering this for a while, and eventually you decided to take on the challenge?

Speaker B: I kept thinking about this for a while. Eventually, I had a business trip to India, to Bangalore. After hesitating for a long time, I thought: "Okay, let me try to build one." I set up solid benchmarks. It's like catnip for an engineer: "Can I do all this? Can I make it faster than everything else out there?" People had tried using Swiss Tables, and Go experts would say you have to navigate through this and that hurdle...

Peter: There were a few ideas, and that's kind of what brought them together. And then the implementation, the benchmarks, exactly the quorum, and then the idea came from someone on the Go team once they were able to.

Well, mine was accurate for my company. It worked well for us to use, but actually the implementation at the runtime environment, that's a totally different level.

You just showed them, here's an implementation, and they took inspiration and gave ideas: "Yes, that's great. We want to do everything..." They're always looking for ways to make the runtime faster, and they just said, "Oh, wait." And you made this with many contributions over the years.

Gergely: You left Google years ago, right? Yeah? That's all done from the outside. This is from the outside. That's great.

Peter: Yeah, and other people are contributing from the outside. You know, we had another colleague. He contributed one of the CRC implementations.

Gergely: Some changes to CRC?

Peter: Um, checksum, cyclic redundancy check. Hmm. Yeah, Intel published some articles on doing very fast CRC in assembler, and he contributed one of the implementations from that. You see a lot of these things where people just contribute from the outside. This is not unusual. I mean, I don't really know all the details, but people regularly contribute to these things.

Gergely: Peter just described how the work on Swiss Tables came out of tracking Go tracker issues, with different people participating in the work, and then the Go team taking it all across the finish line.

Here's a reminder about our season sponsor Linear, which is about work coordination between people and agents.

One thing I've noticed is that, like most of us working with agents, it's pretty much a single-player game. You open a terminal interface, talk to an agent, and that usually results in a PR. But the rest of your team has no idea what happened in that chat unless you tell them and copy the whole history. And when everybody on the team works like that, a lot of work happens invisibly to the rest of the team.

Linear's perspective is that an agent's job is to be a team player. Even today, you decide what work needs to be done. Now in Linear, you can already delegate an issue to an agent coder. This agent can be a Linear integrated agent, for example from Codex, or whatever agent you prefer. In either case, the engineer who delegates the task remains responsible for the outcome.

What I really like about Linear's operating principle is how visible the process remains. Your teammates can monitor the agent's session, check the PRs it creates, and jump into reviews. We've moved from single-user work to multiplayer engineering work with agents.

Oh, and another thing I like about Linear is the focus on costs. Linear Agent's auto-routing chooses the most suitable model for the task. Teams can also check usage and set limits, tracking usage so they can empower developers to use agents without letting costs spiral out of control. Join at linear.app/pragmatic.

Gergely: Peter earlier mentioned Gaia, the internal system Google uses for user login and authentication. What Google built is amazing; all the systems, including this one, are neatly ordered, but most of us don't have our own internal Gaia.

That brings us to our seasonal sponsor: WorkOS, which is kind of like Gaia for the rest of us. It provides the infrastructure you'd otherwise spend quarters building yourself. WorkOS has single sign-on, SCIM directory sync, audit logs, role-based access control—virtually everything major enterprise security teams ask for—delivered through clean APIs. This way, companies can move beyond basic login and sell to Fortune 500 customers without having to build their own internal identity platform.

And WorkOS is already developing the next version of authentication and authorization. Their latest product is authorization for AI agents: WorkOS Airlock.

Think about what happens when you give an agent a task, like the ability to clean up our sales pipeline. The last thing you want is to give it permanent, wide-open permissions where it could delete everything if something goes wrong.

Airlock sits between your agent and the tools it calls. It evaluates the agent's intent, and then allows, denies, or redirects it to a human for approval. Simply put, the agent never sees your credentials, and all calls and verdicts are registered. It works with coding agents like Claude Code and Codex, as well as via MCP gateways.

So if you're working on ways to let agents actually do production work without giving them excessive permissions, check out WorkOS Airlock at workos.com/airlock.

Gergely: Now let's get back to Peter and why he left Google after creating Colossus. You're at Google, building the Colossus distributed file system. You were probably working on what is honestly one of the largest systems on Earth. Why did you even think of leaving?

Peter: Yeah, yeah. Well, after Colossus, I spent some time busy with other projects, like Google Goggles. Do you remember that?

Gergely: The glass hole project?

Peter: Yeah. These ideas always come back, and they're still around today. This was one of Google's early projects. And yeah, it was technology whose time hadn't quite come. I don't think it was ready for the moment. Apparently, actually creating the glasses was far more difficult. You know, on Android smartphones, we tried to launch this, but it just wasn't powerful enough. After that, you know, a desire to travel and explore rose up in me. You know, I was standing there at Google—it's the weirdest thing to say, but other people feel it too: you want to test your strength in another startup company.

Gergely: That didn't work out. It was acquired by Square. So, yeah, for a moment I'd like to stop there. What was this company?

Peter: Well, what we founded was called Viewfinder.

Gergely: Viewfinder, yeah.

Peter: Yeah, it was in the space of mobile photo sharing. Sounds familiar. It was like Instagram; it was kind of like Snapchat. In 2012.

Gergely: Yeah, just as Instagram was taking off.

Peter: Yeah, yeah. No, we didn't figure out the right go-to-market, how to attract users, how to achieve viral growth, anything like that.

Gergely: From the outside, reading up on it, I see that you're a co-founder of a startup that got acquired by Square. You probably had bigger ambitions, and while this wasn't a bad outcome, it wasn't the ultimate dream either.

Peter: So, the term I used was "purchased." Sometimes a company is acquired for their intellectual property, their product, their business, and sometimes they're bought just for the talent. The people. And for us, they really just bought us for the talent. They purchased the IP, but I doubt Square ever did anything with it. That wasn't what they were working on. But we had a very strong technical team, and that was exactly why we were acquihired.

And, well, I don't remember all the details. We raised a small amount of money. We were able to return the capital to our investors, so they were reasonably satisfied. Maybe there was a slight haircut on the funds, but they essentially got their money back. I mean, as a founder—investors are adults, they're used to losing money, but if you lose a ton of their money, it weighs on you. So returning their capital makes you feel a bit better.

Gergely: Yeah. Yeah. And you stayed for a while after the Square acquisition, and then you started to get that itch again?

Peter: Yeah, yeah. Because, you know, we work on these distributed file systems and distributed data storage systems, like Colossus. And one of the flagship projects built on top of Colossus is Spanner.

Gergely: And how is Spanner different from Colossus?

Peter: Well, Spanner is fundamentally a distributed database. Colossus is a distributed storage system. And considering the difference between a distributed storage system and a distributed database, someone might think, "Oh, they both store data."

Gergely: I was just about to ask that. Because any distributed database eventually has to rely on a data storage system.

Peter: Yeah, yeah. For Colossus, you see, it's focused on huge files, and those huge files are append-only.

Peter: 不能随机覆写，只能在文件末尾追加。

Colossus 针对的是巨大的文件块，单文件尺寸通常在 64MB 以上，甚至可以达到数 GB。

但数据库面对的完全是另一类场景。比如对于关系型数据，一张表内可能包含数十亿行记录，每行被划分为不同的列，有具体的类型与索引。数据库工程的核心难点与底层分布式文件系统存在质的鸿沟。

常规的分布式数据库往往搭建在底层的分布式文件系统之上，两者是分层支撑的关系——Spanner 正是构建在 Colossus 的基础之上的。

正因如此，Spanner 中的许多核心设计决策，直接受制于 Colossus 仅支持追加写这一现实：你无法在底层文件中执行原地更新，这就倒逼上层数据库必须依靠不可变文件（Immutable Files）来管理数据状态。

而这正是日志结构合并树（LSM-Tree，Log-Structured Merge-tree）大显身手的领域。LSM-Tree 并不是 Google 发明的，但 Google 确实通过 LevelDB 以及后来的 Spanner，将其真正推向了工程主流应用。后来 RocksDB 更是将这一架构发扬光大。

再后来，我自己也重新独立实现过一套 LSM-Tree 存储引擎，这也正是我们在 CockroachDB（红眼蟑螂实验室）内部所使用的组件，它的名字叫做 Pebble。它的全部底层核心机制，都建立在“将数据保存在一系列只读的不可变文件中”这一原则之上。

诞生 CockroachDB：告别 Square 与重聚创业

Gergely: 那么，你们后来究竟是如何下定决心创办 Cockroach Labs（红眼蟑螂实验室）的？