---
author: The Pragmatic Engineer
date: '2026-11-10'
guest: ''
layout: post.njk
source: https://www.youtube.com/watch?v=OrXKvOjR-vI
speaker: The Pragmatic Engineer
tags:
- technical-debt
- software-testing
- system-observability
- agile-development
- communication-alignment
title: 测试、技术债与“以慢为快”的软件工程实践
summary: 文章探讨了测试作为手段的观点、技术债的偿还时机，以及通过前期规划、明确共识和确保可观测性来“以慢为快”的工程实践，强调避免沟通误解和盲目追求速度。
insight: ''
draft: true
series: ''
category: testing
area: tech-engineering
project: []
people: []
companies_orgs: []
products_models: []
media_books: []
status: evergreen
---
### 测试与技术债的观点
你写过一篇非常有趣的博文，我来引用一下：“测试是达到目的的手段。手段并不重要。”
Original English
You wrote a very interesting blog post which I'm going to quote, "Tests are a means to an end. The means are unimportant."
Original English
>> There is absolutely amazing stuff that you can do with that testing, but in the end, you are still doing something at the late stages in the game. It's true that everybody in the end in some way, whether they admit it or not, is testing and prod. None of the tools that we have are good enough to guarantee whatever it is.
Original English
>> Technical depth only hurts if you have to pay it back. Yes, you can very much get yourself into a position where unfortunately before you even found product market fit, you might have debted yourself into a corner, but the idea that exists in a lot of stuff where it's like, oh no, every single bit of debt is bad. It always has to be paid off. It's not borne out in reality.
Sean, great to have you.
Original English
>> Thank you. We need to start with this crazy fact that we discovered the first company I ever worked for that did professional software, it was called Senset. S E N S E N ne T. What was the first company that you worked for?
Original English
>> Senset. S E N S E N T.
Original English
>> And they're different companies. Yours was in New York.
Original English
>> Yeah.
Original English
>> Mine was in Budapest, Hungary.
Original English
>> Oh, I didn't I didn't know yours was in Budapest. Hey.
Original English
>> Yeah. And rewinding the clock uh a lot before your first job, how did you get into tech? In fact, did you even want to get into tech or was it was music more your thing?
Music was definitely more my thing, but uh I graduated from college and I realized I had no idea what I was going to do and I just started going through uh my address book and the first person I found who said I could come stay with them uh lived in New York City. So I went and a good friend of mine was also living there and uh he eventually he was like I couldn't I couldn't find a job. I got close to getting one which was as a speech writer for a US senator. Uh but I did not get that. And a friend got me a job fixing computers at Scholastic. And then another friend told me, "Hey, if you learn Pearl, you can move to Boston and you can, you know, there was a bunch of people I'd gone to college with who started startup. So I learned Pearl and right about the time that I finished being good enough learning Pearl over the course of three to four months to, you know, do that. uh that startup had collapsed and then it started just be like
Original English
>> there was a dotcom bust right around that time.
No, no, that was that was even before the.com bust. That was a combustible set of human beings uh working together. But yeah, at that point it started being like I I like this. I enjoy it. I'd done it a bit like earlier on uh when I was like a teenager cuz I there was a video game LoadRunner and I I loved LoadRunner and I really wanted to to create things that were like LoadRunner. So I asked my grandmother to get me a book about computer programming, but I didn't know anything about computer programming. my grandmother didn't know anything about computer programming. So, she went to a computer store in uh Fort Worth, Texas, and the guy gave her a book which was uh machine language for the Commodore 64. So, I was like, "Wow, programming is really hard, but yet that's I kind kind of got started that way." And it was the first time I was like, "Oh, there's a thing that people will pay me lots of money for." And I was like, "I have no other thing that people would pay me lots of money for. I should get really good at this.
Original English
>> And then so how did you get really good at at programming? Like you you then worked at different companies, right?
Original English
>> I spent an awful lot of my free time um just trying to get better at it. I think I I leaps and bounds I got interested in uh BOS uh in the Yes. I was really into BOS and um there were some very interesting and helpful people in that community. There was a guy uh all the compiler tools for BOS were made by this company called MetroWorks that also made uh compilers for Mac OS at that time. And for some reason the guy who was the head of their uh their compiler group took an interest on mailing list in helping me learn C++. And there was a there was somebody who was uh knew C++ and was really nice on the mailing list as well who um I know I know basically nothing about sci-fi at all. So it didn't mean anything to me that that person turned out to be Neil Stevenson who
Original English
>> No way.
Original English
>> Yes. Who um
Original English
>> world famous sci-fi author.
Original English
>> Yes. Who basically was just like would exchange like emails with me and advice on on learning like C++ programming which was really early on for me. I mean, there were other people as well, but those were two who really like stood out to me. Like, I've still never read a Neil Stevenson thing. I think I read a couple pages of Snow Crash, but it's it's always a fun story to tell people.
Original English
>> Yeah. And how did you end up at antithesis?
Well, the the short version would be is that Will Wilson, the CEO, sent me a message while I was looking for a new job in December, which was just a link to a thing that they posted on the who's hiring for uh on Hackers Nukes and said, "Would you be interested in this?" It was basically like, "Do you want to do Jeepson style things?" And my response was, "That sounds like it could be fun."
Original English
>> And what do you do at Anticist? Now,
Original English
>> I'm on a team called the Red Team. And I think if you asked everybody at the company what the red team does, you'd get probably different answers from almost everyone. Um, I would I would put it as Will collected a bunch of really smart people together and was like, I want you to use our product the way that customers would use our product. One of the things I've done as part of that is I've intentionally never used the tools which are available to us internally to work with the the product and just done it so that I can only work with it with the exact same tools that uh that customers would have for themselves.
Original English
>> That's kind of a neat concept having an internal team who is a customer team except you know you can talk to them, they will reason with you, they'll they'll explain how the customer thinks.
Original English
>> Yeah. And so we all have a lot of experience and we've been given a lot of free reign to do anything as long as it as long as people were looking at it going, "Oh, that's valuable to the company." So I worked a lot on on skills for helping agents work better with antithesis. But I mean, I think the stuff that I like the most um is just doing stuff to teach customers and teach people internally as well. a lot of the stuff that I've learned over the years for how to how to be sure that your software works well. Um, and that, you know, for me that starts way down the stack, way before anything that anybody calls testing or whatnot. Yeah. And just talking to people about that. I love meeting people. I love talking to people. And it's a subject that I that I really love. I don't know. It's kind of a dream job.
And speaking of dream job, one thing that you've been really going deep into is just different languages which I would call a little bit of exotic languages. Not I'm not talking about the mainstream TypeScript or maybe roster go etc but a few other languages. Can we talk about Okamel specifically? What is Okamel and why are all the hedge funds and finance companies in love with it?
So, Oak camel is a language that came from an academic background primarily uh in France originally. Um and it was it came out at the time when people were trying to combine functional programming with objectoriented stuff. This was a really big deal that people like tried to do particularly in the '9s. At that point, it was bolting objects onto everything. Objects were very much seen as the future etc. And it's interesting because the version of O camel that's actually caught on at places like Jane Street and and other places where it get used a lot is one where none of the O features in O camel really get used.
Original English
>> What is the O?
Original English
>> The O is the object part.
Original English
>> Oh.
Original English
>> And and there's there is a whole series of features which are very object-oriented about it. But I can't think of anyone I know who does O camel who actually really uses any
Original English
of
the
object-oriented
features.
But
it's
it's
a
language
um
which
is
descended
from
ML
which
is
a
entirely
different
languages
of
families
from
like
algo
languages
where
where
like
Python
and
Pearl
and
for
the
most
part
Ruby,
C,
C++
these
are
these
are
all
ALGO
languages
and
then
there's
a
an
the
ML
family
of
which
uh
Okamel
is
is
one
there
there
are
a
number
of
others
in
there.
uh
depending
on
who
you
ask
some
people
will
say
hasll
is
kind
of
in
there
kind
of
not
and
um
all
the
ML
languages
generally
provide
a
a
different
set
of
uh
functionality
and
capabilities
to
people
the
most
famous
I
think
at
this
point
ML
uh
influenced
language
would
be
Rust
there's
an
awful
lot
of
ML
type
stuff
in
Rust
the
the
sum
types
the
pattern
matching
etc
all
of
this
is
stuff
that
came
from
Original English
>> ML
this
sounds
like
a
bit
like
more
of
a
declar
erative
language
or
what
is
the
ML
how
would
you
describe
it
Original English
>> the
ML
languages
are
generally
uh
they're
built
around
strong
type
systems
right
in
the
same
way
that
that
Rust
is
built
around
a
strong
type
system
where
you
take
some
types
which
is
basically
saying
it
could
be
one
of
these
three
things
they're
kind
of
like
enumerations
uh
from
something
like
Java
that
that
people
might
be
familiar
with
or
hey
Rust
is
a
lot
more
popular
now
so
some
people
might
be
familiar
with
it
from
there
and
you
can
combine
that
together
with
uh
pattern
matching
and
and
if
you
can
then
also
combine
that
together
with
exhaustive
matching
um
where
what
it
means
is
that
I
have
this
sum
type
which
could
be
say
five
different
things
right
and
the
way
that
the
way
that
I
will
work
on
it
is
instead
of
an
if
statement
I
do
pattern
matching
but
I'm
matching
on
the
type
that
the
thing
is
[clears throat]
Original English
>> um
and
then
you
can
combine
that
with
exhaustive
matching
what
you
can
end
up
with
is
a
code
base
where
you
can
make
one
change
and
say
actually
now
I
have
six
different
things
of
this
instead
of
five
and
the
compiler
will
tell
you
every
single
place
in
your
codebase
where
you're
not
handling
Original English
>> that
six
type
and
it
it
is
a
gloriously
wonderful
uh
experience.
It
it
it
sounds
like
the
compiler
gives
you
additional
testing
that
you
might
not
get
from
a
compiler
in
let's
say
a
language
like
C
or
or
TypeScript
whatnot.
The
compiler
there
will
tell
you
the
types
but
if
I
understand
correctly
the
exhaustive
matching
is
the
key
here.
And
the
fact
that
there's
matching
it
goes
and
checks
kind
of
maybe
conditions
or
or
assumptions
that
are
in
these
languages
like
TypeScript.
I'll
just
use
it
as
an
example.
It's
kind
of
you
know
you'll
figure
it
out
runtime
if
something
breaks
and
here
in
compile
time
there's
a
lot
of
structure
rules
and
there's
of
course
like
logic
it
goes
through
there.
So
you
kind
of
frontload
a
lot
of
that
validation
if
you
will.
Original English
>> Yeah.
All
all
of
the
languages
which
are
MLike
in
some
type
of
way
or
are
in
that
general
family
are
very
much
languages
where
it's
like
I
want
to
put
as
much
as
possible
into
static
checking
in
the
compiler
and
have
the
compiler
do
work
for
me.
Okay,
so
now
I
understand
why
some
people
say
who
are
into
Rust
that
Rust
helps
you
ship
fewer
bugs
because
it's
an
ML
it's
type
language
which
the
compiler
will
do
a
lot
more
work
and
it
will
have
a
lot
you
stricter
rules
that
you
need
to
become
familiar
with
but
then
in
return
the
compiler
just
like
it
a
lot
of
things
and
it'll
give
you
a
compile
error.
Original English
>> Yeah.
Early
on
when
I
was
learning
to
program,
um
I
was
really
into
like
different
programming
languages
and
just
learning
like
different
things
that
were
possible.
And
uh
fairly
early
on
I
decided
I
was
going
to
pick
one
thing
and
I
was
going
to
dedicate
myself
to
that
for
a
while.
And
uh
the
two
languages
I
ended
up
settling
on
were
uh
C
and
ADA
And
it
was
mostly
it
was
C.
the
the
choice
of
C
came
because
I
was
doing
a
lot
of
programming
for
I
think
it
was
Mac
OS
it
was
either
six
or
seven
I
think
it
was
six
at
that
point
and
Apple
was
converting
an
awful
lot
of
their
stuff
from
Pascal
to
C
at
that
time
so
I
was
like
oh
okay
well
C
could
be
one
that
could
be
interesting
and
then
ADA
was
just
entirely
different
and
Original English
>> what
what
is
ADA
like
for
those
of
us
like
myself
who
don't
know
Original English
>> ADA
is
much
more
in
that
very
very
strict
static
checking
thing
[clears throat]
um
ADA
is
a
language
that
in
a
lot
of
ways
was
driven
by
the
needs
and
desires
of
like
what
the
defense
department
wanted
to
have
for
um
for
knowing
that
there
were
not
going
to
be
bugs
of
of
a
language
which
is
very
deeply
specified
and
will
not
let
you
write
incorrect
programs.
Original English
>> Yeah.
We're
talking
like
literal
mission
critical
stuff.
Original English
>> Yes.
Original English
>> Yes.
Mission
critical
stuff.
NASA,
the
defense
department,
etc.
These
these
were
types
of
people
who
ADA
was
was
aimed
at.
ADA
still
exists.
It
has
a
small
but
robust
market
for
people.
And
the
number
one
thing
that
the
ADA
companies
really
go
after
is,
"Oh,
so
you
like
Rust.
How
would
you
like
to
have
some
more
guarantees?"
Like
[laughter]
that's
that's
that's
where
that's
where
the
ADA
stuff
is.
And
this
is
this
is
one
of
my
great
regrets
was
that
um
I
didn't
really
know
much
about
programming
and
I
didn't
really
know
much
about
programming
languages
and
I
could
not
get
programs
to
compile
in
ADA
Original English
>> in
ADA
because
it's
so
damn
strict,
right?
Original English
>> And
it
was
so
frustrating
and
and
I
could
get
C
programs
to
compile.
It
didn't
really
dawn
on
me
the
fact
that
my
C
programs
were
seging
all
the
time
and
didn't
really
run
for
very
long,
but
it
was
like
thank
you
for
the
cookie
that
it
compiled.
you
know,
now
I
can
move
on
to
the
other
thing.
But
I
spent
years
developing
an
intuition
of
how
I
could
do
stuff
in
C
so
that
my
programs
didn't
die
and
collapse.
And
and
now
all
I
want
is
I
like
I
recognize
like
I
had
all
these
rules
for
myself
of
writing
concurrent
C
and
C++
and
to
keep
myself
from
uh
seging
programs.
And
even
even
though
I
became
really
rigorous
with
that,
I'm
a
human
and
I
screw
stuff
up
all
the
time.
And
I
would
much
rather
have
a
compiler
help
me
with
that.
And
so
there's
a
lot
of
stuff
in
like
O'
camel
and
other
languages
like
it.
Uh
F.
Original English
>> Can
can
we
talk
can
we
talk
a
little
bit
about
F?
Original English
>> Yes.
Original English
>> For
folks
who
have
not
used
F
uh
but
they
might
have
used
C#
or
TypeScript.
How
would
you
describe
F
on
why
is
different?
And
right
it's
it's
a
language
that
Microsoft
created
or
or
they
championed.
And
I'm
not
sure
if
they
created
it.
Original English
>> It
so
it
came
out
of
Microsoft
research.
Um
there's
a
fellow
Don
Sim
who
is
really
responsible
for
everything
F
and
it
basically
started
as
uh
O
camel
fornet
wheren
net
at
the
time
was
just
car
and
then the
net
runtime
Original English
>> and
and
net
is
Microsoft's
framework
where
it
was
really
easy
to
access.
I
remember
Windows
APIs.
It
had
a
lot
of
utility
functions.
It
was
their
everything
framework.
it
basically
their
version
of
the
JVM.
Original English
>> Yeah.
Original English
>> Right.
The
Java
virtual
machine.
Original English
>> Yeah.
Original English
>> And
I
have
no
special
insight
into
it,
but
from
somebody
who
was
around
when
all
those
things
happened,
there
was
a
whole
bunch
of
legal
kurfuffle
with
Microsoft
adding
Windows
specific
APIs
to
their
version
of
Java
and
Sun
threatening
to
sue
them.
Original English
>> There
was
a
lawsuit
and
so
I
think
Anders
Hayesburg
on
this
podcast
shared
the
detail
that
they
needed
to
develop
C
and
I
guess.NET
net
as
well
because
of
a
pending
or
or
active
sun
lawsuit
and
so
they
could
no
longer
use
Java
even
though
they
wanted
to
use
it
and
that
kind
of
settled
it.
So
they
built
.NET.
Original English
>> It
did
and
andnet
was
was
great
because
they
took
like
pretty
much
everything
they
learned
from
the
JVM
and
things
where
ideas
have
been
baked
into
the
JVM
that
would
be
kind
of
hard
to
change
at
that
point.
And
they
did
different
things
which
at
the
time
seemed
like
much
better
ideas.
People
argue
about
whether
they
are
or
not
at
this
point
but
doesn't
really
matter.
But
you
know
net
was
a
very
nice
platform
um
etc
for
everything
and
they
had
one
language
basically
that
was
on
it
which
was
C
which
is
an
awful
lot
like
Java.
It
has
differences
but
fundamentally
the
basic
semantics
of
the
language
are
the
same.
Yeah.
Original English
>> Um
and
so
F
was
a
hey
let's
bring
some
different
semantics
let's
bring
some
different
stuff
to
this
which
was
originally
Original English
like let's try to bring o camel. And I think the thing that's amazing about F is is not really specifically any language features from it. Um because it it has an awful lot of the things that O camel has, but then it doesn't have some. And some people really like to complain about the stuff that it doesn't have who are big O camel fans or whatever. But the thing that F brought in a way that I've never really seen for anything else is it is it fits so well onto net as a platform and it fits so well. It's really the most seamless experience I've ever seen for something like I I did a lot of stuff with Scola and other languages on the JVM. And there are lots of rough edges with with interacting with um Java where F doesn't have that.
而且 F# 在很多方面的推进步伐要慢得多。你可以看到 Don 在这方面做得非常出色，他没有让 F# 盲目向前冲，没有把它做成一种在集成上面临困难、充满棱角的语言。他们的做法就像是：我们要把这门语言往前推进一点点，然后等待 .NET 运行时赶上我们，以此确保一切都能持续正常运行；接着，哦，现在 .NET 运行时已经添加了一些新东西，你就会看到这样一种景象——F# 稍稍往前推进一步，然后显然这又在某种程度上启发了 .NET 团队，两者之间就形成了一种共生关系。而且多年来，F# 中大量非常有趣且酷炫的特性都逐步融入到了 C# 中。在语义层面，真正遗漏的大概就只有带有穷尽性匹配的和类型（sum types）了。根据过去几年 C# 发生的变化来看，最终甚至连这项特性大概也会被纳入到 C# 中。
Original English
And F developed in a lot of ways a lot more slowly. And you can see how Dawn did this beautiful job of not running ahead with F and and making a language which would be hard on the integration and having lots of rough edges and they would be like we're going to push this a little bit forward and we're going to wait for the net runtime to catch up to us so that everything continues to work and oh now the net runtime has added some stuff and it's you can see this thing where it's like ah F pushes forward a little and then oh clearly this inspired like the net team some and there's like a symbiotic relationship who's there and over the years an awful lot of the really interesting and cool features from F have made their way into uh C and about the only thing that's really missing at a semantic level is actually the uh the some types with exhaustive matching and based on the changes that have happened in the last few years with C it really looks like eventually even that would probably be incorporated into C
### Pony 语言的渊源与低延迟需求
>> 还有一门我想向你了解的语言，那就是 Pony。
Original English
>> and then one language which I wanted to ask you about is Pony.
>> 是的。
Original English
>> Yes.
>> 什么是 Pony？
Original English
>> What is Pony?
>> Pony 是一门我参与维护的开源语言。它最初是由我的一位朋友创建的。我们俩当年都上过一所专门招收高中阶段表现不是特别好的学生的大学。是的，招收那些面临挂科甚至高中辍学风险的人，那是一所会接收这类在高中表现不好的学生的早期大学（early college），我的朋友 Sylvan 和我都是这种情况。实际上是 Sylvan 当时对我说：“嘿，如果你去学 Perl……”
Original English
>> Pony is a uh is an open-source uh language that I work on. It was originally created by uh a friend of mine who uh uh we both went to a uh a college for people who had not done particularly well in high school. Yep. and were in danger of say failing out and being high school dropouts where it was a it was an early college that would take people in like that who were not good in high school of which both my friend Silvin and I were like that and Silvin was actually the one who was like hey if you learn Pearl
>> “……你就可以来波士顿，我们能一起做所有这些事情。”
Original English
>> you can come to Boston we can work on all of this stuff.
>> 哦，是的。后来我去了另一家机构工作，我们当时试图为银行构建高速数据处理系统。我们一开始是通过为纽约市的一家大银行提供咨询服务来开展这项工作的，当时我们接手不同的咨询项目，并调研各种技术方案。在我们做这些事情的那段时间，我一直在思考这样一个系统应该长成什么样，我想如何去构建它等等。其中一个关键点就在于，它必须非常快，对吧？这样一来就排除了一整类语言。它不仅需要快，还需要具备确定性的延迟保证，而这在 JVM 本身上是极难实现的。在银行内部，这种要求通常只能依赖特殊手段，要么使用 C++，要么使用 Azul……
Original English
>> Oh yeah. So I had gone off to work at a uh a place that was uh we were trying to build a high-speed data processing stuff for banks and we started off by doing that for doing consulting with one of the big banks in New York City and we were just doing different consulting jobs and we were looking at stuff and around the time we were doing that I was thinking about what a system like this would look like and how I'd want to build it etc and everything and a big part of it was it was like it needs to be really fast, right? So this rules out a whole class of languages and it needed to be fast and have latency guarantees in a way that uh the JVM itself made really difficult and was for and for inside of banks was reserved for specialized things of either using C++ or um the Azul
>> 当你提到 JVM 导致实现困难时，是因为垃圾回收吗？是因为……
Original English
>> when when you're saying the JVM made it difficult was it because of garbage collection was because of
>> 垃圾回收、延迟问题等等。围绕垃圾回收已经有许多改进工作。但这就像你的延迟偶尔会出现尖刺，比如一个原本应该耗时若干毫秒的普通函数，可能会耗费长得多的时间，因为垃圾回收并非完全随机地触发了。
Original English
>> garbage collection latency issues etc there's been a lot of work on garbage collection It's like sometimes your latency will spike like a normal function that should I don't know take this many milliseconds. It could take way longer cuz garbage collection just kicked off not randomly
>> 尤其在那个时期，在面向所有人提供的标准 JVM 版本中，所有的垃圾回收都是“全停顿”（stop the world）机制，从字面意思上讲，就是为了回收垃圾，所有进程都必须停下来。因此，当一个请求刚进来时，你的延迟就会从几毫秒飙升到巨大的尖刺峰值。这在交易系统等场景中会带来极其糟糕的后果。哦，是的。所以 JVM 主要是被用于运行在交易系统周边的系统，也就是……但即使在那种情况下，它依然会引发问题。比如你有一个风险控制系统，它本该实时评估是否应该叫停某些特定标的的交易，一旦它出现那样的延迟尖刺，代价依然可能极为昂贵等等。当时有一家公司，他们开发过——现在也依然在做——Azul Systems，他们做了一款无停顿（pauseless）的 JVM，但它的费用非常昂贵，所以大多数项目实际上并不会使用它等等。
Original English
>> particularly at that time on the standard of uh available to everyone uh JVM versions uh all the garbage collection was stop the world which literally means in order to collect garbage everything has to stop and so you have a request that's just come in and your latency is going to go from yeah milliseconds to huge spikes. This would be very bad in like trading systems etc. Oh yeah. So like the JVM was used primarily for systems that ran alongside trading systems which were you know where but even even then it could still cause problems. You have you have a risk system which is supposed to be evaluating uh whether you should shut down trading on particular things or not. Um and it has a spike like that that can still be hugely expensive and whatnot. So there was a company uh they made a uh they still do um uh Zuul Systems and they made a JVM which was pauseless um but it cost a lot of money so most projects didn't actually use it etc and whatnot
### Pony 的诞生与 Wallaroo 的采用
所以我们当时在考虑，好的，我们要为那些不愿意为这类方案付费的场景寻找替代 JVM 类系统的方案。我们需要某种速度非常快的东西，同时我们必须向客户做出的保证是：我们绝不会丢失你的任何数据，也不会把任何数据搞乱。这意味着需要利用并发来实现速度，在某种程度上要贴近底层硬件（close to the metal）——无论这意味着在数据局部性等方面拥有某种程度的控制，以及所有有助于实现高性能的因素。如果大家对此感兴趣，去了解一下“机械同理心”（mechanical sympathy），Martin Thompson 在这方面做的大量工作可能会非常有启发性。但我们当时就是在寻找这样一组特质：我们需要确保系统能稳定工作，而且对我们而言，确保不搞乱你的数据这一保证至关重要。
Original English
and so we we're looking at okay we're trying to replace the JVM type systems for stuff where people don't want to pay for a and we need something which will be really fast and we also need something where the the guarantee that we had to make for people was we're not going to lose any of your data and we're not going to mess any of it up which is concurrency for speed close to the metal for whatever that means in terms of having some level of control over data locality etc and all of the things that go into being fast um if folks are interested in it looking looking into mechanical sympathy a lot of the stuff that Martin Thompson has done there um can be really interesting but we were looking for this collection of characteristics where we need to know it's working and we and like our guarantees around not messing up your data were incredibly important to us.
我当时心想：我懂 C。我很擅长 C。嗯，我写 C++ 的风格还停留在 1998 年。所以我不确定自己算不算精通 C++，大体上我写 C++ 依然停留在 1998 年的水平。我当时觉得，我们恐怕无法达成所有这些保障要求；而且我坚信，在初创公司的薪资预算下，我绝对招不到能够凭借那样的工具把这套系统做出来的人才。我当时心想，真不知道该怎么办才好，但我们依然继续做着咨询和这些事情。后来发生了一件可以说是领英（LinkedIn）带给我的为数不多的有用事情之一：我有几年没联系过的 Sylvan 给我发了条消息，说：“嘿，看看我做的这个东西。”那是 Pony 的一个早期版本，当时他一直在摩根士丹利（Morgan Stanley）工作。
Original English
And I was like, I know C. I'm good at C. Um I write C++ like it's still 1998. So I don't know if I would call myself good at C++. I still write C++ like it's 1998 for the most part. And I was like, I don't think that we can meet all of these guarantees and I certainly don't think that I can hire people to work on a startup salary who would be able to pull this off with something like that. And I was like, I don't know what we're going to do for this, but we continue to do consulting and this. And for one of the few useful things that's ever happened to me with LinkedIn, um, Silvin, who I hadn't talked to in a few years, sent me a message and said, "Hey, look at this thing that I made." And it was a it was an early version of Pony where he'd been working at uh Morgan Stanley. Y
>> 他构建了一个看起来像是……基本上就是 C 语言的东西，对吧？
Original English
>> and he had built a thing that looked like a it was it was basically C, right?
>> 嗯，它是一个 Actor 模型风格的东西，稍后我们可以深入聊聊 Actor 模型。但它全是用……人们必须在上面用 C 来编写代码。
Original English
>> Um and it was an actor model type thing which we could go into the actor model in a bit, but uh it was all written in people had to write stuff in C.
>> 是的。
Original English
>> Yep.
>> 在其之上开发，而且它附带了 C 语言在并发、段错误（segfault）等方面的所有遗留问题——就像当年我选择它而不是 Ada 时它所带有的那些问题一样。运行时本身运转得非常出色，但在其之上编写 C 语言代码的人必须极其专业，必须把每件事都处理得分毫不差，否则就会把运行时赋予他们的所有优良特性给毁掉。于是 Sylvan 决定：“我要离开去深入学习大量的数学知识，这样我就能在类型系统领域变得非常厉害，然后我要去创造属于我自己的编程语言。”为了完成所有这些，他在英国帝国理工学院（Imperial）攻读了博士学位。那段经历最终结出的果实就是 Pony。他给我发了消息说“嘿，看看我做出来的东西”，那正是 Pony。结合我们在 Wallaroo 想要实现的目标，我审视了它，当时觉得这其实有点……
Original English
uh on top of it and it came with all of the problems that C still had in terms of concurrency and seg faulting and everything that it had way back when I picked it instead of ADA and the runtime worked really well, but the people who were writing the C on top of it had to be really expert and had to do things just right or they would mess up all of the great things that the runtime could give them. And so Silvin decided, I'm going to go off. I'm going to learn a whole bunch of math so I can get really good at type system stuff and I'm going to go off and I'm going to make my own programming language. And he did a PhD at Imperial in the UK to do all of that. And the thing that popped out of the end of that was Pony. And he sent me the thing of hey look what I made and it it was Pony and I looked at it for what we were trying to do at Walaroo and I was like this is actually kind of
>> Wallaroo 是那家初创公司，也就是那家 AI 公司的前身，对吧？
Original English
>> Walu was a startup where it worked at an AI company, right?
>> 是的。Wallaroo 最终发展成了一家 AI 公司。在那个时间节点上，它是一家……我们正试图向银行等机构销售高速数据处理技术等等。至于 Pony，当时根本没有任何人用过它。我们在决定采用它时，所能拿出的最好说辞——当人们问起“那是什么？”时，我们就会说：“噢，其实它最初是摩根士丹利内部的一个构想，这在试图向银行推销时往往能为你争取到极大的回旋空间。”再加上简街资本（Jane Street）当时基本上是全世界 OCaml 的主要使用者，而且在那个时期 OCaml 已经完全与简街资本绑定在一起了。人们就会说：“噢，那你们是不是……”
Original English
>> Yeah. Walloo eventually became an AI company. At that point in time, it was a we're trying to sell high-speed data processing to banks, etc. and whatnot. Pony, nobody nobody had used it at all whatsoever. Um, there were like the best that we could say when we decided to adopt it and people would be like, "What's that?" We'd be like, "Oh, well, it started as an idea at Morgan Stanley, which can actually buy you an awful lot of uh awful lot of room when you're trying to sell to banks." and the fact that Jane Street was basically the primary user of uh of Oakl anywhere in the world and Oakl had become completely associated with Jane Street at that point. People are like, "Oh, so are
“你们会像 Jane Street 之于 OCaml 那样，成为 Pony 的代表吗？”我们回答说：“嗯，如果我们足够成功的话，是的，这确实有点像我们的想法。”所以，这可以说是回答这个问题的一大段背景。
Pony 是一门绝对在某种程度上脱胎于 ML 家族的语言。它和 OCaml 有很多共同点。它和 F# 也有一些共同点。因此，它和 Rust 也有一些共通之处。它的定位非常明确：旨在让编写高度异步的快速并发程序变得轻松，并让我为了防止段错误（segfault）而学到的所有规则，都由编译器为你强制执行，而不是像我那样——我做的话总是免不了把事情搞砸……
Original English
you going to are you going to be like to pony what Jane Street is to Oak camel?" And we're like, "Well, if we're successful enough, yes, that that's kind of the idea." So, and that's a lot of backstory to to answer the question. Pony is a a language which is absolutely in some way descended from the ML family of stuff. It has some it has a lot of stuff in common with O'Amel. Um it has some stuff in common with F. Um it has some stuff in common with Rust because of that. um where it's very much aimed at making it easy to write fast concurrent programs which are highly asynchronous and make it so that all of the rules that I learned uh to keep stuff from seg faulting are things that the compiler will enforce for you instead of well instead of instead of you doing which what I what I would do which is invariably screw something up
Original English
>> um and and have it crash on you.
Original English
>> Yeah, I think you described pulling makes illegal states unre unrepresentable.
Original English
>> A large majority of them. Yes. Which and in that way it has a lot in common with rust like one of the but the uh the model for how it does that in terms of concurrency is very different. I find the actor model which where um which is probably at this point most popular from elixir and maybe that used to be you could say that it was it was Erlang but now it's probably really elixir on that that the actor model makes it uh much easier to reason about uh concurrency and um uh Pony gives you a built-in concurrency model the same way that Elixir and Erlang do in a way that you don't get with uh something like Rust and is very much the type system and everything is very much built around making it really easy to do asynchronous programming. Um, and I think it has just one of the best experiences for doing that combination of things where there are other languages like Rust which are very good at providing you with um uh guarantees and whatnot but aren't necessarily this full kind of uh uh omccas u sort of thing for no actually here's how you do fearless concurrency. You're you're not picking and choosing a bunch of things. We've picked a bunch of things and we put them together for for you.
Original English
And can we talk a little bit of garbage collection? And we just mentioned how it's slow, but I'll quote you in a blog post you previously wrote or this I think this was a talk. It says most engineers thinks garbage collection is slow. They're not wrong about their experience. Stop the world collectors really do destroy all your tail latencies. But garbage collection is slow confuses the symptom with the cause. What is the cause? And I mean the symptom is that sometimes the world stops, right?
Original English
>> Yeah. So well the first thing I would say is that most people when they think of garbage collection they think of like what the JVM does. Um I think of garbage collection as a different thing which is that garbage collection is a mechanism by which your program frees memory that it no longer needs because um there is there is definitely this thing where sometimes people think about you on one side you're doing like malach and free and whatnot and on the other side you have garbage collection but if you want to go fast you you don't malo and free all the time because this is really slow. These are system calls. The first step that people do is they they allocate a whole bunch of memory at once and then they start like slicing off little bits to themselves using like there are things called like slab allocators. They might do arena allocation. There's a whole whole set of memory allocators and then it becomes well I am now managing this memory. How do I know when it's free and when I can you when I can when I can free it etc and whatnot? You're already halfway to what most people think of as garbage collection at that point. And it becomes it becomes a a story of how am I doing these things and what is the what are the trade-offs that these things come with and in terms of the JVM uh all of the memory is basically reachable from anywhere from any thread and the and so it's very hard to be able to free memory in a way that doesn't cause you to have to do a stop the world and the stuff [clears throat] that concurrent collectors like uh Java has the Shenandoa collector Azul has theirs do is actually really impressive engineering for how they managed to do what's called concurrent garbage collection. Pony has a garbage collector, which is instead of having to stop the world, what you have instead is that this is this is a bit of a lie in order to get to the truth without going too deep. But each actor has its own heap is the way that somebody could think about it, which is also how people describe stuff for the Erlang VM Beam, which is also not technically true there, but it makes it easier to think about. and that each actor is responsible for freeing memory that it's used. So you do not you do not garbage collect and stop the entire thing. What you do is instead is that when an actor is done doing some work, it can check to see through some heruristic do I have memory that I can that I can
Original English
>> and with an actor we can think of like as a something like a thread maybe a bit more lightweight than that but it's kind of your own kind of like world that's kind of separate from other actors or
Original English
>> Yep. And so instead of instead of I'm going to stop my entire program in order to free memory, it's like oh hey well I did 10 application operations and now I'm going to do one memory freeing operation. And what you what you end up with with with like pony and with other concurrent memory freeing things that are like that where the things can operate independent of one another is you end up with these things where instead of having these really spiky latency things like people are used to seeing if they run things like Cassandra or whatnot on a standard JVM setup where it's just like whoop my tail latencies are really bad is that you just get basically blat latencies for the most part or or a tiny little bridge. And the thing that you give up on that is that at your fastest moment, you have a little bit less throughput than you might otherwise. And this is still garbage collection, but it's just it's it's an entirely different like take on garbage collection. And and it's it's a really the difference between that and what a lot of people end up doing in sophisticated, you know, memory allocators that they write for themselves and whatnot is that at this point they've completely basically taken themselves as a programmer out of the equation. They've come up with an algorithm or something for deciding when memory should be freed instead of them having to try and code in what the particular rules for it would be.
Original English
So let's talk about software correctness. How do you define software correctness?
Original English
>> I would define software correctness as my program does what I expected it to do at at a at a really loose level. I tend to talk about things in terms of invariance, right? which is a more formal specification term but in in the end invariant or it's it's a guarantee of which even more loosely is my my software is doing what I expect it to do but in a lot of cases people don't necessarily know what they expect their software to do when it's not working but they don't necessarily know when it's working but software correctness is really about like it is a way of approaching software and a way of thinking about it of being like of of knowing knowing that it is it is upholding what it is supposed to do. It is software which is you know which is going to do its business purpose or whatever it was written for and it's not going to wake anybody up at 2:00 in the morning. That that is correct software.
Original English
Now for correct software clearly you want to you know do some testing but you wrote a very interesting blog post which I'm going to quote you. No one cares about testing. That's going to hurt the feelings of the people who do care about testing. But I'm going to stand by the statement. That might sound all coming from someone who has a reputation for talking about testing, but I don't value tests. Tests are a means to an end. The means are unimportant. The end is all that is important. I wrote that
Original English
because I was thinking about stuff where I'm at antithesis and people think of antithesis as a testing company and I don't think of it that way. I think of it as a software verification company and a correctness company. Um, which is very different to me. Tests are things that you put in uh in order to verify things are working the way that you can for things that you had no better way to do it. Like the worst case for a test is manual testing that people do a lot of times. And if you walk backwards from that, you can maybe I can automate this and you can get more and more sophisticated with the testing that you're doing. And there is absolutely amazing stuff that you can do with that testing. But in the end, you are still you are still doing something at the late stages in the game, right?
Original English
Whereas if you go back again to the the C C and ADA thing that like for a decision I made really early on, C gave me almost nothing in terms of correctness. It was just a big giant gun that I was pointed at my own foot and shot my foot off with over and over again. And ADA would give me an awful lot of support for actually constructing correct software from the beginning of not allowing me to do things that were going to be incorrect. The whole reason why my ADA programs wouldn't compile was because they were fundamentally memory unsafe was what I was trying to do and they were going to lead to some sort of memory corruption, a crash, a data data race, any anything like that.
Original English
And uh you know I I believe very much that you should be you should be doing as much as possible before you ever get to testing that you should you should make it so that these things where people are testing to make sure my thing is only is only ever in one of these three states. If you have a if you have a language and ML languages are particularly good um at this sort of thing. you can usually represent within the type system like what the states are, what the allowable transitions are, etc., and make sure that things can't ever transition to a state that you weren't expecting. This is just it's huge because when you're trying to think about does my system work, you're not thinking about the entire thing all the way down. You're just thinking about the layer of stuff that your type system won't allow you to express.
Original English
So I understand this with ML languages that they allow you to do a lot more of this. But could we not play devil's advocate that if you do test different testdriven development, you write the tests first and then you develop, you might do something similar.
Original English
>> So I I love TDD. Um, I think TDD is one of the best mechanisms for writing great APIs that people are going to work with because you're actively looking at how do I actually work with this particular thing. The TDD ecosystem um is very much in love with these things that are called katchas and these are little uh programming exercises that you do and
Original English
>> the daily ones.
Original English
>> Yes. And with there's and there's there's there's one kata that usually comes in different forms uh which is which is always a pair programming one and it is it is one person's job to write the test and it's the other person's job to implement the thing which will make the test pass. And and the thing that you're supposed to do as the person who's doing the implementing is you're supposed to do the simplest thing possible in order to make it pass. And the thing that people when people would do this with me, they quickly discover is I took that very literally. And I would I would just hardcode things over and over and over again into the implementation to make their particular examplebased test, which is what most people do with with TDD and unit tests pass. Right? So if it's like, hey, what's 2 plus 2? And that's the only test they have, I'm just going to have it return four, right? And if they do 2 + three, I can do a little thing that says, "Oh, okay. One of them was two, one of them was three. I'm not even going to do this. I'm just going to return five." The kochi is not supposed to stop until the person who has the test is happy with the implementation, the person who's writing the test. And the only way to defeat somebody who's being very, very, very little at it is that you have to start randomizing the inputs. And the moment that you start randomizing the inputs, great. You're now into the land of what normally what people call property based testing, right? Absolutely. Because you're generating results in order to get it to cover more space than you might come up with for examples.
Original English
>> I mean, if if I was on the other side of that, I I' I'd probably be thinking of like, all right, Sean's being a bit of a jerk here. [laughter]
Original English
>> If you take that and and you make it a bit more profane, that was what people would say to me on a regular basis. Yes.
Original English
be because like basically you're saying that you would go to the efforts of like let's say if I you know like addition right like there's a question of like all right like a sum like you put in numbers and it spits it out very simple to implement right if I start writing the tests and you write the implementation you you you might just literally like start like write edge cases for like oh if this return that if that return this and just just the more tests you write you'll be like I I can keep doing this I'll just add more if statements in there and you're saying the only way to beat you was to either have your tests do random things or invent or come across the the area of property based testing where you can say for this property just run every single one of them. [laughter]
Original English
>> Yeah. Yes. Some something like that. And and yeah, people would get frustrated with me with that. But by the time we were done with the exercise, you we we the one place I worked with we we periodically would do this. would like all the engineers would do this and whoever it was who got paired with me would come out and be like I'm the only one who didn't finish the exercise but they would be like also oh my god I think I'm going to be much better at writing tests now
Original English
>> and it's great and this is this is this is a thing that can lead to much better tests but in the end like awful lot of the things that were being tested were things that you wouldn't necessarily have to test if you had a stronger compiler
Original English
>> and you know that particular place was a Java a place and there there's a type system for Java which can prevent a number of things but still allows an awful lot of freedom in the same way that uh the same way that C allows an awful lot of freedom for years um I was I was I was a person who loved like uh dynamic programming languages small talk is still one of my favorite programming languages I absolutely love it I wrote a lot of Ruby I wrote a lot of Pearl I wrote a lot of Python and I was always like I don't really need the compiler's help for this right and I eventually realized it was well yeah but that's cuz I primarily worked on small systems
Original English
>> where I could hold everything in my head
Original English
>> and I only worked with one system and it was just always fresh of mind and when I started working on bigger systems and at bigger companies where there were more things that I had to work with and there were more people who I was working with more things entered the system where I didn't really know what it because I started to really value like things even even with with Java you could just be like okay well I'm supposed to be replacing this class I'm just going to delete this class and the compiler at least tell me every single place that breaks and I can start to have an idea of where do I need to explore the codebase one thing you wrote about is this thing called workloads and you actually did a talk on this uh and you wrote you're likely missing an important thing in your testing strategy workloads. What are workloads?
Original English
>> So, um, with things like property based testing or
Original English
>> and property based testing meaning that you you say my function here's a property that can be passed. Let's say this is an integer and then your test hardness can like just
Original English
>> if it's an in32, it could literally go through all the integers or it can
Original English
do random or it can do a thousand of them, etc. Yeah. And but once you get past the the simple cases that most people use for property based testing which is like here's a property for showing that that you know you can reverse a list and then you can reverse it again and you have the same thing you started with which is really simplistic but if you want to start applying this to I want to do property based testing on my distributed system or on my database for example
Original English
>> the the workload is whatever it is that you're going to use to validate that the thing is actually working combined with the inputs that you're going to create in order to drive your code in order to in in order to test it, right? As as a simple example, if if you wanted to test a key value store, right?
Original English
>> Then
Original English
>> like Reddus or your own key value store,
Original English
>> you have a key value store. It has three operations, get, put, and delete. Um those are those are all things which are going to be input into the system and are part of your workload. But as an additional part of your workload, you're going to have to decide how many keys am I going to have? What is the range of keys I'm going to look like? How big are the keys? How big are the values that I'm going to put in there? And your workload is this combination of all the stuff. It's your system. It's your things that you're going to use to decide whether it's working. And it's the inputs into the system as well. And it's very easy for people when they read about property based testing and they see these simple examples of like, oh, reverse a list to be like, oh, okay, great. It's randomness. Randomness is going to get this for me. And so a a simplified version of what somebody might do is be like, okay, I have three I have three operations, a get, put, and a delete, and I'm going to pick one of those operations to do. Talking about it in the simplified manner where we're ignoring the fact that you could have different keys and whatnot. If you do that, you're not actually going to trigger too many interesting places in your code because you have an equal chance of you do you do you do a put and then you're probably going to do a delete at some point nearby. And the odds of you doing say 10,000 puts in a row and potentially pushing into areas where your thing might need to start swapping stuff out to disk or or do some type of garbage collection or
Original English
>> or resharding etc
Original English
>> or resharding any of those. They're simply never going to happen because you're just moving right around this little
Original English
>> Yeah. just up and down for like one or two elements which is
Original English
>> it's it's a it's a drunken walk and drunken walks never get very far from where they start. So yeah, the the the workload is very much in in part about like a big part of it is just what are the inputs that can that can drive interesting behavior here. And sometimes for something like the key value one, it might mean well I I need to drive it to really interesting places. But also other other times uh workloads can be deceptively simple and and validate really really really powerful things for the simple side. um because we need a whiteboard if we're going to do the the input side. Um for for the simple side, there's there's there's a there's a workload which is commonly used for testing uh isolation guarantees in asset databases which
Original English
>> I don't know I think is probably the single most important thing that you want to guarantee with a with an asset database and it's called the bank test and it is just about the simplest thing you could possibly it's amazing. So you set up some number of accounts that all exist in a table, right? And you say in this system there's $10,000 and we're going to divide this up amongst these what however many accounts we have. So we say 10 accounts. Each one of them starts with $1,000. And if there's $10,000 in the system and all we're going to do for this workload is we're going to just inside of a transaction, we're going to transfer money from one account to another, which just means moving it from one column to another. And we're going to have another thing that runs at the same time and is just constantly querying what is the total amount of money that's in the system. And if the isolation guarantees hold, then the only value it should ever see is 10,000.
Original English
>> Mhm.
Original English
>> And this is a very simple test
Original English
>> because you keep transferring between one another.
Original English
>> Yep. And and you and you run many different transactions. And the odds are if if I mean if the database is good then in a normal situation where you just run this on your laptop you should never see any any any weird failures. However you take this as like a distributed database and you put it into an environment where you start in introducing faults into it like say introducing network faults in between nodes killing nodes off etc. Um Kyle Kingsbury with his Jeepson stuff made this hugely famous regularly going to you know like the top of hacker news and everything. It's in the same vein of what we do at antithesis as well and all sorts of interesting bugs start popping out from this really simple workload. And it's because it's exercising like a really core constraint there of of the system an invariant which is that I should never be able to read something that's happening in inside another transaction. But the workload itself is is the workload you describing the system. Here's the $10,000. Here's the accounts. Here's how you can move between them or is the workload like one one of these things like moving moving money between accounts and you're like let's repeat this like many many times depending on on how maximalist of a view you want to take, right? You could you could you could be talking about it in a sense where the workload is also the particular database that you're testing in this entire thing because Postgress has a slightly different way of doing those things perhaps than like my SQL than some other than some other database and therefore you could talk about that as a different workload or you could just talk about at a conceptual level the bank test as a workload where you're not even considering the database that's in it or any of the particular implementation details and people regularly talk about workloads from both of those ends from just the the bank test in in in a platonic sense right is a workload right but then there's also the you know the non idealized version of it where there's a particular implementation you know that that exists here
Original English
>> so could I substitute in my mental model workload as a complex test case or a complex series of test cases and like this thing that we're we're using to test right like a workload is is is meant to test something or or or play or replay a scenario or something like that, right?
Original English
>> Usually not replay a scenario. That would be
Original English
>> not replay. Yeah. But like construct a scenario.
Original English
>> So I would avoid the word construct because now when we're we're talking about construct, we're we're getting back in towards a thing that looks like the example based stuff that we're doing with unit test.
Original English
>> Yeah. When you write the specific thing. So, so it's more like the point of the workload is that you set the rules and then it kind of works itself out. Like in in this case with the bank transfer stuff, we never say exactly how we should transfer. We just kind of want transfers back and forth randomly, non-randomly, ideally hit interesting edge cases, you know, just just get creative with it, right? And that is the workload.
Original English
>> Yeah. And and if it's and if it's like if it's like most software that's been around for a while and is and has gone through some sort of rigorous testing, then no bug should pop out of a workload like the bank test. They they shouldn't, right? Perhaps more complicated workload stuff might pop out. But it really depends. But for something like the bank test, it shouldn't. But bugs will start popping out like crazy from something like the bang test when you start subjecting it to like a Jeepson or an antithesis type experience which is also the type of experience that happens all the time in production environments and that like uh Lauren Hawkstein does a incredible job all the time writing up for
Original English
hey
here's
what
happened
in
this
incident
here's
what
happened
in
this
incident
let
me
let
me
as
somebody
who's
like
been
versed
in
this
stuff
for
a
long
time
try
to
break
it
down
for
you
and
One
interesting
thing
you
said
about
quality
and
I'll
quote
you
again
is
is
uh
on
how
quality
is
not
a
gate.
And
so
you
wrote
quality
is
not
a
gate.
The
whole
gate
idea
rests
on
a
belief
we're
dragging
into
the
light.
The
belief
is
that
quality
is
something
you
hold
by
keeping
the
bad
out.
Catch
the
bug
right
at
the
door.
It's
a
comforting
picture,
but
it
isn't
how
good
good
teams
stay
good.
So
this
is
this
is
this
is
really
in
a
lot
of
ways
why
I
don't
like
talking
about
testing
as
a
thing
and
I
like
talking
about
correctness
because
like
Original English
>> for
as
much
as
possible
your
system
for
whatever
it
is
should
not
allow
bad
things
to
happen
period
from
just
how
you
can
write
the
software
and
static
checks
that
you
should
be
able
to
get
like
in
an
ideal
world
you
should
be
able
to
write
a
specification
in
like
a
language
like
TA+
plus
or
or
alloy
or
something
like
that
and
have
a
perfect
representation
of
that
as
an
implementation
or
ideally
just
take
that
and
and
put
it
like
be
able
to
magically
go
from
that
into
an
implementation
which
somehow
also
has
all
of
your
performance
stuff
etc.
We're
nowhere
near
close
to
that
world,
but
that
would
be
an
ideal
world.
And
you
just
simply
know
that
no
bad
thing
can
happen
because
you've
you've
used
a
model
checker
to
prove
this.
And
they
can
do
incredible
things
for
you.
And
we
don't
really
have
that.
Original English
Um,
and
so
another
level
that
we
can
go
to
is
just
static
checking
inside
of
type
systems
for
things
for
for
the
stuff
that
like
Pony
does,
for
the
stuff
that
like
Rust
does,
for
the
stuff
that
Okamel
does.
Um,
in
an
interesting
way,
Erlang
does
a
lot
of
this
type
of
stuff
as
well,
but
it
does
it
without
a
type
system,
but
it
does
it
with
the
actor
model
for
we're
going
to
limit
what
you
can
actually
do
in
terms
of
stuff
where
you
have
to
send
everything
by
messages
and
every
single
message
the
data
that
one
actor
has
is
copied
uh
and
sent
to
another
actor
such
that
there
can't
be
any
data
races
with
the
sharing
of
this
memory
because
it's
never
actually
shared.
It's
just
copied.
And
in
this
way,
we've
also
done
a
correct
by
construction
sort
of
thing
that
you
could
also
get
from
a
type
system
or
whatnot.
Original English
And
we
want
to
have
this
foundation
to
build
upon.
And
the
the
testing
aspect
should
be
for
the
things
that
whatever
our
static
means
that
we
have
available
to
us
just
can't
enforce
because
in
the
end
we
only
have
we
have
programming
languages
and
the
programming
languages
all
come
with
tradeoffs.
and
like
um
the
programming
language
that
maybe
has
the
perfect
type
system
to
represent
everything
that
you
need
to
do.
Well,
one,
it
probably
doesn't
exist.
And
then
two,
even
if
it
did
exist,
it
probably
would
not
have
some
characteristic
that
you
also
needed
as
well
in
terms
of
performance
or
something
else.
So,
you
need
to
settle
some
and
pick
a
language
that
has
better
performance
guarantees
or
or
what
whatever
it
is.
And
then
at
that
point
what
you're
left
with
is
testing
and
and
verification.
Original English
>> Yeah.
But
I
kind
of
want
to
push
back
on
this
like
the
quality
gates
because
the
idea
of
quality
gates
is
like
okay
you
build
a
code
you
have
a
compiler
it
does
some
stuff
the
languages
we
use
for
practical
reasons
will
there
be
a
bunch
bunch
of
bugs
for
application
level
bugs
even
if
the
compiler
was
perfect.
It
couldn't
test
for
the
the
business
logic
being
correct.
And
then
the
the
way
we
used
to
think
about
it
or
or
I
think
a
lot
of
people
still
think
about
it
is
okay
you
write
this
thing
and
you
have
quality
gates
you
have
like
code
review
CI/CD
static
analysis
I
guess
in
production
you
have
observability
and
there's
these
gates
where
like
it's
a
little
bit
like
sometimes
you'll
see
the
Swiss
cheese
model
like
each
layer
will
catch
some
stuff
like
code
review
sometimes
it
might
catch
bugs
although
we
know
it's
not
a
great
place
to
do
it
but
especially
now
we're
not
sure
what
happening
with
code
review
but
it
could
catch
bugs
or
or
readability
inside
of
Google
they
have
readability
review
like
it
will
catch
reasonability
issues
in
in
their
case
and
then
CI
CD
will
catch
some
some
bugs
with
the
test
or
some
some
regressions
that
we've
seen
and
we've
coded
things
for
them
and
so
on.
You're
saying
that
quality
gates
is
is
that
quality
is
not
a
gate.
So
does
that
mean
that
you
don't
believe
in
having
these
I
guess
let's
just
call
them
gates
in
your
release
process
or
you're
just
saying
just
think
about
it
differently
like
think
about
it
from
the
beginning
you're
it
feels
like
you're
taking
back
the
idea
of
quality
to
the
to
the
very
beginning
right
Original English
>> it
very
much
there
and
also
I
would
say
I
I
disagree
with
your
statement
that
uh
business
logic
can't
be
statically
uh
enforced
and
whatnot
um
um
but
let's
let's
leave
that
let's
leave
that
rabbit
hole
aside
for
now
but
uh
additionally
like
Original English
>> testing
particularly
in
the
way
that
it's
it's
normally
done
outside
of
the
Jeepson
and
antithesis
type
style.
Um
Original English
>> the
unit
testing
and
testing
when
they're
run
CIC
is
the
um
honeycomb
has
awesome
shirts,
right?
I
test
in
prod
which
is
Original English
>> observ
platform.
Yeah.
Original English
>> Yeah.
And
and
and
it
it's
true
that
like
everybody
in
the
end
in
some
way,
whether
they
admit
it
or
not,
is
testing
and
prod.
None
of
the
tools
that
we
have
are
good
enough
to
guarantee
whatever
it
is.
Like
even
if
you're
doing
an
incredible
job
with
like
an
antithesis
Jeepson
type
tool
and
you're
thinking
up
all
these
great
and
interesting
inputs,
there
going
to
be
some
data
which
got
corrupted
at
some
point
in
time
which
exists
in
your
production
system
which
you
haven't
randomly
hit.
your
generator
simply
can't
generate
whatever
it
might
be
that
is
going
to
happen
in
production
and
we
do
not
know
it's
out
there
and
we
are
we
are
testing
it
and
the
idea
that
somehow
we
have
this
this
gate
where
it's
like
oh
if
it's
gone
through
this
it's
quality
it
can
go
out
is
just
it's
it's
kind
of
a
lie
right
like
we
have
hopefully
increasing
levels
of
assurance
as
we
go
along
but
we
don't
actually
have
a
gate
where
we
know
it's
quality
Original English
>> so
but
you're
you're
saying
like
these
things
can
help.
It's
just
like
know
that
there
will
be
holes.
Like
Original English
>> yes,
there
going
to
be
holes.
We
have
We
have
no
Original English
>> things
will
sink
through
and
adding
another
gate.
You're
saying
there's
no
one
gate
that
you
could
add
that
will
be
like,
"All
right,
this
now
seals
off
all
the
remaining
holes."
As
long
as as
long
as
we're
looking
at
these
kind
of
gates
of
like
one
more,
you
know,
like
the
same
the
same
as
if
if
you
add
one
more
test
or
two
more
tests
or
three
more
tests,
when
you
like
hardcode
the
values,
those
will
they'll
make
it
better,
sure,
but
they
will
they
will
not
guarantee
quality.
So
most
of
the
places
I've
been
where
there
are
teams
which
I
think
are
what
I
would
call
low
functioning
which
are
ones
where
they
tend
to
move
slowly,
they
don't
deliver
stuff
a
lot
or
they
deliver
stuff
quickly
but
like
they
regularly
have
to
do
emergency
releases
to
fix
stuff
etc.
are
people
who
think
about
like
some
sort
of
thing
around
traditional
CI/CD
stuff
now
at
least
traditional
now
that's
built
around
like
pull
requests
or
or
or
a
staging
environment
where
they
run
some
tests
before
it
goes
out
to
prod
that's
generally
how
they
think
about
it
is
like
oh
this
is
this
is
where
I
have
these
quality
gates
and
and
the
people
who
move
quickly
and
with
confidence
and
then
it
actually
holds
up
and
they
you
know
they're
not
getting
alerted
all
the
time
are
people
who
think
about
quality
from
the
very
beginning
of
sitting
down
and
being
like
oh
hey
I
should
validate
for
example
uh
if
I'm
doing
a
thing
which
is
a
distributed
database
which
is
particularly
one
I
should
be
using
something
like
a
TA
plus
to
make
sure
that
my
consensus
algorithm
actually
works
etc
and
everything
and
to
make
sure
that
I
don't
have
bugs
in
that
or
to
take
a
Jeepson
antithesis
type
tool
and
put
the
very
first
version
of
your
stuff
before
you
ever
ship
it
to
production
in
there
and
So,
does
it
behave
like
I
expect
it
to?
Because
it's
really
easy
to
fix
stuff
early
on
when
you
Original English
first
create
the
thing.
And
it's
really
hard
if
you
go
like
which
I've
seen
many
places
do
where
it's
like,
"Oh,
here's
our
new
system."
And
they
don't
really
do
any
testing
on
it
until
like
somewhere
way
down
the
line
and
nobody
has
any
idea
whether
it's
like
actually
working
or
not
or
even
necessarily
any
idea
to
know
what
does
working
really
mean.
like
they
haven't
asked
themselves
from
the
beginning
how
will
I
know
this
thing
is
working
Original English
>> but
uh
I
want
to
push
back
on
like
you
know
that
you
can
just
write
a
test
if
it's
working
I'll
give
you
an
example
where
it
might
be
really
hard
to
write
test
that
it's
working
a
Google
search
I
type
in
the
best
testing
tool
in
the
world
right
Original English
>> y
Original English
>> and
if
I
ask
10
people
and
five
of
those
work
at
vendors
they
all
say
something
different
should
be
the
number
one
right
the
people
who
work
at
vendors
they
will
say
my
vendor
should
be
Number
one,
obviously
I'm
working
here.
We're
doing
amazing
stuff
in
in
testing.
Some
people
will
think,
oh,
like
they'd
like
the
Wikipedia
article.
Some
people
will
say
like
I
think
this
this
should
happen
Original English
>> and
and
of
course
when
I
run
the
same
Google
search
day
after
day,
those
things
will
change
a
little
bit
like
the
the
results
will
change
and
we
can
assume
that
Google
is
is
knows
what
they're
doing
and
and
it's
correct.
But
now
if
I
wanted
to
just
even
write
a
test
for
for
a
complex
system
and
of
course
we
know
the
reason
it
changes
because
Google
calculates
the
page
rank
and
they
take
a
bunch
of
signals
and
they
also
take
things
like
real
time
so
like
news
will
be
fresher
but
now
we
have
this
really
complex
system
and
if
I'm
I
would
be
given
the
simple
task
of
like
well
write
some
tests
so
that
we
can
tell
if
this
small
segment
of
Google
search
works
fine
let's
say
inside
of
Google
I
would
now
struggle
because
I'm
like
well
you
know
if
we
went
like
all
right
you
can
use
any
any
formal
language
or
or
TLA
plus
Original English
>> I
would
struggle
to
code
like
what
am
I
expecting
even
though
when
you
look
at
it
you
can
kind
of
squint
and
say
like
this
looks
good
enough
or
if
you
have
enough
data
Original English
>> but
but
there
are
some
things
that
we
that
we
we
we
can
test
for
example
there's
some
invariance
we
can
do
which
like
for
example
I
put
a
thing
in
the
box
that
I
submitted
I
expect
to
get
results
back
Original English
>> that's
one
so
that's
one
one
invariant
for
example
Original English
>> there
is
there
is
an
invariant
there
and
then
we
start
getting
into
things
and
you
know
I
did
the
machine
learning
stuff
for
a
while
which
was
actually
in
recommendation
systems
and
that
gets
that
gets
into
the
interesting
thing
of
you
like
oh
well
what
should
the
number
one
ranking
be
right
and
that
is
that
is
a
that
is
a
very
different
thing
than
is
it
working
right
like
that's
very
different
is
it
working
is
that
I
put
a
thing
into
the
box
and
I
got
something
back
the
the
is
it
relevant
is
an
entirely
different
thing
like
Original English
>> oh
and
then
with
the
is
it
working
we
can
now
ask
questions
like
if
if
if
I
submit
like
a
thousand
requests
in
a
second
like
is
it
still
working
if
if
if
we
have
concurrent
requests
coming
is
is
it
working?
If
in
the
middle
of
submitting
a
request,
I
submit
a
second
or
navigate
away
and
so
on.
So
like
so
now
you're
saying
we
can
decompose
this
into
different
parts
of
the
problem.
Original English
>> And
are
the
results
that
I'm
getting
results
that
are
for
my
actual
query
Original English
>> where
where
what's
in
it
might
not
matter,
but
we
we
can
we
can
set
up
tests
so
that
each
query
gets
some
type
of
marker
that
goes
along
with
it.
And
we
make
sure
that
those
actually
came
back
and
that
the
server
didn't
get
confused
and
give
me
your
results,
which
okay,
maybe
for
search
I
don't
know.
I
don't
know
how
any
would
ever
know
what
happened,
but
I
think
everybody's
heard
about
the
incidents
where
uh
like,
you
know,
the
ex
service
started
showing,
you
know,
somebody
else's
account
to
people
and
whatnot.
That's
a
really
bad
one
there.
You
know,
as
somebody
who's
had
their
identity
stolen
three
times,
I
really
hope
that
that,
you
know,
nobody's
showing
my
PII
to
uh
somebody
else
accidentally.
Original English
>> So,
I
I
want
to
talk
about
some
stuff
that
are
interesting
learnings
in
the
industry,
and
I
know
you're
passionate
about
them.
One
is
supply
chain
security.
Original English
>> Yes.
Original English
>> What
what
is
supply
chain
security?
I
mean
I
I
think
we're
hear
hearing
it
a
lot
more
unfortunately
especially
in
open
source,
right?
Original English
>> Yep.
So
basically
supply
chain
security
is
talking
about
if
if
you
have
something
which
is
popular,
right?
Then
the
easiest
way
to
compromise
that
thing.
We're
talking
software
open
source
software
Original English
>> software
software
Original English
>> is
to
find
the
weakest
link
for
it
and
to
attack
that
right
which
and
this
this
applies
to
software
but
this
also
applies
to
hardware
for
example.
Original English
>> Oh
Original English
>> right
where
there
can
be
all
sorts
of
security
guarantees
that
like
um
AMD
makes
where
they
have
uh
trusted
uh
computing
modules
which
are
in
there
which
Intel
makes.
But
if
you
if
you
really
want
to
attack
those
systems,
what
you
want
to
have
is
somebody
who
when
they're
putting
this
together
can
put
a
malicious
firmware
onto
those
machines.
How
do
you
know
how
do
you
know
that
that
is
actually
like
the
firmware
that
is
supposed
to
be
and
you
know
I
I
have
I
have
a
security
background
as
well
and
this
this
gets
talked
about
as
like
the
trusting
computing
base.
But
people
usually
talk
about
this
um
when
they
talk
about
supply
chain
security
at
this
point
now
usually
about
software
and
things
like
wasn't
left
was
leftpad
a
supply
chain
security
one
years
ago
where
somebody
was
like
it
was
like
this
little
module
that
was
on
mpm
that
did
something
and
somebody
put
like
a
crypto
miner
or
something
in
it.
Original English
>> It
was
typically
like
I'm
not
sure
how
trivial
it
is
but
like
it's
replacing
a
package
that
is
widely
used.
There
was
some
very
simple
package
that
was
doing
some
trivial
stuff.
Original English
>> But
like
so
a
a
great
way
that
this
could
happen
for
example
if
you
can
go
after
something
which
people
use
and
they
don't
think
about
too
much
particularly
say
in
like
a
CI/CD
system
and
you
can
get
your
code
running
inside
of
the
CI/CD
system.
You
now
have
access
to
all
of
the
secrets
that
are
running
inside
that
CI/CD
system
which
means
that
you
can
start
publishing
software
which
might
have
malware
in
it.
who
knows
what
else
and
whatnot.
Great.
That's
that's
one.
But
even
more
so
like
almost
every
single
programming
language
that
exists
right
now.
Just
it's
basically
it's
basically
like
here
you
go.
There's
something
I
downloaded
it
off
the
internet.
Maybe
I
maybe
I
checked
its
hash
or
whatnot
and
it
can
do
literally
anything.
I
downloaded
a
thing
which
is
supposed
to
pad
out
some
text,
but
it's
actually
allowed
to
I
don't
know
run
a
crypto
miner,
make
make
network
connections,
etc.
and
everything
and
and
all
of
these
these
are
these
are
supply
chain
attacks
and
they
can
have
really
devastating
effects
Original English
>> and
they're
happening.
We
saw
it
in
mpm.
I
think
we
we're
I
think
we're
seeing
it
in
the
Python
community
as
well.
There's
a
quite
good
question
of
what
are
efficient
defenses
because
you
know
like
there's
a
couple
of
naive
defenses.
One
is
oh
let's
just
have
a
cool
down
period.
Let's
not
get
any
new
new
packages
for
at
least
a
week
and
we're
hoping
that
if
there's
a
supply
chain
attack
it
will
be
big
enough
that
it'll
be
published
and
it'll
be
resolved.
Of
course
the
downside
is
that
if
there's
a
hot
hot
fix,
there's
literally
a
security
vulnerability.
We're
now
leaving
ourselves
open
for
a
week.
But
that's
kind
of
one
naive.
The
other
very
naive
is
let's
get
rid
of
our
dependencies.
Let's
just
bring
it
in
a
house
or
let's
and
then
the
third
one
is
let's
fork
it.
Let's
have
this
this
fork
and
maybe
we
can
have
a
security
team
who
can
decide
to
like
update
or
merge
or
keep
it
in
house.
But
these
are
the
kind
of
like
very
all
of
them
are
very
naive
and
Original English
>> and
I'm
I'm
wondering
what
are
you
seeing
work
and
being
practical
cuz
you
it
seems
you're
trading
off
kind
of
latency
of
being
able
to
get
fresh
packages
which
often
have
either
critical
fixes
or
really
good
functionality
for
being
afraid
that
what
if
that
thing
is
poisoned
Original English
>> and
and
and
if
everybody
decides
to
do
the
cool
down
period
who's
going
to
notice?
Original English
>> Exactly.
We
haven't
even
talked
about
Original English
that,
right?
Original English
>> Yes.
Um,
so
I
think
part
of
this
is
is
that
there
is
no
good
answer
for
almost
every
single
programming
language
that
exists
and
any
of
the
ones
where
they
might
have
a
good
uh
answer
to
this
um
are
ones
that
nobody
uses
because
it
it
it
really
has
not
been
a
concern
for
a
long
time.
the
way
that
pretty
much
almost
all
package
management
and
almost
all
open
source
does
dates
back
to
like
the
way
that
Pearl
did
with
CPAN
and
which
was
a
which
is
an
incredibly
open
and
trusting
thing
in
the
same
way
that
almost
all
early
internet
stuff
was
where
it's
really
easy
to
spoof
somebody
sending
an
email
all
of
this.
It
was
it
was
a
network
that
just
assumed
everybody
was
a
good
actor.
Original English
>> Yeah.
And
I
guess
back
then
there
was
a
higher
bar
for
entry
as
well.
Like
if
you're
a
malicious
actor,
you
at
least
needed
to
learn
to
code
or
you
need
to
hire
someone,
which
again
is
now
a
little
bit
trickier
because
once
they
know
what
we're
doing,
it's
a
higher
barrier
to
entry.
But
but
that's
changed,
right?
Even
the
barrier
to
entry
has
changed.
Original English
>> But
e
even
back
then
it
wasn't
true.
One
of
the
one
of
the
people
who
was
a
high
school
messup
who
I
went
to
college
with
was
somebody
who'
been
arrested
by
the
US
government
for
unleashing
when
he
was
a
teenager
a
worm
like
across
a
whole
bunch
of
government
computers
that
he'd
gotten
into.
and
and
and
that
actually
got
further
out
and
was
one
of
the
one
of
the
big
incidents
at
at
that
time.
So
like
I
mean
it
was
still
happening
then
it
was
just
it
was
at
a
smaller
scale
and
not
everybody
was
depending
on
these
things
at
the
this
particular
point
in
time
and
and
the
fundamental
thing
is
that
for
any
dependency
pretty
much
for
any
language
like
JavaScript
or
Ruby
or
or
or
Rust
or
anything
that
that
comes
in
for
the
most
part
basically
the
code
that
you
wrote
has
the
exact
same
capabilities
for
what
it
can
do
as
the
code
that
you
happen
to
bring
in
from
the
internet
and
your
code
can
do
absolutely
anything
probably
and
that
other
code
could
do
absolutely
anything
as
well.
And
so
how
are
you
possibly
going
to
vet
all
of
this
code
if
you're
going
to
have
all
of
these
dependencies
which
gets
back
to
the
to
that
problem
and
in
the
end
nobody
is
going
to
solve
this
using
the
existing
stuff
we
have.
people
might
try
to
do
sandboxing
or
whatnot,
but
that's
just
in
the
end
like
every
sandbox
can
be
escaped
from.
It's
usually
not
that
hard
and
whatnot.
And
there's
a
whole
area
of
uh
study
which
which
is
in
these
areas.
Uh
uh
there's
a
guy
named
Mark
Miller
who
developed
a
language
called
E
which
has
some
stuff
called
object
capabilities.
Um
where
there's
a
lot
of
interesting
research.
We're
we're
here
right
now
as
as
part
of
bug
bash
in
in
Copenhagen
and
the
first
talk
of
the
day
was
uh
from
Sophia.
She's
a
professor
at
uh
Imperial
which
was
actually
talking
about
like
some
object
capability
work
etc
they've
done
but
the
basic
idea
of
object
capability
is
that
in
order
to
take
any
action
within
the
system
you
have
to
be
given
the
capability
in
some
fashion
to
do
it.
So
that
the
system
would
not
allow
you
for
example
to
open
uh
a
network
connection
unless
you
have
the
capability
to
do
that
and
the
originating
program
your
code
that
is
written
uh
has
to
give
that
capability
to
other
stuff
and
there
there
you
know
there's
no
popular
languages
that
do
this
um
because
you
know
because
pony
is
written
by
like
Silvin
initially
and
myself
now
the
object
capabilities
are
built
in
but
this
is
this
is
a
this
is
one
means
to
like
to
be
able
to
have
it
be
like
I
can
trust
this
code
more.
But
but
you
you
need
to
go
you
need
to
go
further
than
that
because
if
your
if
your
programming
language
allows
for
some
type
of
foreign
function
interface
which
almost
all
of
them
do
so
that
you
can
utilize
like
CC
code
or
or
rust
code
or
anything
using
a
Cabi
um
if
you
don't
have
some
way
to
control
that
completely
then
it
doesn't
matter
if
you
have
object
capabilities
because
somebody
can
just
go
out
the
uh
the
FFI
door
and
all
of
these
are
things
which
need
to
be
addressed
and
there's
no
way
that
these
things
will
be
addressed
by
just
incrementally
doing
stuff
within
the
system
that
we
currently
have.
Original English
>> Well,
it
sounds
like
there's
a
reason
that
software
security
is
getting
more
important
and
it's
a
complex
topic.
I
we
did
a
podcast
episode
on
software
security,
but
it
felt
we
were
just
scratching
the
surface
cuz
as
with
anything,
right,
it's
it's
a
nuance.
It's
it's
a
detail
where
it
gets
complicated
and
and
I'm
sure
there's
so
many
strategies.
every
every
company
and
team
decides
on
what
strategies
to
do
from
the
very
simple
when
you're
a
small
startup
and
or
well
you're
just
yolo
at
first
and
then
and
then
you
start
to
do
again
a
simple
thing
is
just
a
cool
down
hoping
that
not
everyone
cools
down
and
it'll
be
okay
and
then
it
gets
a
lot
more
nuance
scanning
for
dependencies
evaluating
them
reducing
unnecessary
ones
scanning
dependency
chains
and
so
on
like
you
can
go
really
deep
you
can
go
to
go
all
to
point
to
have have
a
security
team
who
will
vet
them
or
try
to
verify
them
or
or
even
maybe
have
contact
with
the
know
who's
building
it
and
so
on
and
so
forth.
Original English
>> And
the
the
really
horrifying
part
of
all
of
this
is
we
could
do
all
of
these
all
of
these
things,
the
ones
I've
mentioned
and
even
more
and
none
of
it
in
the
end
would
be
a
really
good
defense
because
we
made
a
we
made
a
tradeoff
years
ago
in
order
to
go
faster
with
our
CPUs,
which
is
we
introduce
speculative
execution.
And
speculative
execution
works
because
it's
like,
hey,
I
have
extra
cycles
and
this
thing
could
go
down
this
branch
or
it
could
go
down
this
branch
and
I'm
going
to
execute
both
at
the
same
time,
which
has
given
us
the
vast
majority
of
the
speed
ups
that
we've
gotten
for
like
the
last
Original English
>> over
20
years
at
this
point.
Original English
>> Um,
and
and
people
have
probably
seen
stuff
for
like
side
channel
attacks
is
generally
what
they're
called.
And
there's
a
whole
variety
of
these
because
the
thing
is
is
that
those
speculative
branches
have
to
actually
exist
in
memory.
And
so
if
I
am
good
enough
as
an
attacker,
I
might
you
you
executed
a
branch
for
what
happens
if
I'm
authorized
and
what
happens
if
I'm
not
authorized.
And
it
turns
out
for
a
sufficiently
like
knowledgeable
person
which
doesn't
actually
take
that
much
knowledge
to
get
even
if
they
don't
actually
have
the
access
to
the
system
can
use
a
side
channel
attack
to
read
the
memory
that
was
created
for
this
is
what
the
access
actually
is
or
even
better
they
can
find
like
they
can
they
can
get
the
the
keys
out
of
out
of
memory
that
are
used
for
security
etc
and
whatnot.
So
we
could
do
everything
else,
right?
And
as
long
as
like
I
worked
at
I
worked
at
Microsoft
research
on
the
confidential
computing
group
and
we
built
uh
a
confidential
container
offering
which
is
which
is
still
there
at
Microsoft
and
the
thing
that
we
always
said
about
our
security
guarantees
which
were
great
was
we
tacked
on
at
the
tail
end
modulo
side
channel
attacks
which
is
we
can't
protect
you
against
side
channel
attacks.
Every
single
guarantee
that
we
have
is
violated
by
side
channel.
And
what
is
confidential
computing?
Original English
>> Confidential
computing
is
computing
where
basically
the
end
goal
is
that
the
software
can
be
run
in
an
environment
where
the
operators
of
the
environment
are
not
trusted
Original English
>> because
in
in
you
play
something
on
the
cloud
and
you're
not
sure
who's
running
that
cloud.
years
ago,
there
was
a
thing
that
that
happened
where
it
turned
out
uh
some
Twitter
employees
were
actually
uh
working
as
agents
for
the
Saudi
Arabian
government
and
were
spying
on
the
activities
through
DMs,
etc.
and
stuff
that
uh
that
Saudi
dissident
were
doing.
it
it's
it
could
be
this
sort
of
thing
or
it
could
be
the
thing
where
you
know
oh
you
might
trust
Microsoft
or
Amazon
and
you
don't
think
they're
going
to
steal
your
data
but
actually
you're
trusting
everybody
who
has
their
access
which
means
that
you
're
trusting
every
employee
not
to
get
fished
or
you're
entrusting
every
employee
not
to
have
a
money
problem
where
they
could
be
bribed
etc
or
whatnot
and
so
confidential
computing
is
an
attempt
to
make
it
so
that
you
don't
have
to
trust
those
people
because
in
security
the
more
things
you
trust,
the
Original English
less secure that your system is. Yeah. Right. And so, yeah, in the confidential computing stuff, we could provide these great guarantees for people except for side channel attacks,
Original English
>> which could which can violate all of the guarantees like the only the only way to possibly to fully lock it down at that point would be is to say that speculative execution is going to be removed from the system. I don't and when the first speculative execution attacks started coming out like they patched the Linux kernel to handle just one individual case and I don't remember the exact numbers but it was like oh like here's a I think the first one they did was like a 50% performance drop and they tried to like claw back performance from that and then but it was like oh and no here's another attack and here's another attack and here's another attack and at a certain point I think people just kind of realized well where we're just going to play whack-a-ole because we don't we don't want to give up on the um on the performance we can get from that. And I'm very concerned about security, but I also in open- source stuff I write, I absolutely do things where I'm taking advantage of speculative execution. And the side of me that wants to make stuff go fast loves it because I can just be like, "Oh yeah, the branch predictor, which works off of this and knows that normally it goes down this path, will make these this branch, which used to be really expensive, really cheap to do, and this means I can do things that would be really hard to make go fast now and make go fast." And we've all kind of made that like that deal of we're willing to go fast, but oh, it's just fundamentally insecure.
Original English
I wanted to ask you about tech depth. Uh 10 years ago, you wrote something and I'll read it back to you and I I want to I'm curious how your thinking might have changed. You you wrote this about a horror story at a at a company or horror stories that you've heard.
Here's the thing, though. We were making money even though these kind of horror stories were happening. We were growing and supporting ourselves. We were continually creating value for the company. Yeah, we had a lot of technical debt, but we kept finding inventing ways to sidestep having to pay it. The debt metaphor we've adopted is misleading. You don't always have to pay off the debt you created. You can work around it and avoid ever having to pay it off. Technical debt only hurts if you have to pay it back.
Original English
>> Yes, I I I 100%.
Original English
>> Still stand by it.
Original English
>> Still still
Original English
>> I mean I am very much a person who wants to build malleable software etc and whatnot. Malleable as in
Original English
>> malleable as in stuff which will can adapt to like the new situation because the situation within which software was built is not going to be the situation that's going to be in a year in two years in 3 years etc. And so it needs to be stuff which can be made to change and that doesn't necessarily mean change the code but can be made to fit like the new environment that it needs to be in where one of the answers to that is often times just throw the thing away. That particular one I believe was motivated by one where um there was an awful lot of debt that was stacked around how the database was set up and a limit to sharding that could happen from my SQL and it getting harder and harder. And that ended up being completely sidestepped by throwing money at a uh a distributed uh a distributed database which still made it look like a single database. The trade-off there was one you pay we paid them a decent amount of money and two there were weird edge cases in terms of like the actual like guarantees that you would expect from like a single node my SQL system you wouldn't actually always get but they were so very rare that it's like this is not a company which is using ADA um
Original English
>> etc. So this incredibly rare thing which I would occasionally track like things down to like is is not that important. And you could just completely sidestep it in a lot of cases. You know, it could be, hey, the business has changed and that system is no longer needed. Like it ran for 3 years, it's not needed anymore. It's gone. You never had to pay off any debt that you you had.
Original English
So, so you're you're you're saying that maybe a lot of us are not thinking about tech depth the way that actually happens, which is you often create it like you just do a bunch of hacks or or just like stuff that, you know, would not make sense, let's say, at a massive production company, but maybe you're a startup or early stage and then later on you might never have to pay it because you might just either write new software or just introduce a new piece of technology that just invalidates it
Original English
>> or not find product market fit
Original English
>> and
Original English
>> run out of runway. And then why did it matter?
Original English
>> Yeah. Yes.
Original English
>> Yeah. I I still remember when I worked at Uber, I was on a rides team or like our we were building a rides product which was Uber's mature product. It's been around for years. It was making serious money and the Uber Eats team was just starting out and one of the engineers came into our codebase and like committed a change cuz they needed the shared functionality. they needed to use some of that platform and they came in uh they deleted a test that was failing after they made the change and they went out and they made the change and we're like what are you doing like you cannot delete the test and we got into the standoff they were like well look I just need to just need to get it done or it wasn't even neat it was it was jump jump bikes and at some point I started to understand that actually like these people like they they really are thinking only about tomorrow cuz they're not sure if they're going to be around the week after and sure enough Uber sold up that whole division 6 months later and it didn't find I guess product market fit inside of Uber. It became its own company. But I was like, "Oh, okay. I guess when you're thinking like that, you actually use one of the lead test like you're like works. Let's roll it out. Let's see if they didn't even care if it worked for 100% of customers. They cared if it kind of worked for their purposes for at that point."
Original English
>> Yeah. And and you know, sometimes you really do pay the cost, right? You do. And usually you're paying the cost because you were successful. However, you could also I mean it could also happen that people are not paying attention to they're not paying attention to debt, but they're also not thinking about it in a way of of what is the most malleable way to do this particular thing which I can so that I can do it quickly in order to validate things because it's very different between I'm just going to hack this thing in here versus a I'm going to come up with a thing where I'm going to do it quickly as possible when it's not the full thing. It fits some tiny little sliver perhaps and there's all sorts of edge cases that it won't handle, but it's easy for me to remove and if we build five more things on it like in the next 3 months, it's not going to slow us down because ultimately we are we are a startup for example that is just trying to figure out what we're doing as quickly as possible. And yes, you can very much get yourself into a position where unfortunately before you even found part of market fit, you might have debted yourself into a corner. But the idea that that exists in a lot of stuff where it's like, oh no, every single bit of debt is bad. It always has to be paid off is just I mean it it's not borne out in reality.
Original English
>> So one one theme of both what we're talking about, but as we're talking earlier about your career and how you built software is how you love to go fast.
Original English
>> Yes.
Original English
>> and that you've learned to go fast. Sometimes you need to go slow. And you you write about this as well, so I'll quote you on this. Go slow to go fast. It means writing a planning document and coming to agreement on what we're doing before anyone touches code. It means knowing how we'll test the thing before we start building it. It means taking a few hours or a couple of days to throw away spike on the parts we don't understand yet. It means stopping when
Original English
production is on fire at 2 a.m. Some of these are are so counterintuitive, right?Original English
>> Not to me.Original English
>> Not to you.Original English
>> Not to me. I mean, I think they absolutely probably were counterintuitive like earlier in my career and everything where it was just like, "Oh, no. Need to get that thing done." But it it's kind of like the thing where where eventually you realize actually working 80 hours every week is not resulting in me going faster. It's just resulting in me making more mistakes because I'm tired all the time and I'm constantly now spending my time tracking down bugs instead of working on new functionality. That really when I get tired I should just step away from the computer. That is a I'm going to go slow to go fast. I'm going to make sure that I'm rested, that I'm healthy, that my my brain, which is a key part of all of this, is actually functioning correctly. One of the worst time wasters that I' I've seen in my career is people not spending time to make sure that when they're talking to each other and using words that they actually mean the same thing.Original English
>> And like 3 weeks later the thing pops out the other end and somebody looks at it and goesOriginal English
>> that's not what I wanted. Like this is this is why like with agile that this idea of sprints was supposed to bake in this we do not go very long without looking at it to make sure that we are in agreement for the direction that we're going in. If you're going to build a big complicated thing, it behooves you to spend a few days where everybody just the first part of development is not writing code, but it's just writing down how the thing is going to work, what it's supposed to do, and making sure everybody actually agrees on all of these things because it's very easy to change things there. You haven't invested a lot of time in coding the thing, etc. you're just working to make sure are we all actually talking about the same thing. Years ago, there was a a wonderful argument from people on the internet in the distributed systems community where the two sides were just going back and forth at each other and they were going back and forth at each other because they were talking about what exactly once means in terms of like delivering messages.Original English
>> Exactly once delivery. Yeah.Original English
>> Yes. and and like one group of people meant exactly once processing and they were talking about Iden potence and a thing only actually having an impact one time and the other group was talking about the message literally arriving at a computer only once and these people over here are like you're liars you can't do exactly once like and the and they just spent all of this time yelling at each other because nobody actually bothered to sit down and go this is what I mean by exactly once and the same sort of thing like can come up like all the time you send somebody off it's like oh hey I jotted down this little thing on a card I guess people use like linear they probably still use Jira whatever it is now and I jotted down this little thing you go pick up the thing you do it and then like this person spent I don't know an hour a day 5 days building this thing and it's like oh yeah no I that's not what I that's not what we were expecting and you went fast and you got nowhere, right? Like you spend the time to establish the foundation for whatever that is. Make sure everybody's pointed in the same direction. And as part of it, like for example, if you want to go fast later on, you should be thinking when you're designing a thing for how you're going to know that it's actually working right from the beginning. Because there are ways you can build a system where it's almost impossible to know whether it's working. Or you can build it where it's easy to know that it's working, which can include some of the stuff I've talked about from static stuff. But even just making it so that you have good observability into the system, right?Original English
>> Like we're building a system that we're going to put out there in production. How are we going to know whether it's down? How are we how are we going to know whether it's got gray failures where a whole bunch of people are getting a really bad experience, but it's still up and pinging in the load balancer or whatnot? just sitting down and thinking about these things and making sure that they're things that will be easy to do and are not going to drag you down later on so that you spend a little bit of time up front, get everybody on the same page, figure out what you're going to do, and then you can just go really fast after that. Awesome. Well, well, Sean, this was awesome. Thanks for having this conversation.Original English
>> Yes, this this was wonderful.