测试、技术债与“以慢为快”的软件工程实践

Sean, great to have you.

Music was definitely more my thing, but uh I graduated from college and I realized I had no idea what I was going to do and I just started going through uh my address book and the first person I found who said I could come stay with them uh lived in New York City. So I went and a good friend of mine was also living there and uh he eventually he was like I couldn't I couldn't find a job. I got close to getting one which was as a speech writer for a US senator. Uh but I did not get that. And a friend got me a job fixing computers at Scholastic. And then another friend told me, "Hey, if you learn Pearl, you can move to Boston and you can, you know, there was a bunch of people I'd gone to college with who started startup. So I learned Pearl and right about the time that I finished being good enough learning Pearl over the course of three to four months to, you know, do that. uh that startup had collapsed and then it started just be like

No, no, that was that was even before the.com bust. That was a combustible set of human beings uh working together. But yeah, at that point it started being like I I like this. I enjoy it. I'd done it a bit like earlier on uh when I was like a teenager cuz I there was a video game LoadRunner and I I loved LoadRunner and I really wanted to to create things that were like LoadRunner. So I asked my grandmother to get me a book about computer programming, but I didn't know anything about computer programming. my grandmother didn't know anything about computer programming. So, she went to a computer store in uh Fort Worth, Texas, and the guy gave her a book which was uh machine language for the Commodore 64. So, I was like, "Wow, programming is really hard, but yet that's I kind kind of got started that way." And it was the first time I was like, "Oh, there's a thing that people will pay me lots of money for." And I was like, "I have no other thing that people would pay me lots of money for. I should get really good at this.

Well, the the short version would be is that Will Wilson, the CEO, sent me a message while I was looking for a new job in December, which was just a link to a thing that they posted on the who's hiring for uh on Hackers Nukes and said, "Would you be interested in this?" It was basically like, "Do you want to do Jeepson style things?" And my response was, "That sounds like it could be fun."

And speaking of dream job, one thing that you've been really going deep into is just different languages which I would call a little bit of exotic languages. Not I'm not talking about the mainstream TypeScript or maybe roster go etc but a few other languages. Can we talk about Okamel specifically? What is Okamel and why are all the hedge funds and finance companies in love with it?

So, Oak camel is a language that came from an academic background primarily uh in France originally. Um and it was it came out at the time when people were trying to combine functional programming with objectoriented stuff. This was a really big deal that people like tried to do particularly in the '9s. At that point, it was bolting objects onto everything. Objects were very much seen as the future etc. And it's interesting because the version of O camel that's actually caught on at places like Jane Street and and other places where it get used a lot is one where none of the O features in O camel really get used.

the

object-oriented

features.

But

it's

it's

a

language

um

which

is

descended

from

ML

which

is

a

entirely

different

languages

of

families

from

like

algo

languages

where

where

like

Python

and

Pearl

and

for

the

most

part

Ruby,

C,

C++

these

are

these

are

all

ALGO

languages

and

then

there's

a

an

the

ML

family

of

which

uh

Okamel

is

is

one

there

there

are

a

number

of

others

in

there.

uh

depending

on

who

you

ask

some

people

will

say

hasll

is

kind

of

in

there

kind

of

not

and

um

all

the

ML

languages

generally

provide

a

a

different

set

of

uh

functionality

and

capabilities

to

people

the

most

famous

I

think

at

this

point

ML

uh

influenced

language

would

be

Rust

there's

an

awful

lot

of

ML

type

stuff

in

Rust

the

the

sum

types

the

pattern

matching

etc

all

of

this

is

stuff

that

came

from

this

sounds

like

a

bit

like

more

of

a

declar

erative

language

or

what

is

the

ML

how

would

you

describe

it

ML

languages

are

generally

uh

they're

built

around

strong

type

systems

right

in

the

same

way

that

that

Rust

is

built

around

a

strong

type

system

where

you

take

some

types

which

is

basically

saying

it

could

be

one

of

these

three

things

they're

kind

of

like

enumerations

uh

from

something

like

Java

that

that

people

might

be

familiar

with

or

hey

Rust

is

a

lot

more

popular

now

so

some

people

might

be

familiar

with

it

from

there

and

you

can

combine

that

together

with

uh

pattern

matching

and

and

if

you

can

then

also

combine

that

together

with

exhaustive

matching

um

where

what

it

means

is

that

I

have

this

sum

type

which

could

be

say

five

different

things

right

and

the

way

that

the

way

that

I

will

work

on

it

is

instead

of

an

if

statement

I

do

pattern

matching

but

I'm

matching

on

the

type

that

the

thing

is

[clears throat]

and

then

you

can

combine

that

with

exhaustive

matching

what

you

can

end

up

with

is

a

code

base

where

you

can

make

one

change

and

say

actually

now

I

have

six

different

things

of

this

instead

of

five

and

the

compiler

will

tell

you

every

single

place

in

your

codebase

where

you're

not

handling

six

type

and

it

it

is

a

gloriously

wonderful

uh

experience.

It

it

it

sounds

like

the

compiler

gives

you

additional

testing

that

you

might

not

get

from

a

compiler

in

let's

say

a

language

like

C

or

or

TypeScript

whatnot.

The

compiler

there

will

tell

you

the

types

but

if

I

understand

correctly

the

exhaustive

matching

is

the

key

here.

And

the

fact

that

there's

matching

it

goes

and

checks

kind

of

maybe

conditions

or

or

assumptions

that

are

in

these

languages

like

TypeScript.

I'll

just

use

it

as

an

example.

It's

kind

of

you

know

you'll

figure

it

out

runtime

if

something

breaks

and

here

in

compile

time

there's

a

lot

of

structure

rules

and

there's

of

course

like

logic

it

goes

through

there.

So

you

kind

of

frontload

a

lot

of

that

validation

if

you

will.

All

all

of

the

languages

which

are

MLike

in

some

type

of

way

or

are

in

that

general

family

are

very

much

languages

where

it's

like

I

want

to

put

as

much

as

possible

into

static

checking

in

the

compiler

and

have

the

compiler

do

work

for

me.

Okay,

so

now

I

understand

why

some

people

say

who

are

into

Rust

that

Rust

helps

you

ship

fewer

bugs

because

it's

an

ML

it's

type

language

which

the

compiler

will

do

a

lot

more

work

and

it

will

have

a

lot

you

stricter

rules

that

you

need

to

become

familiar

with

but

then

in

return

the

compiler

just

like

it

a

lot

of

things

and

it'll

give

you

a

compile

error.

Early

on

when

I

was

learning

to

program,

um

I

was

really

into

like

different

programming

languages

and

just

learning

like

different

things

that

were

possible.

And

uh

fairly

early

on

I

decided

I

was

going

to

pick

one

thing

and

I

was

going

to

dedicate

myself

to

that

for

a

while.

And

uh

the

two

languages

I

ended

up

settling

on

were

uh

C

and

ADA

And

it

was

mostly

it

was

C.

the

the

choice

of

C

came

because

I

was

doing

a

lot

of

programming

for

I

think

it

was

Mac

OS

it

was

either

six

or

seven

I

think

it

was

six

at

that

point

and

Apple

was

converting

an

awful

lot

of

their

stuff

from

Pascal

to

C

at

that

time

so

I

was

like

oh

okay

well

C

could

be

one

that

could

be

interesting

and

then

ADA

was

just

entirely

different

and

what

is

ADA

like

for

those

of

us

like

myself

who

don't

know

is

much

more

in

that

very

very

strict

static

checking

thing

[clears throat]

um

ADA

is

a

language

that

in

a

lot

of

ways

was

driven

by

the

needs

and

desires

of

like

what

the

defense

department

wanted

to

have

for

um

for

knowing

that

there

were

not

going

to

be

bugs

of

of

a

language

which

is

very

deeply

specified

and

will

not

let

you

write

incorrect

programs.

We're

talking

like

literal

mission

critical

stuff.

Mission

critical

stuff.

NASA,

the

defense

department,

etc.

These

these

were

types

of

people

who

ADA

was

was

aimed

at.

ADA

still

exists.

It

has

a

small

but

robust

market

for

people.

And

the

number

one

thing

that

the

ADA

companies

really

go

after

is,

"Oh,

so

you

like

Rust.

How

would

you

like

to

have

some

more

guarantees?"

Like

[laughter]

that's

that's

that's

where

that's

where

the

ADA

stuff

is.

And

this

is

this

is

one

of

my

great

regrets

was

that

um

I

didn't

really

know

much

about

programming

and

I

didn't

really

know

much

about

programming

languages

and

I

could

not

get

programs

to

compile

in

ADA

ADA

because

it's

so

damn

strict,

right?

it

was

so

frustrating

and

and

I

could

get

C

programs

to

compile.

It

didn't

really

dawn

on

me

the

fact

that

my

C

programs

were

seging

all

the

time

and

didn't

really

run

for

very

long,

but

it

was

like

thank

you

for

the

cookie

that

it

compiled.

you

know,

now

I

can

move

on

to

the

other

thing.

But

I

spent

years

developing

an

intuition

of

how

I

could

do

stuff

in

C

so

that

my

programs

didn't

die

and

collapse.

And

and

now

all

I

want

is

I

like

I

recognize

like

I

had

all

these

rules

for

myself

of

writing

concurrent

C

and

C++

and

to

keep

myself

from

uh

seging

programs.

And

even

even

though

I

became

really

rigorous

with

that,

I'm

a

human

and

I

screw

stuff

up

all

the

time.

And

I

would

much

rather

have

a

compiler

help

me

with

that.

And

so

there's

a

lot

of

stuff

in

like

O'

camel

and

other

languages

like

it.

Uh

F.

can

we

talk

can

we

talk

a

little

bit

about

F?

folks

who

have

not

used

F

uh

but

they

might

have

used

C#

or

TypeScript.

How

would

you

describe

F

on

why

is

different?

And

right

it's

it's

a

language

that

Microsoft

created

or

or

they

championed.

And

I'm

not

sure

if

they

created

it.

so

it

came

out

of

Microsoft

research.

Um

there's

a

fellow

Don

Sim

who

is

really

responsible

for

everything

F

and

it

basically

started

as

uh

O

camel

fornet

wheren

net

at

the

time

was

just

car

and

then the

net

runtime

and

net

is

Microsoft's

framework

where

it

was

really

easy

to

access.

I

remember

Windows

APIs.

It

had

a

lot

of

utility

functions.

It

was

their

everything

framework.

it

basically

their

version

of

the

JVM.

The

Java

virtual

machine.

I

have

no

special

insight

into

it,

but

from

somebody

who

was

around

when

all

those

things

happened,

there

was

a

whole

bunch

of

legal

kurfuffle

with

Microsoft

adding

Windows

specific

APIs

to

their

version

of

Java

and

Sun

threatening

to

sue

them.

was

a

lawsuit

and

so

I

think

Anders

Hayesburg

on

this

podcast

shared

the

detail

that

they

needed

to

develop

C

and

I

guess.NET

net

as

well

because

of

a

pending

or

or

active

sun

lawsuit

and

so

they

could

no

longer

use

Java

even

though

they

wanted

to

use

it

and

that

kind

of

settled

it.

So

they

built

.NET.

did

and

andnet

was

was

great

because

they

took

like

pretty

much

everything

they

learned

from

the

JVM

and

things

where

ideas

have

been

baked

into

the

JVM

that

would

be

kind

of

hard

to

change

at

that

point.

And

they

did

different

things

which

at

the

time

seemed

like

much

better

ideas.

People

argue

about

whether

they

are

or

not

at

this

point

but

doesn't

really

matter.

But

you

know

net

was

a

very

nice

platform

um

etc

for

everything

and

they

had

one

language

basically

that

was

on

it

which

was

C

which

is

an

awful

lot

like

Java.

It

has

differences

but

fundamentally

the

basic

semantics

of

the

language

are

the

same.

Yeah.

and

so

F

was

a

hey

let's

bring

some

different

semantics

let's

bring

some

different

stuff

to

this

which

was

originally

here's

what

happened

in

this

incident

here's

what

happened

in

this

incident

let

me

let

me

as

somebody

who's

like

been

versed

in

this

stuff

for

a

long

time

try

to

break

it

down

for

you

and

One

interesting

thing

you

said

about

quality

and

I'll

quote

you

again

is

is

uh

on

how

quality

is

not

a

gate.

And

so

you

wrote

quality

is

not

a

gate.

The

whole

gate

idea

rests

on

a

belief

we're

dragging

into

the

light.

The

belief

is

that

quality

is

something

you

hold

by

keeping

the

bad

out.

Catch

the

bug

right

at

the

door.

It's

a

comforting

picture,

but

it

isn't

how

good

good

teams

stay

good.

So

this

is

this

is

this

is

really

in

a

lot

of

ways

why

I

don't

like

talking

about

testing

as

a

thing

and

I

like

talking

about

correctness

because

like

as

much

as

possible

your

system

for

whatever

it

is

should

not

allow

bad

things

to

happen

period

from

just

how

you

can

write

the

software

and

static

checks

that

you

should

be

able

to

get

like

in

an

ideal

world

you

should

be

able

to

write

a

specification

in

like

a

language

like

TA+

plus

or

or

alloy

or

something

like

that

and

have

a

perfect

representation

of

that

as

an

implementation

or

ideally

just

take

that

and

and

put

it

like

be

able

to

magically

go

from

that

into

an

implementation

which

somehow

also

has

all

of

your

performance

stuff

etc.

We're

nowhere

near

close

to

that

world,

but

that

would

be

an

ideal

world.

And

you

just

simply

know

that

no

bad

thing

can

happen

because

you've

you've

used

a

model

checker

to

prove

this.

And

they

can

do

incredible

things

for

you.

And

we

don't

really

have

that.

and

so

another

level

that

we

can

go

to

is

just

static

checking

inside

of

type

systems

for

things

for

for

the

stuff

that

like

Pony

does,

for

the

stuff

that

like

Rust

does,

for

the

stuff

that

Okamel

does.

Um,

in

an

interesting

way,

Erlang

does

a

lot

of

this

type

of

stuff

as

well,

but

it

does

it

without

a

type

system,

but

it

does

it

with

the

actor

model

for

we're

going

to

limit

what

you

can

actually

do

in

terms

of

stuff

where

you

have

to

send

everything

by

messages

and

every

single

message

the

data

that

one

actor

has

is

copied

uh

and

sent

to

another

actor

such

that

there

can't

be

any

data

races

with

the

sharing

of

this

memory

because

it's

never

actually

shared.

It's

just

copied.

And

in

this

way,

we've

also

done

a

correct

by

construction

sort

of

thing

that

you

could

also

get

from

a

type

system

or

whatnot.

we

want

to

have

this

foundation

to

build

upon.

And

the

the

testing

aspect

should

be

for

the

things

that

whatever

our

static

means

that

we

have

available

to

us

just

can't

enforce

because

in

the

end

we

only

have

we

have

programming

languages

and

the

programming

languages

all

come

with

tradeoffs.

and

like

um

the

programming

language

that

maybe

has

the

perfect

type

system

to

represent

everything

that

you

need

to

do.

Well,

one,

it

probably

doesn't

exist.

And

then

two,

even

if

it

did

exist,

it

probably

would

not

have

some

characteristic

that

you

also

needed

as

well

in

terms

of

performance

or

something

else.

So,

you

need

to

settle

some

and

pick

a

language

that

has

better

performance

guarantees

or

or

what

whatever

it

is.

And

then

at

that

point

what

you're

left

with

is

testing

and

and

verification.

But

I

kind

of

want

to

push

back

on

this

like

the

quality

gates

because

the

idea

of

quality

gates

is

like

okay

you

build

a

code

you

have

a

compiler

it

does

some

stuff

the

languages

we

use

for

practical

reasons

will

there

be

a

bunch

bunch

of

bugs

for

application

level

bugs

even

if

the

compiler

was

perfect.

It

couldn't

test

for

the

the

business

logic

being

correct.

And

then

the

the

way

we

used

to

think

about

it

or

or

I

think

a

lot

of

people

still

think

about

it

is

okay

you

write

this

thing

and

you

have

quality

gates

you

have

like

code

review

CI/CD

static

analysis

I

guess

in

production

you

have

observability

and

there's

these

gates

where

like

it's

a

little

bit

like

sometimes

you'll

see

the

Swiss

cheese

model

like

each

layer

will

catch

some

stuff

like

code

review

sometimes

it

might

catch

bugs

although

we

know

it's

not

a

great

place

to

do

it

but

especially

now

we're

not

sure

what

happening

with

code

review

but

it

could

catch

bugs

or

or

readability

inside

of

Google

they

have

readability

review

like

it

will

catch

reasonability

issues

in

in

their

case

and

then

CI

CD

will

catch

some

some

bugs

with

the

test

or

some

some

regressions

that

we've

seen

and

we've

coded

things

for

them

and

so

on.

You're

saying

that

quality

gates

is

is

that

quality

is

not

a

gate.

So

does

that

mean

that

you

don't

believe

in

having

these

I

guess

let's

just

call

them

gates

in

your

release

process

or

you're

just

saying

just

think

about

it

differently

like

think

about

it

from

the

beginning

you're

it

feels

like

you're

taking

back

the

idea

of

quality

to

the

to

the

very

beginning

right

very

much

there

and

also

I

would

say

I

I

disagree

with

your

statement

that

uh

business

logic

can't

be

statically

uh

enforced

and

whatnot

um

um

but

let's

let's

leave

that

let's

leave

that

rabbit

hole

aside

for

now

but

uh

additionally

like

particularly

in

the

way

that

it's

it's

normally

done

outside

of

the

Jeepson

and

antithesis

type

style.

Um

unit

testing

and

testing

when

they're

run

CIC

is

the

um

honeycomb

has

awesome

shirts,

right?

I

test

in

prod

which

is

platform.

Yeah.

And

and

and

it

it's

true

that

like

everybody

in

the

end

in

some

way,

whether

they

admit

it

or

not,

is

testing

and

prod.

None

of

the

tools

that

we

have

are

good

enough

to

guarantee

whatever

it

is.

Like

even

if

you're

doing

an

incredible

job

with

like

an

antithesis

Jeepson

type

tool

and

you're

thinking

up

all

these

great

and

interesting

inputs,

there

going

to

be

some

data

which

got

corrupted

at

some

point

in

time

which

exists

in

your

production

system

which

you

haven't

randomly

hit.

your

generator

simply

can't

generate

whatever

it

might

be

that

is

going

to

happen

in

production

and

we

do

not

know

it's

out

there

and

we

are

we

are

testing

it

and

the

idea

that

somehow

we

have

this

this

gate

where

it's

like

oh

if

it's

gone

through

this

it's

quality

it

can

go

out

is

just

it's

it's

kind

of

a

lie

right

like

we

have

hopefully

increasing

levels

of

assurance

as

we

go

along

but

we

don't

actually

have

a

gate

where

we

know

it's

quality

but

you're

you're

saying

like

these

things

can

help.

It's

just

like

know

that

there

will

be

holes.

Like

there

going

to

be

holes.

We

have

We

have

no

will

sink

through

and

adding

another

gate.

You're

saying

there's

no

one

gate

that

you

could

add

that

will

be

like,

"All

right,

this

now

seals

off

all

the

remaining

holes."

As

long

as as

long

as

we're

looking

at

these

kind

of

gates

of

like

one

more,

you

know,

like

the

same

the

same

as

if

if

you

add

one

more

test

or

two

more

tests

or

three

more

tests,

when

you

like

hardcode

the

values,

those

will

they'll

make

it

better,

sure,

but

they

will

they

will

not

guarantee

quality.

So

most

of

the

places

I've

been

where

there

are

teams

which

I

think

are

what

I

would

call

low

functioning

which

are

ones

where

they

tend

to

move

slowly,

they

don't

deliver

stuff

a

lot

or

they

deliver

stuff

quickly

but

like

they

regularly

have

to

do

emergency

releases

to

fix

stuff

etc.

are

people

who

think

about

like

some

sort

of

thing

around

traditional

CI/CD

stuff

now

at

least

traditional

now

that's

built

around

like

pull

requests

or

or

or

a

staging

environment

where

they

run

some

tests

before

it

goes

out

to

prod

that's

generally

how

they

think

about

it

is

like

oh

this

is

this

is

where

I

have

these

quality

gates

and

and

the

people

who

move

quickly

and

with

confidence

and

then

it

actually

holds

up

and

they

you

know

they're

not

getting

alerted

all

the

time

are

people

who

think

about

quality

from

the

very

beginning

of

sitting

down

and

being

like

oh

hey

I

should

validate

for

example

uh

if

I'm

doing

a

thing

which

is

a

distributed

database

which

is

particularly

one

I

should

be

using

something

like

a

TA

plus

to

make

sure

that

my

consensus

algorithm

actually

works

etc

and

everything

and

to

make

sure

that

I

don't

have

bugs

in

that

or

to

take

a

Jeepson

antithesis

type

tool

and

put

the

very

first

version

of

your

stuff

before

you

ever

ship

it

to

production

in

there

and

So,

does

it

behave

like

I

expect

it

to?

Because

it's

really

easy

to

fix

stuff

early

on

when

you

create

the

thing.

And

it's

really

hard

if

you

go

like

which

I've

seen

many

places

do

where

it's

like,

"Oh,

here's

our

new

system."

And

they

don't

really

do

any

testing

on

it

until

like

somewhere

way

down

the

line

and

nobody

has

any

idea

whether

it's

like

actually

working

or

not

or

even

necessarily

any

idea

to

know

what

does

working

really

mean.

like

they

haven't

asked

themselves

from

the

beginning

how

will

I

know

this

thing

is

working

uh

I

want

to

push

back

on

like

you

know

that

you

can

just

write

a

test

if

it's

working

I'll

give

you

an

example

where

it

might

be

really

hard

to

write

test

that

it's

working

a

Google

search

I

type

in

the

best

testing

tool

in

the

world

right

if

I

ask

10

people

and

five

of

those

work

at

vendors

they

all

say

something

different

should

be

the

number

one

right

the

people

who

work

at

vendors

they

will

say

my

vendor

should

be

Number

one,

obviously

I'm

working

here.

We're

doing

amazing

stuff

in

in

testing.

Some

people

will

think,

oh,

like

they'd

like

the

Wikipedia

article.

Some

people

will

say

like

I

think

this

this

should

happen

and

of

course

when

I

run

the

same

Google

search

day

after

day,

those

things

will

change

a

little

bit

like

the

the

results

will

change

and

we

can

assume

that

Google

is

is

knows

what

they're

doing

and

and

it's

correct.

But

now

if

I

wanted

to

just

even

write

a

test

for

for

a

complex

system

and

of

course

we

know

the

reason

it

changes

because

Google

calculates

the

page

rank

and

they

take

a

bunch

of

signals

and

they

also

take

things

like

real

time

so

like

news

will

be

fresher

but

now

we

have

this

really

complex

system

and

if

I'm

I

would

be

given

the

simple

task

of

like

well

write

some

tests

so

that

we

can

tell

if

this

small

segment

of

Google

search

works

fine

let's

say

inside

of

Google

I

would

now

struggle

because

I'm

like

well

you

know

if

we

went

like

all

right

you

can

use

any

any

formal

language

or

or

TLA

plus

would

struggle

to

code

like

what

am

I

expecting

even

though

when

you

look

at

it

you

can

kind

of

squint

and

say

like

this

looks

good

enough

or

if

you

have

enough

data

but

there

are

some

things

that

we

that

we

we

we

can

test

for

example

there's

some

invariance

we

can

do

which

like

for

example

I

put

a

thing

in

the

box

that

I

submitted

I

expect

to

get

results

back

one

so

that's

one

one

invariant

for

example

is

there

is

an

invariant

there

and

then

we

start

getting

into

things

and

you

know

I

did

the

machine

learning

stuff

for

a

while

which

was

actually

in

recommendation

systems

and

that

gets

that

gets

into

the

interesting

thing

of

you

like

oh

well

what

should

the

number

one

ranking

be

right

and

that

is

that

is

a

that

is

a

very

different

thing

than

is

it

working

right

like

that's

very

different

is

it

working

is

that

I

put

a

thing

into

the

box

and

I

got

something

back

the

the

is

it

relevant

is

an

entirely

different

thing

like

and

then

with

the

is

it

working

we

can

now

ask

questions

like

if

if

if

I

submit

like

a

thousand

requests

in

a

second

like

is

it

still

working

if

if

if

we

have

concurrent

requests

coming

is

is

it

working?

If

in

the

middle

of

submitting

a

request,

I

submit

a

second

or

navigate

away

and

so

on.

So

like

so

now

you're

saying

we

can

decompose

this

into

different

parts

of

the

problem.

are

the

results

that

I'm

getting

results

that

are

for

my

actual

query

where

what's

in

it

might

not

matter,

but

we

we

can

we

can

set

up

tests

so

that

each

query

gets

some

type

of

marker

that

goes

along

with

it.

And

we

make

sure

that

those

actually

came

back

and

that

the

server

didn't

get

confused

and

give

me

your

results,

which

okay,

maybe

for

search

I

don't

know.

I

don't

know

how

any

would

ever

know

what

happened,

but

I

think

everybody's

heard

about

the

incidents

where

uh

like,

you

know,

the

ex

service

started

showing,

you

know,

somebody

else's

account

to

people

and

whatnot.

That's

a

really

bad

one

there.

You

know,

as

somebody

who's

had

their

identity

stolen

three

times,

I

really

hope

that

that,

you

know,

nobody's

showing

my

PII

to

uh

somebody

else

accidentally.

I

I

want

to

talk

about

some

stuff

that

are

interesting

learnings

in

the

industry,

and

I

know

you're

passionate

about

them.

One

is

supply

chain

security.

what

is

supply

chain

security?

I

mean

I

I

think

we're

hear

hearing

it

a

lot

more

unfortunately

especially

in

open

source,

right?

So

basically

supply

chain

security

is

talking

about

if

if

you

have

something

which

is

popular,

right?

Then

the

easiest

way

to

compromise

that

thing.

We're

talking

software

open

source

software

software

to

find

the

weakest

link

for

it

and

to

attack

that

right

which

and

this

this

applies

to

software

but

this

also

applies

to

hardware

for

example.

where

there

can

be

all

sorts

of

security

guarantees

that

like

um

AMD

makes

where

they

have

uh

trusted

uh

computing

modules

which

are

in

there

which

Intel

makes.

But

if

you

if

you

really

want

to

attack

those

systems,

what

you

want

to

have

is

somebody

who

when

they're

putting

this

together

can

put

a

malicious

firmware

onto

those

machines.

How

do

you

know

how

do

you

know

that

that

is

actually

like

the

firmware

that

is

supposed

to

be

and

you

know

I

I

have

I

have

a

security

background

as

well

and

this

this

gets

talked

about

as

like

the

trusting

computing

base.

But

people

usually

talk

about

this

um

when

they

talk

about

supply

chain

security

at

this

point

now

usually

about

software

and

things

like

wasn't

left

was

leftpad

a

supply

chain

security

one

years

ago

where

somebody

was

like

it

was

like

this

little

module

that

was

on

mpm

that

did

something

and

somebody

put

like

a

crypto

miner

or

something

in

it.

was

typically

like

I'm

not

sure

how

trivial

it

is

but

like

it's

replacing

a

package

that

is

widely

used.

There

was

some

very

simple

package

that

was

doing

some

trivial

stuff.

like

so

a

a

great

way

that

this

could

happen

for

example

if

you

can

go

after

something

which

people

use

and

they

don't

think

about

too

much

particularly

say

in

like

a

CI/CD

system

and

you

can

get

your

code

running

inside

of

the

CI/CD

system.

You

now

have

access

to

all

of

the

secrets

that

are

running

inside

that

CI/CD

system

which

means

that

you

can

start

publishing

software

which

might

have

malware

in

it.

who

knows

what

else

and

whatnot.

Great.

That's

that's

one.

But

even

more

so

like

almost

every

single

programming

language

that

exists

right

now.

Just

it's

basically

it's

basically

like

here

you

go.

There's

something

I

downloaded

it

off

the

internet.

Maybe

I

maybe

I

checked

its

hash

or

whatnot

and

it

can

do

literally

anything.

I

downloaded

a

thing

which

is

supposed

to

pad

out

some

text,

but

it's

actually

allowed

to

I

don't

know

run

a

crypto

miner,

make

make

network

connections,

etc.

and

everything

and

and

all

of

these

these

are

these

are

supply

chain

attacks

and

they

can

have

really

devastating

effects

they're

happening.

We

saw

it

in

mpm.

I

think

we

we're

I

think

we're

seeing

it

in

the

Python

community

as

well.

There's

a

quite

good

question

of

what

are

efficient

defenses

because

you

know

like

there's

a

couple

of

naive

defenses.

One

is

oh

let's

just

have

a

cool

down

period.

Let's

not

get

any

new

new

packages

for

at

least

a

week

and

we're

hoping

that

if

there's

a

supply

chain

attack

it

will

be

big

enough

that

it'll

be

published

and

it'll

be

resolved.

Of

course

the

downside

is

that

if

there's

a

hot

hot

fix,

there's

literally

a

security

vulnerability.

We're

now

leaving

ourselves

open

for

a

week.

But

that's

kind

of

one

naive.

The

other

very

naive

is

let's

get

rid

of

our

dependencies.

Let's

just

bring

it

in

a

house

or

let's

and

then

the

third

one

is

let's

fork

it.

Let's

have

this

this

fork

and

maybe

we

can

have

a

security

team

who

can

decide

to

like

update

or

merge

or

keep

it

in

house.

But

these

are

the

kind

of

like

very

all

of

them

are

very

naive

and

I'm

I'm

wondering

what

are

you

seeing

work

and

being

practical

cuz

you

it

seems

you're

trading

off

kind

of

latency

of

being

able

to

get

fresh

packages

which

often

have

either

critical

fixes

or

really

good

functionality

for

being

afraid

that

what

if

that

thing

is

poisoned

and

and

if

everybody

decides

to

do

the

cool

down

period

who's

going

to

notice?

We

haven't

even

talked

about

right?

Um,

so

I

think

part

of

this

is

is

that

there

is

no

good

answer

for

almost

every

single

programming

language

that

exists

and

any

of

the

ones

where

they

might

have

a

good

uh

answer

to

this

um

are

ones

that

nobody

uses

because

it

it

it

really

has

not

been

a

concern

for

a

long

time.

the

way

that

pretty

much

almost

all

package

management

and

almost

all

open

source

does

dates

back

to

like

the

way

that

Pearl

did

with

CPAN

and

which

was

a

which

is

an

incredibly

open

and

trusting

thing

in

the

same

way

that

almost

all

early

internet

stuff

was

where

it's

really

easy

to

spoof

somebody

sending

an

email

all

of

this.

It

was

it

was

a

network

that

just

assumed

everybody

was

a

good

actor.

And

I

guess

back

then

there

was

a

higher

bar

for

entry

as

well.

Like

if

you're

a

malicious

actor,

you

at

least

needed

to

learn

to

code

or

you

need

to

hire

someone,

which

again

is

now

a

little

bit

trickier

because

once

they

know

what

we're

doing,

it's

a

higher

barrier

to

entry.

But

but

that's

changed,

right?

Even

the

barrier

to

entry

has

changed.

e

even

back

then

it

wasn't

true.

One

of

the

one

of

the

people

who

was

a

high

school

messup

who

I

went

to

college

with

was

somebody

who'

been

arrested

by

the

US

government

for

unleashing

when

he

was

a

teenager

a

worm

like

across

a

whole

bunch

of

government

computers

that

he'd

gotten

into.

and

and

and

that

actually

got

further

out

and

was

one

of

the

one

of

the

big

incidents

at

at

that

time.

So

like

I

mean

it

was

still

happening

then

it

was

just

it

was

at

a

smaller

scale

and

not

everybody

was

depending

on

these

things

at

the

this

particular

point

in

time

and

and

the

fundamental

thing

is

that

for

any

dependency

pretty

much

for

any

language

like

JavaScript

or

Ruby

or

or

or

Rust

or

anything

that

that

comes

in

for

the

most

part

basically

the

code

that

you

wrote

has

the

exact

same

capabilities

for

what

it

can

do

as

the

code

that

you

happen

to

bring

in

from

the

internet

and

your

code

can

do

absolutely

anything

probably

and

that

other

code

could

do

absolutely

anything

as

well.

And

so

how

are

you

possibly

going

to

vet

all

of

this

code

if

you're

going

to

have

all

of

these

dependencies

which

gets

back

to

the

to

that

problem

and

in

the

end

nobody

is

going

to

solve

this

using

the

existing

stuff

we

have.

people

might

try

to

do

sandboxing

or

whatnot,

but

that's

just

in

the

end

like

every

sandbox

can

be

escaped

from.

It's

usually

not

that

hard

and

whatnot.

And

there's

a

whole

area

of

uh

study

which

which

is

in

these

areas.

Uh

uh

there's

a

guy

named

Mark

Miller

who

developed

a

language

called

E

which

has

some

stuff

called

object

capabilities.

Um

where

there's

a

lot

of

interesting

research.

We're

we're

here

right

now

as

as

part

of

bug

bash

in

in

Copenhagen

and

the

first

talk

of

the

day

was

uh

from

Sophia.

She's

a

professor

at

uh

Imperial

which

was

actually

talking

about

like

some

object

capability

work

etc

they've

done

but

the

basic

idea

of

object

capability

is

that

in

order

to

take

any

action

within

the

system

you

have

to

be

given

the

capability

in

some

fashion

to

do

it.

So

that

the

system

would

not

allow

you

for

example

to

open

uh

a

network

connection

unless

you

have

the

capability

to

do

that

and

the

originating

program

your

code

that

is

written

uh

has

to

give

that

capability

to

other

stuff

and

there

there

you

know

there's

no

popular

languages

that

do

this

um

because

you

know

because

pony

is

written

by

like

Silvin

initially

and

myself

now

the

object

capabilities

are

built

in

but

this

is

this

is

a

this

is

one

means

to

like

to

be

able

to

have

it

be

like

I

can

trust

this

code

more.

But

but

you

you

need

to

go

you

need

to

go

further

than

that

because

if

your

if

your

programming

language

allows

for

some

type

of

foreign

function

interface

which

almost

all

of

them

do

so

that

you

can

utilize

like

CC

code

or

or

rust

code

or

anything

using

a

Cabi

um

if

you

don't

have

some

way

to

control

that

completely

then

it

doesn't

matter

if

you

have

object

capabilities

because

somebody

can

just

go

out

the

uh

the

FFI

door

and

all

of

these

are

things

which

need

to

be

addressed

and

there's

no

way

that

these

things

will

be

addressed

by

just

incrementally

doing

stuff

within

the

system

that

we

currently

have.

it

sounds

like

there's

a

reason

that

software

security

is

getting

more

important

and

it's

a

complex

topic.

I

we

did

a

podcast

episode

on

software

security,

but

it

felt

we

were

just

scratching

the

surface

cuz

as

with

anything,

right,

it's

it's

a

nuance.

It's

it's

a

detail

where

it

gets

complicated

and

and

I'm

sure

there's

so

many

strategies.

every

every

company

and

team

decides

on

what

strategies

to

do

from

the

very

simple

when

you're

a

small

startup

and

or

well

you're

just

yolo

at

first

and

then

and

then

you

start

to

do

again

a

simple

thing

is

just

a

cool

down

hoping

that

not

everyone

cools

down

and

it'll

be

okay

and

then

it

gets

a

lot

more

nuance

scanning

for

dependencies

evaluating

them

reducing

unnecessary

ones

scanning

dependency

chains

and

so

on

like

you

can

go

really

deep

you

can

go

to

go

all

to

point

to

have have

a

security

team

who

will

vet

them

or

try

to

verify

them

or

or

even

maybe

have

contact

with

the

know

who's

building

it

and

so

on

and

so

forth.

the

the

really

horrifying

part

of

all

of

this

is

we

could

do

all

of

these

all

of

these

things,

the

ones

I've

mentioned

and

even

more

and

none

of

it

in

the

end

would

be

a

really

good

defense

because

we

made

a

we

made

a

tradeoff

years

ago

in

order

to

go

faster

with

our

CPUs,

which

is

we

introduce

speculative

execution.

And

speculative

execution

works

because

it's

like,

hey,

I

have

extra

cycles

and

this

thing

could

go

down

this

branch

or

it

could

go

down

this

branch

and

I'm

going

to

execute

both

at

the

same

time,

which

has

given

us

the

vast

majority

of

the

speed

ups

that

we've

gotten

for

like

the

last

20

years

at

this

point.

and

and

people

have

probably

seen

stuff

for

like

side

channel

attacks

is

generally

what

they're

called.

And

there's

a

whole

variety

of

these

because

the

thing

is

is

that

those

speculative

branches

have

to

actually

exist

in

memory.

And

so

if

I

am

good

enough

as

an

attacker,

I

might

you

you

executed

a

branch

for

what

happens

if

I'm

authorized

and

what

happens

if

I'm

not

authorized.

And

it

turns

out

for

a

sufficiently

like

knowledgeable

person

which

doesn't

actually

take

that

much

knowledge

to

get

even

if

they

don't

actually

have

the

access

to

the

system

can

use

a

side

channel

attack

to

read

the

memory

that

was

created

for

this

is

what

the

access

actually

is

or

even

better

they

can

find

like

they

can

they

can

get

the

the

keys

out

of

out

of

memory

that

are

used

for

security

etc

and

whatnot.

So

we

could

do

everything

else,

right?

And

as

long

as

like

I

worked

at

I

worked

at

Microsoft

research

on

the

confidential

computing

group

and

we

built

uh

a

confidential

container

offering

which

is

which

is

still

there

at

Microsoft

and

the

thing

that

we

always

said

about

our

security

guarantees

which

were

great

was

we

tacked

on

at

the

tail

end

modulo

side

channel

attacks

which

is

we

can't

protect

you

against

side

channel

attacks.

Every

single

guarantee

that

we

have

is

violated

by

side

channel.

And

what

is

confidential

computing?

computing

is

computing

where

basically

the

end

goal

is

that

the

software

can

be

run

in

an

environment

where

the

operators

of

the

environment

are

not

trusted

in

in

you

play

something

on

the

cloud

and

you're

not

sure

who's

running

that

cloud.

years

ago,

there

was

a

thing

that

that

happened

where

it

turned

out

uh

some

Twitter

employees

were

actually

uh

working

as

agents

for

the

Saudi

Arabian

government

and

were

spying

on

the

activities

through

DMs,

etc.

and

stuff

that

uh

that

Saudi

dissident

were

doing.

it

it's

it

could

be

this

sort

of

thing

or

it

could

be

the

thing

where

you

know

oh

you

might

trust

Microsoft

or

Amazon

and

you

don't

think

they're

going

to

steal

your

data

but

actually

you're

trusting

everybody

who

has

their

access

which

means

that

you

're

trusting

every

employee

not

to

get

fished

or

you're

entrusting

every

employee

not

to

have

a

money

problem

where

they

could

be

bribed

etc

or

whatnot

and

so

confidential

computing

is

an

attempt

to

make

it

so

that

you

don't

have

to

trust

those

people

because

in

security

the

more

things

you

trust,

the

Here's the thing, though. We were making money even though these kind of horror stories were happening. We were growing and supporting ourselves. We were continually creating value for the company. Yeah, we had a lot of technical debt, but we kept finding inventing ways to sidestep having to pay it. The debt metaphor we've adopted is misleading. You don't always have to pay off the debt you created. You can work around it and avoid ever having to pay it off. Technical debt only hurts if you have to pay it back.