Runway 创始人 Cristóbal Valenzuela：从视频生成到世界模型与 Continuum 操作系统

Hey guys.

Very very excited to be here. Thank you

for staying for the last talk.

Welcome to my talk about the Matrix and

how we're living through it.

So, you know what? I've been working on

Runway for almost a decade now. Started

here in New York, actually, maybe around

10 years ago.

And

I want to walk you through some of the

learnings we've seen through, you know,

building the company, building this

research lab here in New York. And where

do we see a lot of the work that we do

in video, in pixel generation, really

moving forward? And so, I don't kind of

exaggerate and I joke that this is might

be a simulation or not.

perspective, I'll walk you through a

piece of history. This is like watching

prehistoric ancient ruins.

Uh this is our first AI film that was

done in 2019.

It's called

the orange because as you can tell very

clearly, it's an orange that falls from

the tree and walks through different

parts of the world.

video model and image model that we had

back then that was creating pixels in

128 pixels by 128 pixels that we then

forcedly up sample to

to this kind of resolution.

I think of this as art now. This is this

is like looking at a film from the

1900s, you know? It's it's beautiful.

There 's a lot of history behind it.

But this is 2019. Uh we started the

company in 2018, 2017. And so, we were

trying to figure out if this had any

lags.

Uh we're trying to convince ourselves

that if you scale this, it will

naturally improve to the point where we

can get somewhere close to

this.

This is all

generated. None of these characters are

real. None of those locations are real.

I'm sure you

You've probably here you know this um,

by now. you've probably have seen this

online on Twitter.

Um, the models are just incredibly good.

And the most interesting thing is that

when I look at stuff like this,

um, I still believe this will look like

the orange in just a couple more years.

Um, and so I want to walk you through,

you know, how we're thinking not only

through what we're doing at Runway

today, but also in the next couple of

years and the research that's shaping

that.

intersection of both both art and

science. Uh, we have uh, deep roots in

artistic and creative communities and we

have some of the best researchers in the

world working on diffusion models and

large video models. Um, and I like that

combination because it allows us to, you

know, bring the best of both worlds.

Uh, we're we're a team of scientists and

artists thinking about the problem of

scaling video and world models.

Um, but in a way that the core idea of

the company was already discovered by

cinema and by moving pictures 120, 130

years ago,

which is, if you think about video, and

maybe you remember this from, uh,

from high school, video is an optical

illusion. Video is the illusion of

movement. That if you do

if you stitch frames fast enough, you

will create movement. You will create

the illusion of movement. You create

motion pictures. And so at Runway we're

doing kind of something very similar.

We're taking that idea really to the

next level. How do we do next frame

prediction fast enough, scale enough, at

in in in in such a way that we can

hopefully do more than just pretty

pretty and nice creative videos. And so

by, of course, scaling laws. I think we

believe in scaling laws

uh, not only apply for for language

models, but also for for video. And this

has been true. We're now really ramping

up the amount of compute that we put

behind the the models that you already

have seen. And and I think the the

interesting thing about, you know,

seeing the model scale over time is the

emerging properties that appear, the

intelligence that appears when you start

taking this very simple idea that cinema

found itself uh, figuring out 120 years

ago,

um, which is next frame prediction, and

taking next frame prediction really to

the next level, right? So, if I take an

image like this, which is not real,

those two people are not real, and I

just keep on predicting what the next

frame will look like based on the

previous frames, and if I do it fast

enough, and if I do it with a network

that's big enough, I'll get movement.

I'll get the illusion of movement. I'll

get something that looks so real to us

that we we humans can distinguish. I I

look at that video and it's I've been

doing this for 10 years. I can't tell

you if that's real or not.

measured tried to quantify

um if people can distinguish between

real and generated videos, and I think

that that Turing test we already passed.

Uh

people are not able to distinguish. But,

the more interesting thing is here is

that really what's happening as you

predict next frames is you're learning a

lot more than just pretty frames. You're

learning about physics, you're learning

about object permanence, you're learning

about gravity, you're learning about

occlusion, you're learning about

movements, you're learning about how the

objects work in the real world, and

you're doing that all through just

video, through images, through through

next frame prediction. So, if you look

at a complex thing like this,

um a good video model for it to

understand how to make this a consistent

video will need to naturally understand

uh what all these objects are, how each

of them relate to one another, how a

camel might probably

uh stand on a ball like that, what water

in a container looks like, and if I do

all of that together, and I do it fast

enough, I'll get, hopefully, a

consistent video that looks real to us.

uh benchmarks that we use to evaluate

our video models are not only creative,

it's creative basically means do you

like it or do you don't, are actually

based on uh

on physics, on geometries, on how

objects behave in the real world, and

how those objects should behave, uh and

also simulator or video worlds. So, you

take that idea of next frame prediction,

and you do it fast enough, and you get,

of course, now something that looks more

like what we should expect from a video

application or a game application.

Again, this is just just video, right?

And you take you take that idea to the

next level. What happens if we start

this idea of next frame prediction, but

do it in the context of through the

environment or something that looks

similar to a game, you can start

actually getting closer to a physics

engine, a simulation system. But the

simulation system has

no physics laws that we didn't describe

what the behavior of the object should

be, the model just knows how to do that

by pure force of of of scale. And this

kind of gets gets kind of interesting

because you can only think about playing

worlds. This is not GTA, this is

someone playing someone else

in a environment and a software that

doesn't exist. It's all simulation of a

simulation, which which gets really

meta, right?

So, I'm not kidding when I said welcome

to my matrix simulation.

But the most interesting thing for me is

that everything you saw before was not

real-time. Meaning that it took a couple

minutes or seconds for those videos to

be made.

Um a lot of what we're seeing right now,

when we think about, you know, the next

three, four years of video models and

world models, is to deeply think about

what happens when you start not only

scaling the models and making them much

much much

bigger with much more emerging

properties, but also what happens when

you start making them interactive,

right? So, take the same idea of next

frame prediction, you take a frame, you

predict the next frame, the next, the

next, the next, but then I can act at

any point in time in that frame

prediction, which means I can simulate

any possible divergence or action in

that world. So, in this world, I could

play the video, but I can also come and

touch every part any aspect of that

image and make it interactive to affect

the consequences of of that world.

And it works.

for a variety of different use cases, as

you can imagine. We're now seeing an

entire robotics lab in Runway. So, if

you come to our office in New York, half

of it is what looks like a Hollywood

studio. Half of it is a bunch of robots

walking through and moving through the

office.

And so we're able to make these models

or start making these models the brains

of how autonomous and embodiment systems

walk and reason about the world.

Um but there's a couple of more

interesting use cases that I think go

beyond the two obvious one, which is

video and robotics, which is what

happens when you think about anything

anything any pixel on a screen

as a video model.

Um here's a good example. This is a

video model.

What do you see right now? Well, this is

an image first of all. That image could

be real or not, but I could run one of

our video models in real time and make

this an interactive application. So,

let's say I want to, you know,

turn the lights of the car. Maybe I want

to like turn the car around. Um I can

just click through it. I can turn it

around again. Maybe open the hood and

see what's inside the car. Now,

everything you just saw here has no

physical 3D representation. There's no

Unity. There's no Unreal. There's no

Blender.

It's just a video model. A video model

has been trained to understand actions

and clicks in an interface. We call this

a world action model.

Um sorry, um a world interface model.

And the world interface model, well, I I

feel like you guys understand now. It's

It's It's an interface and it's a world

model. World interface model.

Um allows you to very cool stuff like

this. And so here's another application

another um example of of how you could

think about using a model like this in

real time.

Um I can take an image a representation,

move objects around, and expect them to

move as I was if you if as I was

actually in that room. Again,

remember there's no 3D representation of

this. There This environment might not

actually be have having modeled before.

It's just an image and the model that

understands how objects look moving in a

space because of this very simple idea

that next frame prediction scale well

enough can become a world model or a

simulation engine.

so

with the 8 minutes I have left, I want

to show you this idea taken to another

extreme.

We're we're thinking not only about what

happens if you can control the school

applications and the school interfaces,

but what happens if you keep

extrapolating and I think by the way

extrapolating may be one of the most

important skills these days in in AI in

in company building. What happens over

the next 6 months, 12 months, 18 months

may be one of the most important things

to to try to predict and understand. And

so we ask ourselves that question a lot

at Runway. What happens? What will

happen if we keep scaling the models? If

we keep adding more data, if we keep

making them better simulation systems.

If we make them universal simulation

systems that can predict anything on a

screen. Literally anything on a screen.

So we've been working on this idea that

we call continuum.

And continuum is

the the idea of real-time video models

or world models taken really to the next

the next degree. This is an operating

system. The idea here is that you might

open your computer at some point in the

near future and what you see streaming

to your screen may be coming from a

video model.

I'll show you next specific how this

idea could could shape. This is still

very early research, but I want to show

you this idea of combining video models

as part of your operating system, as

part of your daily work. So here's an

example.

I was researching, you know, San

Francisco and the fog in San Francisco.

I wanted to learn more about when and

how it happens. This is structural data.

This is actually a website. This is

something

you can take from from the internet and

now continuum has this idea that I can

select any part of content in that

website and make it a portal. We call

this a portal. And the portal is a

real-time video. It 's fully interactive

and fully based on the content you're

experiencing reading through. I probably

won't read through all of that text.

It's too much text for me.

So I just want to get a video and I can

make a video that's represented by by

the text you see there.

These portals can be 3D illustrations.

It can be any content that you're

watching or interacting with. On the

right you can have all the

visualizations and portals you've you've

experienced before, anything you've been

trying to learn.

Basically it's a it's a memory. So, it's

the disk of the imaginations or the

visual sensations you've seen before.

So, here I was researching about

batteries, and so I can go back to

the portal I had around batteries

before. And since they're real-time

interactive applications, I can click them anywhere. I can move them around. I

can start asking questions and the

videos and and the content itself will

change based on based on my interactions

and the way I'm I'm managing through it.

when I the first time I saw it when the

team showed it to me, it was like blew

my mind.

Since it's a real-time video, you can

make it fully interactive with anything.

Not only with clicks, but like resizing

the applications would also change the

behavior. Right? So, I can resize. I'm

I'm learning about photosynthesis here.

If I resize the application, I get a

different experience. And since it's

again fully interactive

applications, I can click through it,

navigate through parts of the cell or

how photosynthesis work to get a better

understanding of what I'm trying to

learn and do.

Again,

the one thing that still feels to me

kind of uncertain about this is

I don't know how to call this. I don't

know if this is doesn't feel like a

visualization. It doesn't feel like a

video. It doesn't feel like a game. It's

something in between, and I think we

need to do the collective work of

figuring out how we call these kind of

interactions in the first place. And I'm

sure [snorts] you hear kind of the other

thing is that you can change the

structure of the prompt or what's

actually creating that interaction in

the first place.

I'm going to type something.

And this is one of the ones that I feel

has been the most underexplored yet.

This is we call this visual thinking.

So, we've connected Continuum to a

coding agent. And what the coding agent

can do is it can use Continuum to

visualize through a problem.

So, imagine you ask your coding agent to

create a 3D model for your bike. And

instead of the model trying to just

think through that via code, the the can

spin a version of Continuum to think

through the problem visually.

So, if you ever think about someone or

I'm sure you have friends or maybe some

a lot of people here maybe visual

thinkers. What a visual thinker is is it

thinks through problems and through

ideas visually first. And that helps you

ground and project some of the best

things you want to do. And this is

somehow very similar.

Agents have the ability to do visual

thinking. Agents can spin up

a sort of imagination that you can tap

and and interact with. So, not only the

agent can interact with it, I can also

see it myself. And I can guide the agent

in much better ways that I might have

able to

guide it just via text alone. Which is

for us a very powerful idea. It's a

completely new paradigm. And we call

this this this idea

uh visual visual thinking.

The last example I have

is

how you think about taking, you know,

Continuum to

become much more than just uh using or

navigating interactive structural

content in in the way I've showed you

before, but actually making them fully

live experiences that are reactive. So,

in this case I was searching for Mars.

Um and as as Continuum understands the

intention of what I'm trying to search

from, it also suggests interfaces that

are specifically designed in real time

just for that. So, here it created for

me this Mars outpost that's actually

should be uh accurate with the data

about the actual like robots that are in

Mars right now

uh or the ones that are projected maybe

in 2030. Um and since this is fully

interactive, I can interact with it. I

can go to places. I can move around in

the same way that I might move around,

you know, in a 3D system or in a 3D

Continuum. Um

And it works.

Just fascinating.

So, when I say

everything will be simulated, um

I'm really trying to understand or help

not really our team, but the world

get to the point where a lot of what

we'll see today will sound obvious,

hopefully in a few more years or months.

We'll look back at Continuum and it will

might feel very close to the video of

the orange that we just showed you.

Continuum and a lot of the things we're

doing in in video and in Runway when it

comes to simulation are still in very

early days, but we're getting to a point

where it actually it actually works and

it really scales. So, what we're really

excited about is that for the next 8 to

12 months we're releasing most of what I

showed here for everyone to use. Thank

you, guys.

[applause]