in in this incredibly scary late period

when AI has really automated research

humans do this uh this function of like

auditing uh making it more difficult for

the AIS to conspire together and root

the servers take over the process and

extract information from them within the

set of things that we can verify like

experiments where we can see oh yeah

this works at stopping an AI trained to

get a fast one past human Raiders

the reasons why I think we actually have

such a a relatively good chance of

handling that um are

twofold so one is has we approach that

kind of AI capability we're we're

approaching that from weaker systems if

the really bad uh sorts of uh

motivations develop relatively later in

the training process at least with all

our counter measures

then by that time we may have plenty uh

of ability to extract AI assistance on

further strengthening the quality of our

adversarial examples the strength of our

neural eye detectors the experiments

that we can use to reveal and elicit and

distinguish between different kinds of

reward hacking Tendencies and

motivations so yeah we may have system

that have just not developed bad

motivations in the first place and be

able to use them a

lot um in developing the incrementally

better systems in a safe way and we may

be able to even if some of the early

systems do develop these bad motivations

if we're able to detect that an

experiment and find a way uh to get away

from that then we can win even if these

sort of hostile motivations develop

early

when I combine the possibility

that we get relatively lucky on the

motivations of the earlier AI systems

systems strong enough that we can use

for some alignment research tasks and

then the possibility of getting that

later with AI assistance that we can't

trust fully where we have to have hard

power constraints and a number of things

to prevent them from doing this takeover

uh it still seems plausible we can get a

second saving throw where we're able to

extract work from these AIS on solving

the remaining problems of alignment of

things like neural ey detectors faster

than they can contribute in their spare

time

uh to the project of overthrowing

humanity hacking their servers and

removing the hard power and so if we

wind up in the situation where the AIS

are misaligned and then we need to

uncover those motivations change them

and align them then we get a very scary

situation for us um

because we might we need to do this

stuff very quickly we may fail but it's

a second chance where our work is just

evaluating outputs that the AI are

delivering having the hard power uh and

supervision to keep them from

successfully rooting the servers doing a

takeover during this process and have

them finish the alignment task that we

sadly failed uh to invest enough or

succeed in doing beforehand the

incredibly juicy ability that we have um

working with the AIS is

that we can have as an invaluable

outcome that we can see and

tell whether they got a fast one p past

us on an identifiable situation we can

have here's an air gap computer you get

control of the keyboard you can input

commands can you root the

environment uh and make a blue banana

appear on the screen even if if we train

the AI to do that and it succeeds we see

the blue banana we know we know it

worked even if we did not not understand

and would not have detected the

particular exploit that it used to do it

this can give us a rich empirical

feedback where we're able to identify

things that are even uh an AI using its

best efforts uh to get past our

interpretability Methods at what point

would it be the case that the AI is

contributing significantly in the sense

that it would almost be the equivalent

of having additional researchers to AI

progress and softare the thing to look

for uh is when is it the case that the

the contributions from AI are starting

to uh become as large or larger as the

contributions uh from humans so like uh

when this is boosting their effective

productivity by 50 or 100% And you like

if you then go from you eight month

doubling time say for Effective compute

from software Innovation things like

like inventing the Transformer or

discovering chinchilla scaling and do it

in your training runs more optimally or

creating flash attention it doesn't have

to have been able to automate everything

involved in the process of AI research

it can be it's automated a bunch of

things and then those are being done in

extreme profusion because any I think a

thing that a AI can do you have it done

much more often because it's so

cheap uh and so it's not a threshold of

this is human level AI it can do

everything a human can do with no

weaknesses in any area it's that even

with its

weaknesses it's able to bump up the

performance tens of millions of

gpus each is doing the work of maybe 40

maybe more uh of these kind of existing

workers this like going from a Workforce

of tens of thousands to hundreds of

Millions you immediately make all kinds

of discoveries then you immediately

develop all sorts of tremendous

Technologies so human level AI is deep

deep into an intelligence explosion the

intelligence explosion has to start with

something weaker than that yep yep yep

what is the point of which that feedback

loop starts where you can even you're

not just doing the 05% increase in

productivity that a sort of AI tool

might do but is actually the equivalent

of a researcher or close to it so so I

think maybe a way uh to look at it is to

give some illustrative examples of like

the kinds of capabilities that you might

see what we'll we'll have is intense

application of the ways in which AIS

have

advantages partly offsetting their

weaknesses and so AIS are cheap we can

call a lot of them uh to do many small

problems uh and so you'll have

situations where you have Dumber AIS

that are deployed thousands of times uh

to equal say one human

worker uh and they'll be doing things

like um these voting algorithms where

you with an llm you generate a bunch of

different responses uh and take a

majority vote among them that improves

performance sum uh you'll have things

like the uh alphao kind of approach um

where you use the neural net to do

search uh and you go deeper with the

search by plowing in more compute which

helps to offset the inefficiency and

weaknesses of the model on its own uh

you'll do things that would just be

totally impractical um for humans

because of the sheer number of steps and

so an example of that would be designing

synthetic training data uh so humans do

not learn by just going into the library

and opening books at random Pages um

it's actually much much more efficient

uh to have things like schools and

classes uh where they teach you things

in a an order that makes sense that's

focusing on the skills that are more

valuable to learn uh they give you tests

and exam they're designed to try and

elicit the skill they're actually trying

to teach um and right now we don't

bother with that because we can hoover

up more data from the internet we're

getting towards the end of that but yeah

as the AIS get more sophisticated

they'll be better able to tell uh what

is uh a useful kind of skill to practice

and to generate that and we've done that

in other areas so

alphao the original Al version of alphao

was booted up with data from Human

goplay uh and then improved uh with

reinforcement learning and mon Carlo

research uh but then Alpha zero with a

somewhat more sophisticated model uh

benefited from some some other

improvements um but was able to go from

scratch uh and it generated its own data

through

self-play uh so which getting data of a

higher quality than the human data

because there are no human players that

good uh available in the data set and

also a curriculum so that at any given

point it was playing games against an

opponent of equal skill itself uh and so

it was always in an area when it was

easy to learn if you're if you're just

always losing no matter what you do or

always winning no matter what you do

it's hard to distinguish

uh which things are better and which are

worse and when we have somewhat more

sophisticated AIS that can generate

training data and tasks for themselves

for example if the AI can generate a lot

of unit tests and then can try and

produce programs that pass those unit

tests uh then The Interpreter is

providing a training signal and the the

AI can get good at figuring out what's

the kind of programming problem that is

hard for AIS right now that will develop

more of the skills that I

need uh and then do them and now you're

not gonna have you know employees at

open AI right like a billion programming

problems that's just not going to happen

uh but you are going to have AIS given

the task of producing the enormous

number of programming challenges

so yeah we may have system

that have just not developed bad

motivations in the first place and be

able to use them a

lot um in developing the incrementally

better systems in a safe way and we may

be able to even if some of the early

systems do develop these bad motivations

if we're able to detect that an

experiment and find a way uh to get away

from that then we can win even if these

sort of hostile motivations develop

early when I combine the possibility

that we get relatively lucky on the

motivations of the earlier AI systems

systems strong enough that we can use

for some alignment research tasks and

then the possibility of getting that

later with AI assistance that we can't

trust fully where we have to have hard

power constraints and a number of things

to prevent them from doing this takeover

uh it still seems plausible we can get a

second saving throw where we're able to

extract work from these AIS on solving

the remaining problems of alignment of

things like neural ey detectors faster

than they can contribute in their spare

time

uh to the project of overthrowing

humanity hacking their servers and

removing the hard power and so if we

wind up in the situation where the AIS

are misaligned and then we need to

uncover those motivations change them

and align them then we get a very scary

situation for us um

because we might we need to do this

stuff very quickly we may fail but it's

a second chance where our work is just

evaluating outputs that the AI are

delivering having the hard power uh and

supervision to keep them from

successfully rooting the servers doing a

takeover during this process and have

them finish the alignment task that we

sadly failed uh to invest enough or

succeed in doing beforehand the

incredibly juicy ability that we have um

working with the AIS is

that we can have as an invaluable

outcome that we can see and

tell whether they got a fast one p past

us on an identifiable situation we can

have here's an air gap computer you get

control of the keyboard you can input

commands can you root the

environment uh and make a blue banana

appear on the screen even if if we train

the AI to do that and it succeeds we see

the blue banana we know we know it

worked even if we did not not understand

and would not have detected the

particular exploit that it used to do it

this can give us a rich empirical

feedback where we're able to identify

things that are even uh an AI using its

best efforts uh to get past our

interpretability Methods at what point

would it be the case that the AI is

contributing significantly in the sense

that it would almost be the equivalent

of having additional researchers to AI

progress and softare the thing to look

for uh is when is it the case that the

the contributions from AI are starting

to uh become as large or larger as the

contributions uh from humans so like uh

when this is boosting their effective

productivity by 50 or 100% And you like

if you then go from you eight month

doubling time say for Effective compute

from software Innovation things like

like inventing the Transformer or

discovering chinchilla scaling and do it

in your training runs more optimally or

creating flash attention it doesn't have

to have been able to automate everything

involved in the process of AI research

it can be it's automated a bunch of

things and then those are being done in

extreme profusion because any I think a

thing that a AI can do you have it done

much more often because it's so

cheap uh and so it's not a threshold of

this is human level AI it can do

everything a human can do with no

weaknesses in any area it's that even

with its

weaknesses it's able to bump up the

performance tens of millions of

gpus each is doing the work of maybe 40

maybe more uh of these kind of existing

workers this like going from a Workforce

of tens of thousands to hundreds of

Millions you immediately make all kinds

of discoveries then you immediately

develop all sorts of tremendous

Technologies so human level AI is deep

deep into an intelligence explosion the

intelligence explosion has to start with

something weaker than that yep yep yep

what is the point of which that feedback

loop starts where you can even you're

not just doing the 05% increase in

productivity that a sort of AI tool

might do but is actually the equivalent

of a researcher or close to it so so I

think maybe a way uh to look at it is to

give some illustrative examples of like

the kinds of capabilities that you might

see what we'll we'll have is intense

application of the ways in which AIS

have

advantages partly offsetting their

weaknesses and so AIS are cheap we can

call a lot of them uh to do many small

problems uh and so you'll have

situations where you have Dumber AIS

that are deployed thousands of times uh

to equal say one human

worker uh and they'll be doing things

like um these voting algorithms where

you with an llm you generate a bunch of

different responses uh and take a

majority vote among them that improves

performance sum uh you'll have things

like the uh alphao kind of approach um

where you use the neural net to do

search uh and you go deeper with the

search by plowing in more compute which

helps to offset the inefficiency and

weaknesses of the model on its own uh

you'll do things that would just be

totally impractical um for humans

because of the sheer number of steps and

so an example of that would be designing

synthetic training data uh so humans do

not learn by just going into the library

and opening books at random Pages um

it's actually much much more efficient

uh to have things like schools and

classes uh where they teach you things

in a an order that makes sense that's

focusing on the skills that are more

valuable to learn uh they give you tests

and exam they're designed to try and

elicit the skill they're actually trying

to teach um and right now we don't

bother with that because we can hoover

up more data from the internet we're

getting towards the end of that but yeah

as the AIS get more sophisticated

they'll be better able to tell uh what

is uh a useful kind of skill to practice

and to generate that and we've done that

in other areas so

alphao the original Al version of alphao

was booted up with data from Human

goplay uh and then improved uh with

reinforcement learning and mon Carlo

research uh but then Alpha zero with a

somewhat more sophisticated model uh

benefited from some some other

improvements um but was able to go from

scratch uh and it generated its own data

through

self-play uh so which getting data of a

higher quality than the human data

because there are no human players that

good uh available in the data set and

also a curriculum so that at any given

point it was playing games against an

opponent of equal skill itself uh and so

it was always in an area when it was

easy to learn if you're if you're just

always losing no matter what you do or

always winning no matter what you do

it's hard to distinguish

uh which things are better and which are

worse and when we have somewhat more

sophisticated AIS that can generate

training data and tasks for themselves

for example if the AI can generate a lot

of unit tests and then can try and

produce programs that pass those unit

tests uh then The Interpreter is

providing a training signal and the the

AI can get good at figuring out what's

the kind of programming problem that is

hard for AIS right now that will develop

more of the skills that I

need uh and then do them and now you're

not gonna have you know employees at

open AI right like a billion programming

problems that's just not going to happen

uh but you are going to have AIS given

the task of producing the enormous

number of programming challenges