so suppose that it's the case that in a

year a multimodal model can solve Arc

let's say get 80% whatever the average

human would get then AGI quite possibly

yes I think if you if you start so

honestly what I would like to see is uh

an llm type model solving Arc at like

80% but only trained on information that

is not um explicitly trying to

anticipate what it's going to be in the

AR test set but isn't the isn't the

whole point of Arc that you can't sort

of it's a new chart of type of

intelligence every single time is the

point so if Arc were perfect Flawless

Benchmark it would be impossible to

anticipate within the test set and you

know Arc was released uh more than four

years ago and so far it's been resistant

to memorization so I think it has uh to

some extent passed a test of time but I

don't think it's perfect I think if you

try to make by hand uh hundreds of

thousands of AR tasks and then you try

to multiply them uh uh by

programmatically generating variations

and then you end up with maybe hundreds

of millions of tasks uh just by brute

forcing the task space there will be

enough overlap between what you're train

on and what's in the test set that you

can actually score very highly so you

know with enough scale you can always

cheat

if you can do this for every

single thing that supposedly requires

intelligence then what good is

intelligence apparently you can just

Brute Force intelligence if if the world

if your life were athetic distribution

uh then sure you could just bruteforce

the space of possible behaviors could

like you know the way we think about

intelligence there are several metaphors

SEL actives but one of them is you can

think of intelligence as a past finding

algorithm in future situation space like

I don't know if you're familiar with

game development like RTS game

development but you have a map right and

and you have it's like a 2d 2D 2D map

and um you have partial information

about it like there is some fog of War

on your map there are areas that you

haven't explored yet you know nothing

about them and then there are areas that

you've explored but um you only know how

they were like in the past you don't

know how they like today and

um and and now instead of thinking about

Tod I think about the space of possible

future situations that you might

encounter and how they're connected to

each other intelligence is a past

finding algorithm so once you set a goal

it will tell you uh how to get there

Optimum

um but of course it's it's constrained

by the information you have uh it it

cannot pass fine in an area that you

know nothing about it cannot also

anticipate uh uh changes and um the the

the thing is if you had complete

information about the map uh then you

could solve the pathf finding Problem by

simply memorizing every possible path

every mapping from uh point A to point B

uh you could you could solve the problem

with pure memory but where the reason

you cannot do that in real life is

because you don't actually know what's

going to happen in the future life is

Ever Changing

I feel like you're using

words like memorization which we never

use for human children if if like your

kid learns to do algebra and then like

now learns to do calculus you wouldn't

say they memorized Calculus if they can

just solve any arbitrary algebraic

problem you wouldn't say like they've

memorized algebra they say they've

learned algebra humans are never really

doing pure memorization or pure

reasoning but that's only because you're

semantically labeling when the human

does the skill it's a memorization when

the exact same skill is done by the llm

as you can measure by these bench

marks and you can just like plug in any sort

of math problem humans are doing the

exact same as LM is doing which is just

for instance I know if you learn to add

numbers you're memorizing an algorithm

you're memorizing a program and then you

you can reapply it you are not

synthesizing on the Fly uh the addition

program so obviously at some point some

human had to figure out how to do

addition but like the way a kid learns

it is not that they sort of out from the

accents of SE Theory how to do addition

I think what you learn in school

is mostly memorization right so my claim is

that listen these models are vastly

underparameterized relative to how many

flops or how many parameters you have in

the human brain and so yeah they're

they're not going to be like coming up

with new theorems like the smartest

humans can but most humans can't do that

either um what most humans do it sounds

like a similar to what you're calling

memorization which is memorizing skills

or

memorizing um you know uh techniques

that you've learned and so it sounds

like it's compatible in your tell me if

this is wrong is it compatible in your

world if like all the remote workers are

gone but they're doing skills which we

can potentially make synthetic data of

so we record everybody's screen and

every single remote worker screen we

sort of understand the skills they're

performing there and now we've trained a

model that can do all this all the

remote workers are unemployed we're

generating trillions of dollars of

economic activity from Mii uh remote

workers in that world is are we still in

the memorization regime so sure uh with

memorization you can automate almost

anything as long as it's it's a static

distribution as long as you don't have

to deal with change