Speaker 0: yeah, it litererdiscussts me like i literally think it's in humane. my mission remains the same as it has been for, like twenty years, which is to stop people working like this jeremy howard,

Speaker 1: a deep learning pioneer. er a cangle grandmaster. he is a huge advocates for actually understanding what we are building through an interactive loop, a notebook, a repple the acts of poking at a problem until it pushes back back. he argues this is where real ininght happens,

Speaker 0: and this hunny thing they're both right right LM cospplay understanding things like they pretend to understand things, no one's actually creating shift times more high quality software than they were before. so we've actually just stone a study of this, and there's a tiny uptic, tiny uptic and what people are actually shipping. they thing about AI based coding is that it's like a slot machine and that you you have an illusion of control. you know, you can get the craft you're prompt and your list of mc pys and your skills and whatever. and then, but in the indiof, a pull l lvor, right, here's a piece of code that, that would understands. and am i going to bet my company's product on it? and i the answers i don't know because like II don't like like i don't know what to do now because no one's like being in in this situthat, they're really bad. it's software engineering uh. and then i think that's possibly always going to be true. they the idea that a human can do a lot more with a computer when the human can like manipulate the objects in inside that computer in real time, study them, move them around and combine them together. whoever you listen to. we know whether they move them or whatever like is, if you always always here from the great scientists, how they build deeper intuition by by building mental models, which they get overtime by interacting with the things that they're learning about a machine called kind kind of build at effective hiereracio distractions, about what the world isn't, how it works entirely through looking at the statisticcorrelations of a huge coppus of text using a declloning model that was my premise.

Speaker 1: this video is brought to you by envideo GTC. it's running much the sixteen until the nineteen teen ince and hoza and streaming free online. the key topics this year are agentic, AI and reasoning, high performance, inference and training, open models and physical AI and robot tics. i'm so excited about the DGX spari've been on the waiting list for over a year. now it's a personal supercomputer that is about the size of a mac mini. so the perfect aornment to a macbook pro by the way, and you can find tune a seventy billion parameter language model with one of these things. and i'm giving one away for free. well, you have to do is sign n to the the conference and attend one of the sessions using the link in the description. as for the sessions, i'm interested in attending amen, sang 's talk. so he's the cto of as and a session is code with context, build and ergentic tic d. the truly understandand your code base. now, obviously, jenson's kino is on mars to sixteen. he said he's going to unvy LA new ship that will surprise the world. so their gengenerararchitecture very a rubin is already in full producture theyre's speculation. we might even get an early glimpse of their new finineman architecture. if you don't forget focused, the link is in the description. if you're attending virtually, it's completely free. don't miss it. join me how to welcome to MLSTI.

Speaker 0: mean, welcome to my home. thanks are coming. yeah,

Speaker 1: where where where are we now?

Speaker 0: we are in beautiful motin bay in southeast queensland. we are by the sea in my backyard,

Speaker 1: whether didn't disappoint II certainly didn't.

Speaker 0: it doesn't often. but if you were here yesterday would have been very different.

Speaker 1: well, i don't know way to start i'i've ve been a huge fan, probably since about two thousand and seventy eighty. and of course, you had the famous you were in fit paper. and when i was at microsoft, i member doing a presentation about that because it is actually, i mean, now we take it for granted that we find tune language models on a copies of text. and then we kind of like continue to join them. and and specialize them, but apparently this was not received wisdom.

Speaker 0: now this was the first time it happened. yeah, kind of the first old second. so cockly and andrew die had done something a few years ago, but admissed the key point, which is the thing you pray train on has to be a general purpose coppus. so no one quite rezze this key thing. and maybe i had a bit of fortune here that my background was in philosophy. and cocology of science. and so i spbeen some decades thinking about this,

Speaker 1: the technical architecture of of ULM it. it just ckechch that up.

Speaker 0: i'm a huge fan of regusization. i'm a huge fan of taking a model that's incredibly flexible and then making it more more constrrained, not by decreasing the size of the architecture by bbby adding regularization. so even that at the time was extremely controversial, but that was by no mains a unique inside of of hours. so what's steven marity had done as you taken the extreme flexibility of an LSM kind of the classic stateful recurrent 're own it towards switch. things are kind of gradually heading back towards now. it is added five different types of regularizzation. he added every type of regularization you can imagine. and then that was the my starting point. this is say, okay, II have mmassively ly lelexible deep mining model that can be as powerful as i wanted to be. and it could also be as constrained as i neeit to be. and then i needed a really big corpposive text from enough. this is also even he had been a common crawl. and i think he helped or made the wikipedia data set you, then i realized actually the wikipedia data set made lots of assumptions. it had all these like ank for unknown words, because it all is huuned plastic NLP approaches. so i read the whole thing created a new weicoppedia data set. et now it's my general corpus, and then i used AWDLTM trained of. so it's actually overnight. so for eight hours on a gaming gpu, you know, because at the university of sentences ago, we didn't have hapes of resources, probably like a twenty IDTI or something i suspect. and then next morning, when i woke up, then it's the same three stage architecture that we do today. you know, pretraining the training post training. so then i figured it okay. now that i've trained something to predict the next way of wikipedia, let must know a lot about the world. i then figured, if i then fine tunit on a corpa specifics of what we could alcohol to pervised fine toting dataset, which, in this case was the dataset of movie reviews. it would become especially good at predicting the next word of those. so i'd learned a lot about movies did that for like an hour. and then like a few minutes of fine trauning, the downduring classifier, which was a classic academic data stets kind of considered the the ghdest one, which was to take like five thousand word movie reviews. and to say, like was a positive tive negative sentiment today is considered easy it at that time. you know, the other things that did it quite well, were highly specialized models that people wrote the whole phds on that. i beate all of their results. you know, five minutes later, what it five shooing that that model was amazing.

Speaker 1: and the other interesting thing is this kind of methodology around how you do the fine tuning air.

Speaker 0: so the how would we do? the finine ununing was something we dedeveloed at fast day. i so this is kind of yeah one of fast day eyes. so it's still still an every early days. and one of the extremely controversial things we did was, we felt that we should focus on fine tuning existing models because we thought fine tuning was important. uh some other folks were doing good contemporonouously that. so um jason is since keid, it's really, really great. um research, i thing it's during his phd, um how to find tratrade models and how good they can be in sather folks in the in the computer vision world. we were, you know, amongst first, there's a bunch of us kind of really investing in fine shooting. and so yeah, we felt that using a single learning rate to find true in the whole thing, all at once made no sense, because the different layers have different behaviors. and this is one of the things jacon you since kess research are so showed we developed this idea of like all. it's also way faster if you just train the last layer, right? because it really has to backproup the last layer. and then once that's pretty good backrop the next the last two. and then the last three and then we use something called discriminative learning rates. so different layers, we would give different learning rates too. and then another critical inside that no one realized for years, even though we'd told everybody was that you actually have to find tune every batch norm. so all the normalization is, you do actually have to find tune because that's moving the whole thing up and down. no, changing it scale. so yeah, when you do that, you can often just find you in the last layer or two, and we found that actually with you AM fit, although we did end up and freezing all the layers, only the last two were really needed to get the close to stay ad. the other result took like took like seconds.

Speaker 1: yeah, because the discriminaof learning rate thing is interesting because i think the received wisdom at the time was when you find una model. if the learning rate is too high, you kind of blow out the representations tions. so i guess the wisdom was, if if you don't have a really low learning rate, you're just destroy the representations.

Speaker 0: i know there was no received wisdom, because nobody talked about it. no one caare. i mean, there just no tight like on not nearly no one care. transfer learning was just not something anybody thought about. and racheland, i felt like it matters more than anything, you know, because only one purs and has to train a really big model once. and then the rest of us cannot plangunit. so i think we just to learn how to do that really well. so we spend a lot of time, just trying lots of things. but in the end, the intuition was pretty straightforward. and what intuitively saying like it all to work, basically always did work, which has another big difference between how people still today tend to do mail research, as i think it's all about abolations, and you can't make any assumptions or guesses. it's not at all true. i find nearly everything that i ct to work almost always works first time because i spent a lot of time building up. there's intuitions that kind of understanding of hell, gradients, behave.

Speaker 1: i think there's a diiccosto me though, between continual learning, which is when we want to keep training the thing, but maintain generality versus find ine uning a thing to do something specifis that there's always been this idea that, yes, you can make a model specific, you can bend it to your will, but you lose generality and you kind of degrade the representation. so tell me about that.

Speaker 0: yeah, there's some truth in that, although not as much as you might think think the whole, the big problem is that people don't actually look at their activations and don't actually look at their gradients. so something we do and our software and now fast day are software. we have built into what this ability as to seeing in a glance, what your entire network looks like. ononce, you've done a few times. it just takes a couple of hours to learn. you can immediately see. oh, i see this over tratraor under trained or this layer that something went wrong. it's not a mystery. it also basically what happens is, for example, you end up with dead ad urons that go to a point, whether they've got zero gradient regardless of what you do with that often happens if they get of head off towards infinity, you cannot always fix that. so yeah, it's not as bad as people think by any means something that trains well for continuous learning when done properly can also be done well to train well for a particular task.

Speaker 1: if you're careful in a sense, you do want the neurons to dial and and explain why what i mean by this, like we want to bend the behavior of models to introduce implicit constrainths. because, without constraints, there is no creativity. there is no reasoning, and so not so possiso. so in a sense, you actually wanted to say, don't do that. you wanted to do something else.

Speaker 0: i don't think of it that way like to be. it's more like i find thinyou humans extremely helpful when it comes to think about ai. i find they behave more similarly than differently. and my intriition about each tends to work quite well, you know, with a human when you learn something new. it's not about unlearning something else. and so something i always found is when i got models to try to learn to do to somewhat similar tasks, they almost always got better at both of them. that one that, that only learned one of them.

Speaker 1: i was reminded a little bit of, you know, where the dino paper from lecoon. and so this whole kind of regime of selsusupervised learning with with mean that, that was that, that was a vision model so that the idea was tracase. we're doing pretraining, and we want to maintain as much diversity and fidelity as possible. so that when we do the downstream task, we can kind of we got more things that we can latch on.

Speaker 0: yeah, yeah, but um, you know, sei, supervised and self supervised learning was such an unappreciated area. and yeah, young lukon was absolute in one of the guys who was also working on it. yeah, actually did a post because i was so annoyed healthy people. people cared about semi supervised learning. i did a whole post about it years ago. younlekcorn looked it for me as well, and you know suggested a few other pieces of work that i i'd missed. and but i was kind of surprised that how you know how incredibly suits useful it is to basically say, look, as you come come of the pretext task, right? so in i we did this envision before, you will them fit. so it's like in medical imaging, he ttake a histology slide and predict, you know, mask ked, a few squares and predict what is to be there. so some of my students USFI had doing stuff with that, it was basically entirely taking stuff the way and others had already done in vision. yes, ah, like this idea of masking out squares, we didn't invent it asking out words. but the obvivithing, you know know in ideidea, um gradually unfreezing layers. we had done before in computer vision. the whole idea of starting with a pretrained model that was general purpose have been in computer vision. there's a really classic paper actually in computer vision in i've been around twenty fifteen was entirely and miracle paper saying, look what it happens when we take a pretrained image net model, predicting what sculptor created this sculpture or predicting what architecture style this is. and like an an every task get got the state of yet result. and it really surprised me, people didn't look at that and think like i bet that ought to work in every other area as well, whether it be genome sequences or a language or whatever. but people have that of a lack of imation imagination. i find they tend to assume things only work in one particular field. that's really true.

Speaker 1: yeah, i guess ss, there's two things. i mean, festival. we were kind of hinting at this notion of almost good hearts law or the short that rule that you get exactly where you optimised for the cost of everything else. but that doesn't seem to be the case because we can, you know, optimize for perplexity in the case of language models. and actually say, what seems to happen is we're getting into the distributional hypothessary a little bit. so you know, you, you know, the word by the company kekeps. so when we have an incredible amounts of associtive data, it might be master red to prediction, or any of these things like that. the model seems to build some thing that we might cool. and understanding II foralways thought of it as a higher acacof of strstractions.

Speaker 0: if it's it's got a predict, you know, if the document is here was the little little opening that it old that bubby fisher used and has chest notation to predict the next thing it needs to know something about chest notation at least openings. if it's like, uh know know this. this was totoed by the nineteen and fifty six US president comma. you need to know like you don't even did, don't just need to know who the president was. but the idea that there are presidents and they folght the idea that there are leaders and theyfoll the idea that there are groups of people who have hieracy es, and they thoufore that there are people, and they thought that there are objects. and you can't predict the next way of a sentence well, without knowing all of these things. so that knowing my hypothesis for where i created you will enfit was to say it would end to to comppress that as well as possible to get that knowledge. it would have to create these abstractions these hierarchh's obstractions somewhere deep inside its model. otherwise, how could it possibly do a good job of predict an expert? you don't want because deep leadning models are universal learning machines, you know, and we had a universal way to train them. i think it, if if we get the data right tion, if the pt was good enough than in theory, we ought to be able to build that next word predicting machine, which ought to implicitly build a hiarchical structural understanding of the things that have being described by the text that it is learning to predict.

Speaker 1: i think that they can know in quite they they know in quite a superficial way. so there's a myriad of surface statistical relationships and they generalize extrorordinary. well, it's no exmiraculous is, but the thing thing is want want to conrost this with other comments you've made about creativities. and i think knowledge is about constraints, and i think creativity is about evolutionship of knowledge respecting those constraints. therefore, AI is not creative. and and you've said the same thing, you've said, AI isn't creative. so like on the one hand, how can you say that they know and not think that they can be created?

Speaker 0: i mean, i think think have fused that exact expression. you know, i have actually remember chatting with peter norvada on camera, and both of us said, well, actually, they kind of are creative like. so we just could be a good careful about that choices of words. i guess, so um know, pilter wasn't yet who's a guy. i really really respect who could of rediscovered space prepeative learning, built the supermemo system and is the modern day greuru of memory. the entire reason he's based his life around remembering things is because he believes the creativity comes from having a lot of stuff remembered, which is to say, putting together stuff you've remembered in interesting ways. it's is a great way to be creative. lms are actually quite good at that, but there's a kind of creativity. they're not at all good at, which is you know, moving outside the distribution. so which i think is where you're heading with your question. um but i just kind of i'm framing it this way to say you have to be so you once about this stuff, because if you say like they're not creative, i gives you the can give you the wrong idea, because they can do very creative saming things. but if it's like well, can they really extrapoulate outside the trading distribution? yes, there's no they can't. but the training english speciation is so big and the number of ways to interpreate between them is so vast. we don't really know yet what the limitations of that is, but i see it every day, you know, because i my my work is irond. i'm constantly on the edge of and outside the training data. i'm doing things that haven't been done before, and this is weird thing. i don't know you''ve ever seen it before. i see, but i see it moderal times every day, where the l en goes from being incredibly clever to like worse than stupid, like like not understanding the most basic fundamental premises about how the world works. yeah and is like, oh, worps. i feel outside the training data distribution. it's gone dumb, and then like it's no point 我是你家人 having that discussionary faith. yes, yes, know know ''ve asted at that point.

Speaker 1: yes. i mean, i love the margagat bobodens. she had this kind of iarqi of creativities. there's like combatorial exploratory and and transformative, but what models can certainly do commondatorial creativity. but for me, it's what about constraints. but that mean, it's is about bodendensaid and you've anido diventiy, he said the creativities, what about constraints and and you've spoken. and and we talk about this dialogue engineering. but what happens is when we we talk about language models, it's a specification acquisition problem. so we go back and for often, actually, when we think the process of intelligence is about building this imaginary lego block in our mind and respecting vous constraints. and when you respect those constraints and you just continue to evolve, then those things are said to be creative. so language models, when you add constraints to them. so this could be via supervision via tictics for a verifies, then they are creative, and we alpha evolve. we see many examples of this, but the illusion sion is on their own constraints. obviously, ly, they have this behavioral shaping stuff that we're talking lob that don't have hard constraints, and that's why they can't go outside their distribution.

Speaker 0: i mean, i think they can't go outside their distribution because it's just something that uum that type of mathebetical model can't do. you know? i mean, it can do it, but it wonder it. well, you know, when you look at the kind of two d case of of fitting a curve to data, once you go outside the area that the data covers the curves disappear off into space in wild directions, you know, and that's what we're doing that. we're doing it in multiple dimensions yethink. i boboden might be pretty shocked at alfa. compositional. creativity can go when you can compose the entirety of the human knowledge corpus. and i think this is where people often get confused, because, as i so, for example, as talking to chris latner yesterday about have clad anthroropic ito, had had got lalad d write the sick compililer. and they were like, oh, this is a clean room sakon pilot. you can. it's it's clean room because it was creit in rest, you know. and that's a chris credit that kind of you know, i guess it's probably the top most widely used to say, say, plus class compiled out, it is playing on top of LLVM, which is the most widely used kind of foundation for compilers. though i go. chris didn't is rest this, you know, and we didn't give it access to what he compile a source code. it's a clean room implementation, but that misunderstands how LMS work, right, which is all of chris 's work, wasn't the training data. many, many times. LLVM is used widely and lots and lots of things are built on it, creating lots of sea and c plus plus compilers, converting a terruisn't, an interpulation between parts of the training data. you know, it's a style transfer problem. um so it's definitely compositional creativity at most, if you can call it creative at all. and you actually see it when you look at the the repole that it created, it's copied parts of the ellaa vencode, which today, chris says, like, oh, i made a mistake. i shouldn't have done it that way. nobody else does it that way. you know, oh, well, look, they're the only other one that did it that way you that doesn't happen accidentally that happens because you're not actually being creative. you're actually just finding the kind of non linear average point in your training data between like finding the building and compilor things.

Speaker 1: 好的 all of that as tram mean, first stivlife. i think we shouldn't underestimate these size of how big this combinatorial creativity is. so all of that is true. so the code is on the internet, but also they had a whole bunch of sts, which was scacuffolded, which meant that every single time some code was was committed. they could run the test and and they basically had a critic, and they could then do this autonomous ous feedback loop. so in ina sense, it's very similar to the recent research by open ai and geerine. i you, you're trying to solve a problem in math, and you already have an evaluation function. the same on the ok, ze, right? you have an evaluation function, and what people discount is, even knowledge of what the evaluation function is, is partial knowledge of the problem. so you can then brute for search, you can use the statisticpattern match and use the verifiy as a constraint, and you can actually,

Speaker 0: and they don't even need to do that, right? like you literally already know how to pass those tests because it's lots of software ether already does it. so it just uses that and translates them to us. i guess that's all it did um which is impressive um and if i'm much less similar with math than i am computer science, but from talking to mathematicians, they tell me that that's also what's happening with like adults, problems and stuff. it's some of them are newly solved.

Speaker 2: yeah,

Speaker 0: but they are not sparks of insight. you know, theare solving ones that you can solve by mesing up together very close regulated things. so humans already figured out so on the subject of claud code.

Speaker 1: now i know you spoke an extensively about five coding. i'd actually rachel had some interesting workout. i mean, she quoted the the meter study, which showed that productivity actually went down when people were vipe coding. but i think and i thought that they went when i wwand and think was there was the anthrotics study mean. and maybe we should rewonder of that mendario had the SA out the other day, i think, was called the athlesence of technology gy something like that. and he was basically saying, look, you know, and we have all these amazing suffreiengineers and ropic, and they are just so productive, and he was extrapulating to the average sofreengineers. so there's going to be mass unemployment because soon we're going to be to automate all of this where they are. i mean,

Speaker 0: it doesn't make any sense. um elon musquks said something a bit similar few days ago, saying like, oh, hellems have just spit out the machine code directly. we want need libraries, programming languages, yeah yeah. look, the thing is none of these guys have have been ssofare engineinerecently. i'm not sure sharrias have have been a software engineer at all software engineering as a unusual discipline, and a lot of people mimike it for being the same as typing code into an ie coding is another one of these style transfer problems. you take a specification of the problem to solve, and you can use your compositional creativity to find out the parts of the training data, which interpulated between them, solve that problem and interprelate that with syin tax of the target language. and you have get cold. there's a very famous sa o pread brooks written ten many decades ago, no silver bullet, and which it almost sounded like he was talking about today. he was specifically saying something he was just putting saying very similar, which is those days as all like ah, what about all these new fourth generation languages and stuff like that? you know, we're not going to need any need coatas anymore any software engineers anymore because softwis now so easy to write anybody can write it um. and he said, well, i guess you could could at maximum m thirty percent ent improvement. specispecifically said a thirty percent improvement in the next decade. but i don't think he needed to limit that much because the vast majority of work in software engineering isn't typing in the code. um so in some sense, parts of watdariia said, we're right just like for quite a few people. now most of their code is being typed by a language model. that's true for me, say, like maybe ninety percent, but it doesn't make me that much more productive because that was never the slow bit. it's also helped me with kind of the research charge and figuring out, you know which files are going to be touched. but any time i've made any attempt getting an an lemonto lake design, a solution to something that hasn't been designed lots of times before. it's it's horrible because what it actually every time gives me is the design of something that looks on its surface a bit similar. and often, that's gta be an absolute disaster because things that look on this surface bit similar. and like i'm literally trying to create something new to get away from the similar thing. it's very misleading.

Speaker 1: first of all, i'm i'm exhauerated by what i see as the tech bro prede election to misunderstand coglesitive signs and philosophand. i'm on us because we've we spoken to so many really interesting people on emolesty, like, for example, says a hiddewgo, he wrote to mispoook the laws of knowledge and and even a movii to erimmoter. she's a philosopopher of neuroscience. and she was talking all about, you know, lithe ficicting. it basically that knowledge is protein. so yeah, i think that that knowledge is perspect divial. i don't think that knowledge can be this abstract perspective free thing that can exist on wikipepediand. and i also think that knowledge is evoldiand. it's alive. it's it's something that exists in in us. and the purpose of an organization is to preserve and evolve knowledge. so when you start delegaate eighating cognitive task to language models, you actually have this weird paradoxicical effect that you erode the knowledge inside the organization.

Speaker 0: well, that's true and that's terrifying. it's often these these arguments online between people who are like ellams don't understand anything. they're just pretending to understand. and then other people were like you don't be ridiculous. look, what this LE mm is did for me, right? and it's one thing is theyy're both right. l lms cos play understanding things, they pretend to understand things. and this is the interesting thing about the early kind of work with, like popular science ence work with, like daniel dinner. um that's basically what the chinese room experiment is is you've got a guy in a room who can't speak chinese at all, but he sure looks like he does, because you can fading questions, and he gives you back answers. but all he's actually doing is looking up things in a huge array of books or machines, or whatever the difference between pretending to being intelligent and actually being intelligent is entirely unimportant, as long as you're in the region in which the pretense is actually effective, you know. so so it's actually fine for great many tasks that ellenms only pretend to be intelligent, because for all intense and purposes, it just doesn't matter until you get to the point where i can't pretend anymore. and then he realize like, oh, my god, this thing so stupid.

Speaker 1: i'm a fan of. so by the way. so you know, he said that i'm understanding is cosely reduceable, but ontologically irreduceable, and he was saying there was a phenomenal components underunderstand. we don't even need to go that like the interesting thing about knowledge being protein. is this idea that, you know, spbasically, this cantian idea, the world is a complex thing. none of the understanding. it's like the blind men in the elephant. we all have different perspectives through a complex thing. and so we all we all do this kind of modeling ling. but the interesting thing is that the language model sscitimes, they seem to understand and they understand, because the supervisor places them in a frame. so inside that frame. so when you have that perspective of the elephants, they're actually surprisingly coherent, but we discount the supervisor, placing the models in that frame.

Speaker 0: yeah, yes. so so cell vis is neor or vers cell. and then it but what everybody was talking about when i back when i was doing my undergrade in phosophy y does. so inconsciousness explained came out about then probably chinese room little bit before that's interesting because the discussions were the same discussions were having now, but they've gone from being abstract discussions to being real discussions. it's helpfully people go back to the abstract discussions because it's it helps you get out of your, you know, it's very distracting at the moment to look at something that's cause playing intelligence so well and go back to the fundamental question. um anyway, i just want to mention that kind of this. this is interesting situation were now in where it's very easy to really get the wrong idea about what i can do, particularly when you don't understand the difference between coding and and soft engineering. yeah, then takes video point point. are your question about about the implications of that for organizyes? you know, a lot of organizations basbasally batting their featachres on a culative premise, which is AI, is gotta be able to do everything better than humans. ah, h least everything encoding better than humans. II worry about AA, both both the organanizations and for the humans, you know, for the humans when you're not actively using your design and engineering and coding muscles, you don't grow, you might even wither, but you at least don't grow and youyou. know speaking of the CEO of an marane start up. you know, if my stuff aren't growing and we gotta fail. you know, we can't let that happen and getting better at the particular prompting skills, whatever details with the current generation of AICI frameworks isn't growing. you know, that's that's like that's as helpful as learning about the details of some AWS api when you don't not want to understand how the internet works. you know, it's not. it's not reusable knowledge. it's a femoral knowledge. so like if you wanted to, you can actually use it as learning superpower, but also it can do the opposite. it know a natural thing. it's going to do with remove your competence over time.

Speaker 1: i agree that that's the natural thing. so and this, this is especially personal view, because your career has been around, basically educating people to forget, you know, technology in AI literacy. so the deep behavior is very similar to a self driving car that you know, there's this tipping point where, at some point, you're not engaged anymore, you're not paying attention and you get this delegation of competence, and you get understanding dead. that's the deep potention. so the study from anthropic couple of weeks ago. so contradicted diario completely because it even said, yeah, there were a few people in the study that were asking conceptual questions that are actually kind of keeping on top of things. and they have a gradient of learning, but most people didn't. and my hypothesis about that is, you know, the ideal situation for jn AI coding. is that like us, we've been writing ssoftware for decades. we already have this abstract understanding. we're using it in demands that we know well, and we can specify. we can remove loads of ambiguity. we can track and we can go back and forth, and we can we can stay in touch of the process. but what happens is that, that the default the track is for people to just go into this autopilot mode, and they got no idea what happening. and it's actually making them dummmi.

Speaker 0: i created a, the, the first deep learning for medicine company called dealytic. c acking was elect twenty fourteen, and our initial focus was on ideology, and a lot of people were worried. this would caused radiologists to become less effective or radiology. and i strongly felt the opposite, which is that i did quite a bit of research into this, like what what happens when re's like, fly by wire in eraplanes or antiop breaks in cars, whatever, if you can successfully automate parts of a task that really are automable, you can allow the the expert to focus on the things that they need to focus on. and we WW happhapps so radiology we found if we could automate identifying the possible nojeells in a long CT skcan. we were actually go at it, which we were. and then we've that the radiologist thing can focus on looking at the nodules and try to decide if they're malignant are what to do about it. so again, it's one of these sebtle things. so if it's things which you can fully automate effectively in a way that you can remove that cocktatative burden from a human so that they can focus on things that they need to focus on that can be good. you know, i don't know where we sit in software development because i've been coding for for forty bish years. so ve written a lot of code, and i can plance at a screen of code, and then don't don't less, it's something quite. we're not specisticated. i can immediately tell you what it does and whether it works. and whatever i can kind of see intuitively things that could be improved, you know, possible things to be careful love, and not sure i could have got to that point if i hadn't have written a lot of code. so the people i'm finding who can really benefit from AI right now, either really junior people who can't code it all who can now write some apps. they haven't 't hidden. it's long as they work reasonably quickly with the current i capabilities, then they happy. and then really experiences people people like me or like chris latina, because we can basically have a dosome of a typing for us, you know. and some of our research for us, people in the middle, which is most people, most of the time. it really worries me because how do you get from point eight of point b, yes, without typing code, it might be possible. but we don't have a we have no experience with that. we don't is it possible. how would you do it like? is it kind of like going back to school? we're a primary school. we don't let kids use calculators so that they develop their number muscle. do we need to do that for like first five years as a developer, you have to write all the coach yourself. i don't know, but if i wasn't it, the seen like two and twenty years of experience developer, i'll be asking that question of myself a lot, because otherwise you might be in the procesve, making yourself obsolete.

Speaker 1: yeah. well, this is another thing about knowledge that this sessor hadoggo guys said. so he said that knowledge is not fununible, and that which means they can't be exchanged. so what what he means by that is, the process of learning is in some important sense, not reducable, right? so you have to have the experience of the experience has to have friction. and when we build models of the world, and we actually learn you, you. there's this phrase reality pushes back. so we make lots of mistakes, and we update on models wewe're just placing these coherence constraints in in our in our model, and that's how we come to learn to use core code. and there's so little friction in the process that's exactly what they stustudy from. anthropics said is said there is so little friction. they didn't learn anything, right?

Speaker 0: yeah. now exactly um desirable difficulty is the concept that kind of comes up in education. but even going back to the work of uebing house, who was the original respect, repeatea space learning guy in the one nine hundred and century. and then peta wasn't near more recently, it we find the same. we we know that memories don't get formed unless it is hard work to form them ah. so you know, that's way you kind of get this somewhat surprising result that is revising true often is a bad idea because it comes to mind too quickly. and so with repetitedly space learning with stuff like anki and supermemo, the algorithm tries to schedule the flashcards at a just before the moment you're about to forget. so then it's hard work. so II studied chinese for ten years um in order to try to learn about learning myself, and i really noticed this that i is dankiy. and because it was always scheduling my cards s. just before i was about to forget them, it was always incredibly hard work, you know, to do reviews because almost all the cards through once. so this on the verge of forgetting, i was absolutely exhausting, but my god, it worked well here i am. i don't really haven't done any study for, fifteen in plus years, and that's still all remember my chinese.

Speaker 1: i know i may also me coming back to your radiology example. um like one example, people give is cool centres. so you know, we have this notion that an organization. we have high intelligence roles and low intelligence roles. and for me, intelligence is just the adapted acquliztion and synthesis of knowledge. so we assume that, that know the low intelligence roles and low intelligence roff. and it doesn't adapt, which means we can there are cercertain that that an organization does that do not change. so we could automate, and we don't need to update on knowledge. and i think that discounts actually may be with the radiology example that having this holistic knowledge, like you know, in a core centre, there are so many weird educaes that comin so many weird things happen, and that filt is up in the organization, and we adapt over time. so when you start to automate things and you actually lose the competence to create the process, which created the thing in the in the first place. and you lose the evolizility of that knowledge in your organization. you're actually kind of cutting your legs off.

Speaker 0: yeah, absolutely. and um so i know know i know was in in my company. i just i tell our stuff all the time. almost the only thing i care about is how much your your personal human capabilities are growing. you know, i don't actually care how many PS you're doing everyfeatfeatres you're doing like there's that nice yeah johnose to help the tc yoga brrecently released some of his, his stanford friday take away lextures. and he has his nice one called a little bit of slope, makes up for a lot of intercept. just basically the idea that, that you know, in your life, if you can focus on doing things that caused you to grow faster, it's way better than focusing on focusing on the things that you 're already good at. you know, that has that i intercepcept. so the only thing i really care about, and i think, is the only thing that matters for my company is that my team i'm focusing on their slope. if you focus on just driving out results at the limit of whatever AI can do right now, you're only caring about the intercept, you know. so i think it's basically a pass to obsolessons through out the company and the people who in it. it's i've really surprise how many executives of big companies are pushing this now because it feels like if they're wrong, which they probably are, and they have nowhere to tell if they are, because this is an area they're not at all familiar with. they ever learned ted their mba thethey're basically setting up their companies to be destroyed and really surprised that you know, shareholders would let them do that. you'll set up such an incredibly speculative action. yeah, here we ah feels like a lot of companies you're gonna fail as a result of the amass tech that that causes them to not be able to maintain or build their products anymore there there.

Speaker 1: lolos of folks out that are like france was relallike. he really, really gethis it. is he understand this? and you know, so he's always said that it's it's about kind kind of mmetic sharing of cognitive models about the domain and how we refine it together on the sharing thing. this is another big scaling problem with gena. i coding this. so the the ideal case i've done this, i know ddeain really well, and i can specify with exquisite detail, and i told cloud code gogoing 't do this thing. and the models in my mind doesn't matter um and then you go into an organization, and now i need to share like my knowledge of all of the other code. and i'm sure you have this sennual companwell like you need to that. this knowledge acquisition. bottneck is a real serious problem in an organizations. so when it's just me, i think i'm probably about of times more productive using clawd code. it's absolutely magic. and i can see why people are so excited about it. but people, i you seem to understand the bottonic and and how that doesn't really translate to many real world organization.

Speaker 0: no one's actually creating fifty ty more high quality ality software than they were before. so we've actually stand to study of this. and there's a tiny up, tick, tiny upck and what people are actually shipping. that's the fact, obviously, i ve an enthusiasst of AI and what it can do. but also, my wife, actual recently pointed out an article, all of the pieces that make gambling addictive are present in. yeah, dog flow.

Speaker 1: yeah, i was. it's it about voting.

Speaker 0: yeah. it's this really awkward situation where it's very awkwding know. it's very, very thuusitic about eepowered coding in recent months have totally changed their mind about it when they finally went back and looked it like how much stuff that i built during those days recquired enthusiasm, am am i using today? my customers using today? am i making money from today? almost all the money is being made by influences, you know, or by the companies that produce the tokens. they think about AI based coding is that it's like a slot machine and that you, you have an illusion of control. you know, you can get the craft you're prompt in your list of mc pays and your skills and whatever. and then but in the end, your ll vlavor, right you put in the prompt and something comes back. and it's like cherry cherry, as like oh, next time i'll changed my prompt a bit. i'd a bit more context, pull the lave or again pull the lavor. again, it's this darastic thing. you get the occasional win. that's like, oh, i won. i got a feature, so it's got. it's got all these horacks of like lost disguised as a wind of somewhat sarcastic feeling of control, all the stuff that gaming companies try to engineer into their gaming rooms. another, that means that AI is not useful, but gosh 's had to tell.

Speaker 1: i know, and rachel just just just to be close. sure. she also said that one of the hormarks of gambling is that you kind of delude yourself that you have some awareness. so what's going on? but it, but actually you don't, but let's do the bull case a little bit. so on the i do think in restricted cases it it is it is very useful. and these are cases where we understand. and and we can place constraints specificficbut. but even in those cases, you could argue on the one hand that we're not. we're not going to be unemployed anytime soon because you just just do more work on the addiction thing. i've noticed that. so i've had forty hour clawd code marathon sessions. and and i actually feel addicted to it like a spot machine. you know it, it really is,

Speaker 0: and they too absolutely yeah.

Speaker 1: i know it's and just i've never felt more drained writing, and i actually need to take a rest after it's like a few days rest, because it it completely.

Speaker 0: and i re's crap, you know, yeah, definitely. i had some successes, right? and so in fact, we've spent the last couple of years building a whole product based around where we know the successes are going to a be, which is when you're working on recently, small pieces that you can fully understand and that you can design, and you can build up your only only as abstraction to create things that are bigger than the patthat. you're building out of had a very interesting situation recently where i just kind kind of experiment, basically, which is we we rely very heavily on something called ipe kernel, which is the thing that's powers, stupid and nobooks. and they're been a major version release of vii pacanal from six to seven, and it stopped working. and it stopped working. and both of the products that we were try to use it with. one was whowas called today ND classic, which is see original jupiter dotebook, and then our own product called solve it. it were just ranomly crash um and ipad canandls over five thousand lines of code. it's very complex code multiple threads events, locks interfaces with i python, you know, with with said MQ, you know, all kinds of different pieces, debag pie. and i couldn't get my head around it, and i couldn't see where it was crashing the test to all passing. i wonder if AIE consolve this get us like i'm always interested the question of like how bigger chunk today. i handle on its own right now. the answer turned out to be. yes. i think it can just it was like, so it's been a couple of weeks. i didn't develop a lot of understanding about how ipad phone or really work in the process, but i did. it's been quite a child kind of pulling out separate comment like, so the answer was in two hours codex five point. i think it was popwite two at that time or maybe three year, just come out, couldn't do it. then if i got the two hundred dollar a month, GPT five point three pro to fix the problems equal. and so by rolling back between those two pieces of software and those too much models, i could get things working uh over a couple of week's period and luck. y say, wasn't at all, thought it very tiring, and it felt stressed because it wasn't really in control. but the interesting thing is, i now am in a situation where i have the only implementation of an of a python jupparkernel that actually works correctly as fas. i can tell with these new version, seven protocoal improveonments. and now like this is fascinating because we don't have a kind of a software engineering theory of what to do. now, like here's a piece of code that, that would understands. am i going to bet my company's product on it? and i the answers, i don't know because like II don't like II don't know to do do now, no one's like been in this situation. and like, will it does it have memory leagues? will it still work a years time if this mind to change the protocol? is there some weird edge case that's going to destroy everything? no one knows because no one understands his code. it's a really curious situation.

Speaker 1: it mean festival, we should acknowledge the peninicious erosion of control. so at the very beginning, you have ten percent AI generated code, and then you can just see how it creeps up and up. and then at some point, six months down the line PR comes in. and now you know, sixty percent of the code is i generated. and and and you see what you see what happens you, you see slowly become disconnected. but the bullcase for this, you know, in AOI, there is an idea called functionalism that we don't care what the intelligent thing is made out of as long as it and does all of the right things. then we know, you know, we would say it's AI, and it's the same thing of software. so the bull cases II understand the demand, i don't need to rights and don't need to know how to write the quick sore alaloriththm. i just need to. i just need to understand it. and then and then you know, so i just need to have all of these tests, and it needs to go into deployment. and these things need to happen. and at that point, you know, i don't actually care. and i could also,

Speaker 0: i quite that i quite that be clear. i quite like that framing, but you're what that actually does says, well, software engineering sure is important. then because after engineering is all about finding what those pieces are and how they should behave, and then power, you can put them together to create a bigger piece. and then how you can put them together to create a bigger piece. and if we do that well that in tenious time, we could have software that it's far more capable than anything we could even imagine today. but you're really gonna get that we're really great sofoftware engineering. yeah, you want to be careful. i think in the end, like i pad coranel, i'm finding, for example, it's just two bigger pace, right? because in the end, the time that made the original ipc ernewe're not able to create a set of tests that correctly exercised it. and therefore, real world downraam projects, including the original NA classic hknow, which is what ipc cha was exjected from, didn't work anymore. so this, this is kind of where our focuses are now on the development side at that answer. i it's finding the right size pieces and making sure they are the right pieces lowing how to recognize what those pieces are and how to design them, how to put them together is actually something that normally requires some decades of experience before you really good at it um saying it's true for me. and where can i got pretty good at it after maybe twenty years of experience? yeah, the big question is, like, how do you build the software engineering chops, which are now even more important than theyever been before? they're the difference between somebody who's good at writing computer software. and somebody who's not that feels like a challenging question.

Speaker 1: i know there's also this notion that there are so many different ways to abstract and represent something. you know, the world is a very complex place. and maybe the way we've been abstracting representing software is mostly a reflection of our own cognitive limitations, right? and even in the sciences in in physics, you tend to have a lot of quite reductive methods of modeling the world. and then you've got complexity science, which is just embracing the constructive disability tive, you know, narly nature, tive of things. and i think a lot of software today, we don't underunderand right. so for exexple ple there, many globally distributed software applications that use the actor pattern 你的. and this is just this is it's basically like a complex system, right? and the only way we can understand it is by doing simulations and tests, because no one actually knows how one of these things fit together. so you could argue, i guess, as a bullcase that maybe we already are doing this at the top of the suffiengineering, and that is what we want to do. eventually anyway,

Speaker 0: yeah, i'd say probably not you see companies like instagram and what's sap dominate their sectors, wealth having ten stuff and beiding companies like google and microsoft in the process. i would argue this way of building software in very large companies is actually failing. and i think we're seeing a lot of these very large companies becoming, you know, increasingly desperate. and you know, for example, the quality of microsoft windows and microo s has very obviously deteriorated greatly in the last five to ten years. you'll back when dave katler was looking at every line of the NT kernel and making sure it was beautiful it. it was a elegant and marvelous pece of software. you know, i don't think as anybody in the world, he's going to say that windows eleven is an elegant and marvelous pece of software. so actually think we do need to find these smaller components that we do fully understand. and that we need to build them up. and here's the problem. um AI is no good at that. so and so i say that imirically they're really bad. it's software engineering. then i think that's possibly always going to be true, because you know, we're we're asking them to often outside outof their training data. you know, if we trying to build something that literally hasn't been built before and it in a better way and has been done before. we're saying, like, don't just copy what was in the training data. so um and again, this is a confusing point for lot of people because they see AI been very good coding. and then you think like all that's software engineering. yeah, it's was god must be good a software engineering, but it's they're different tasks. there's not a huge ament of overlap between them, and there's no current empical data to suggest that LM ms are gaining any compedency at software engineering. every time you look at a piece of soft engineering. they've done like the browser for examwhich, um curse a created or the sea compiler, which um antropic can created, like i've read the source code of those things quite a bit. um chris lton is much more than ly with the compiled. examples may. um but there are their very, very obvious copies of things that already exist. so that's a challenge. you know, is if you want to build something that's not just a copy, then you can't outsource that to an LM. there's no theoretical to reason to believe that you'll ever be able to. and there's no empical data to suggest that you'll never be able to.

Speaker 1: yes. i think the punch line of this conversation is, and i'm i'm sure you degree with this that we need to have the combination of AI and humans working together, right? because the huhus s provithe, the understanding and all of the stuff we were saying about knowledge, but we can still use AI as a tool, but we need to we need to design operating models or ways of working that make that we say we we don't want to diminish our competence and understanding, right? it's so it's a very, very ine light.

Speaker 0: that's that's paying our focus, and we both focus on that for teaching and for own internal development. the stuff i've been working on for twenty years has turned out to be the thing that makes this all work. stephen wolfm should get credit for this. who was the guy that created the notebook into face, although also lots of ideas, kind of go back to smsmall talk and lisp and APO the basically the idea that human can do a lot more with a computer when the human can like manipulate the objects in inside that computer in real time, study them move them around and combine them together. yeah, that's what small talk was all about, you know, with objects and APO with a same with a raise thethebetica. basically is a superpowered list, which they're also added on this very elegant notebook interface that allowed you to construct is a living document out of all this. um. so i build this thing called NBDFA few years ago, which is a way of creating production software inside these notebook interfaces inside these rich dynamic environments. and i found that made me dramatically more um productive as a programmer ing like even even though i've never been a full time. programmer is my job when you look at my kind of get have breail up, what i think get have producsome statistics about it. and i was like just about the most productive programmer er in australia, you know, like it's working. and a lot of the stuff i build has lots and lots of people use it because it's such a rich, powerful way to build things. and so it turns out weve ve discovered that if you put ai in the same environment with a human i get in AA, rich interactive environment, AI is much better as well, which perhaps isn't shocking to hear. but that the normal, like if you use cloud code, i know you do. and it's a very good piece of software. but the environment we give cloud code is very similar. the environment that people had forty years ago. you know, it's it's a line based terminal interface. you know, i can use MCP. you know, have most of the times. it's just nowadays uses bash tools, which again, very powerful. i love bash tools. i use them all the detail CI tools all the time, but it's still just it's using text files. you know, as it's as it's interface to the world, it's it's it's really mega. so so we put the human in the AI inside a python interpreter. um and now suddenly you've got the full power power, very, very elegant, expressive programming language that the human can used to top the ii. the ii can top the computer, the human can tok to the computer. the computer can top the ii like you have this really rich thing. and then we let the human and the II in real time build tools that each other can use. and that's what it's about to me, right? it's about like creating an environment where humans can grow and engage and share. it's like for me when i use sovet. it's the opposite of that experience. you describe with cloud code after a couple of hours. i feel energized and happy and fulfilled.

Speaker 1: i give you give you my site. i think that the thing that you're pointing to here is there's something magic about having an interactive stateful environment that gives you feedback. and that is because our brains kind of they they can do a certain, you know, unit of work. so so we actually think through refining and testing with reality, and that's why i'm i'd doing my PHDI use mathemagic and mat lalab. and i agree. so if we got this rapper environment and you know, here's the matrio, do an image plot, you know, do a change. this is what it looks like now. and it's actually a wonderful way to kind of just just refine my mental model about something. and but but claud code does a lot of this stuff. II think it's mostly a sill issue. i think the people that use cloud code effectively do this. i've wrwrten a contcontact management possipossible it possipossible possipossible so you i've writa contact management system called led script. and when i'm putting together a documentary video, it can go, it can it can pull transcripts, and then i can verify the claims. and you know, part of the illiteracy is just understanding the a symmetry of language models, right? so when you give them a sort of discriminative task, they're actually quite good. so if i tell it in a subbagent to go and verify every individual claim, it's much more accurate. vify was in genereration mode. i was generated a bunch of claims and the stateful feedback thing. again, you know, i can have some kind of like schematized examell dump. and i can have like a an application here on the side, which is visualizing, and it's like a feedback loop. and for me, this is an AI literacy thing. the the good people AI are already doing this.

Speaker 0: yes. so i thought fully agree with you. i agree. you can do it in cloud coand. i agree, it is AA literacy thing as to whether you can, but also cloud code was not designed to do this. it's not very good at it it, and it doesn't make it the natural way of working with it. i don't want to say it's an a literacy problem because that's like saying, like, oh, it's a you problem to me, if at all, is not making it the natural way for a human to become more knowledgeable, more happy, more connected with a deeper understanding and a deeper connection to what they're working on. that's a whoproblem that, that should be how tools are designed to work. so so many models and tools expressely are being evaluated on. can i give it a complete piece of work and have it to go away and do the whole thing, which feels like a huge mistake to me. this is, have you evaluated whether a human comes out the other end with a deep understanding of a topic, you know, so that they can really easily build things in the future.

Speaker 1: i agree with all of that, but then is the other interesting angle, which is that there there is famfamous talk by joe grooce. and we've talk about this, and he said that notebooks are terrible. they're really bad from a sufreiengineering point tity. and and at the time, and maybe still one out of certain extent, i agree with him, because 我 um you know, i've i've done m dedevelos i'worworin larlarge organzzations know trying to figure out how do we bridge like data science and softwer engineering and claud code is already more towards a sffer engineering side. and what that means is it creates eyedepopoant at stateless repeatteable artifacts right? so as you say from a pedogogical point of view, because i good having this stateful feedback, because because i understand what's t's going on, but then i need to translate that into something which is deployable. and can you tell us the story of you responded to john gruce in new? and it was a bit of a fiasco, wasn't it? but just just tell us about that story,

Speaker 0: you did a really good video called. i don't like noteooks. um it was hilarias. it it really well ddone. and yeah, it's totally wrong. and all the things he said, notebooks can't do they can. and all the things they said, you can't do with no books. i do with no books all the time. so it was a very good, very, very uuum incorrect talk. so then i did the kind of a parody of it called um i like notebooks um in which i basically copied with credit, most of his slides and and chold how every one of them was totally incorrect. but like i actually think you're coming about, it does come down to the heart of it, which is this difference between, like how software engineering is normally done. this is how scientific research and similar things is normally done. and i think and i agree, there is a dichoomy there. and i think that economy is a real shame, because i think software development is being done wrong. it's being done in this way, which is yell about reproduceability. and these like dead these dead paces. you know, it's all dead code dead files. i will never be able to express his one million of his clearly, as as bread victor has in his work. so i'd encourage people who haven't watch bread victor to to watch him, but know, know he, he shows again and again, how a direct connection, you know, a direct vistival connection with the thing you're doing is, is all that medits, you know, and that's his mission is to make sure people have that connection. and that's basically my mission as well. so for me, traditional software engineering is as far from that as it is possible to get. i think it's i think it's gross th. i find it disgusting, and i find it sad that people are being forced to work like that like i think it's humane. and i just didn't think it works very well. i mean, empirically it doesn't worfor very well, and it's much less good for AI as well as it's much less good for humans. it hasn't always been that way like it know with with alan, an k and small talk can ivson, and and app l, you know, lispible from with mathematica. to me, these was a golden days when when people were focused on the question of how do we get the human into the computer to work as closely with it as possible? you know, that's whether the mouse came from example, like you got like, like can drag and visualize entities in your computers, things things can move around. so i feel like we've lost that. i think it's really sad, yeah, with cloud code and stuff the the default way of working with them is go super deep into it. it's like okay. there's a whole folder follow files. she'd ever look at them your entire interaction with it is through a prompt. i literally discussed me like, i literally think it's it's inhumane, and it's my mission remains the same as it has been for, like twenty years, which is to stop people working like this.

Speaker 1: i know, but but so costing my mind back, i used to work of data scientists. they were using jokes to notebook ks. what i found found. i typically, i mean, then then you couldn't if you check them into get, it wouldn't look very good. most of these daof scientists didn't know how to use it. they would run the cells out of border, which means it wouldn't be reproduced. but it what t's the things like that. the thing is, i agree with you that you can use them in this in this workflow, but it comes back to what i was same before about. you know, we 're talking about the core censor, and it being like a low intelligence job. you know, the data scientists out the reasway that they they are doing intelligent work is they are actually creating something that doesn't exist. they are figuring out the the contcontls of of problem. they're actually working in a domain that is poorly understood. but you could argue now the bull case is when the data scientists can sucsinically describe the controls of the problem. maybe we could go to cloud code, and we could implement the properly that how do bridge dge between those two world be be driterririidea ah,

Speaker 0: like, you know what to remove people from their explororatory environment. you know, research and science is developed by people building insight. you know, whoever you listen to, we know whether be a anmman or whatever like this is if you always here from the great scientists, how they build deeper intuition by by building mental models, which they get overtime by interacting with the things that they're learning about that, like in fineman's case, because it's theoretical physics, he couldn't actually pick up speeding clock, but he did literally study speding plates. you know, you gota a find ways to deeply interact with with what you're working with. like seven times, i've seen data science teams because you right, data science teams. i'm very familiar with kit, and i'm very familiar with things that they don't need to understand. um so often, i've seen a software engineer will become their manager, and their fixed to this will be id to tell them all to stop using jupiter notebooks. and now they have to use all these reproducable blaablavivirtubut. you know, virtual end of blablah. they destroy these teams over over again. i've seen this keep happening because the solution is not create more discipline and bureaucracy. it's solve the actual problem. so for example, we we built a the thing called an ND merge driver, which a lot of people that realize this. but actually, uh, no books extremely get friendly. it's just that get doesn't ship with a merge driver for them. so get only ships with a merge driver for a line based text files, but it's fully plugble. and so you can easily plug in in one for jason files instead. and so we wrote one. and so you can diff, you know, when you get to get diff withour merge driver, you see sell level difts. if you get a merge conflict, you get surge level lelevel merge conflicts that the notebook is always openable in jupiter. um andy dime did the same things of to independent implementations of this. so yeah, there were problems to solve, you know, but the solution to it was not throw away brebrevicicor's ideas and make people further away from from their exploratory tools, but to fix the expliratory tools. and i think all software developers should be using explloitatory based programming to deep en, their understanding of what they're working with so that they end up with a really strong mental model of the system that they're building. and that working with and then they can come up with better solutions more incrementally than are tested. i basically ever have to use a debegga because i basically never have bugs, and it's not because i'm a particularly good programmr. it's because i build up small little steps and each step works, and i can see it working. and i can interact with it. so there's no room for bags,

Speaker 1: you know, you know, i'm so torn on this, because i agree with you, and i'm also scheduical of people who say that organizations that they they converge on to ways of doing things, and they no longer need to evolve. they no longer and need to adapt. you know, innovation is adaptivity, and we should should rease the surface area of adaptivity as much as we possibly know cancer. we need people that are constantly testing new ideas, finding these constraints. but by the same token, we need to use the cloud, we need to use CRCD. we need to get this stuff into production.

Speaker 0: yeah, so do theit's absolutely like there's absolutely know like it. so BDF chips with out of the box CI integration and the the tests are literally there like, because the sources and notebook the entire exploration of like hatsses API work, you know, what does it look like when you call it the implementation of the functions, the examples of the documentation, often the test of them are in one place. so it's much easier to be a good software engineer in this environment. um so yet like to do do both, you know.

Speaker 1: so remember, there was there was that existential risk should be an urgent priority, and it was signed by fix like hinton and and demiscs sobis. and you responded um basically with a rebuttle. and that was with aravan. you know, the snake oal guy tell me about that. you do think we should be worried about AI exexistentirisst.

Speaker 0: i mean, that was a certain time wasn't. and i feel like things have changed a bit god. i feel like we we not spend an avvan, but broadly aaking community of which you're part kind of probably won that. now we have other problems to worry about. but you know, basically, at that point, the prevailing narrative was AI is about to become autonomous st, could become autonomouous at the moment, and great destroy the world. so very much comes from. you know, eliza you kcaskiis work, which i think clearly, it's been shown to be wrong that many levels this point that they would review that,

Speaker 1: obviously,

Speaker 0: well, they would. yeah, it's one of those things that they can always refuse, just like any doing stay cult unless less you give it a date and the date passes.

Speaker 1: so even i update a little little bit in the sense that i now think i would now say that these models can be said to be intelligent and restraicted demains the archchallenshowed that. so if you place constraints into the problem, you can, you can go faster towards an knogo even agency. you, you can put a planner on that. and you can go if you, if you know where you're go, you can go there faster, but that doesn't help you that you can have all the intelligence and agency in the world, but you don't have the knowledge in the constraints, then you're going in the wrong direction, faster. and i think they don't seem to appreciate that these models don't actually know the world like that of that,

Speaker 0: that even relevant to, even in my point, which was it is that it's misunderstanding where the actual danger is, which is that when you have a dramatically more powerful technology entering the world that can make some draramatically more powful people, people who in love with power, we'll seek to monopolized technology. and the more powerful it is the more strong that urge from those power. hungry people will be so to ignore people who say here's the problem. if you're like, i don't care about a new of that. all i care about is autonomous AI taking off you singsinglarity paper clip, let no good. whatever the obvious solution to that is, oh, let's set tualize power wer, which which is what we kept saying, particularly at time lelegive, um either very rich technology companies or the government, or both all of this power and make sure nobody else has it in in ththreat model. that's the worst possible thing you can do because you've centralized the ability to control in one place. and therefore, these people who are desperate for power just have to take over that thing.

Speaker 1: could we didistinguish though what you mean by power because we we just spend some of this conversation talking about how it's not actually as powerful as people think it is,

Speaker 0: but i'm not even but that man isn't even if thing right. so II just say, even if it turns out to be incredibly powerful, right, like i don't, i don't even want to argue about whether it's going to be powerful, cause that's speculative. even if it's going to be incredibly powerful, you still shouldn't centralize all of that power in the hands of one company or the government. because if you do all of that power is going to be monopolized by power hungry people and used to destroy civilization. basically, you'll i'll end up with the place where all of that wealthy power will be centralized with the kinds of people who we wanted centralized. so like society for hundreds of years have faced this again and again and again, pitknow. so when it's like the writing used to be something that only the most exclusive people had access knowing about writing, and the same arguments were made. if you let everybody write, they're gona use it to write things that we don't want them to write. and it's going to be really bad. you know, did know with printing, did i with the vote like the and again, it again, society has to fight against this natural predalition of the people that have the statscore power to be like. no, this is a threat. so when we saying like, okay, what if AI turned to be incredibly powerful, would it be better of a society to be that to be kept in the hands of a few or spread out across society? my argument was the letter. no, there's also an argument which is like i don't worry about it. thenot going to be that powerful. anyway, i just didn't wanna go there because it's not. and i commate that easy to win, because you can't really say what's going to happen. we're all just guessing, but i can cleclearly say, what if it happens, would it? it be a really good idea to only let elline nust kevt would it going to be a good idea to only let andred tramp hait it dhendrok spoke about this offence defence a symmetrium.

Speaker 1: so it's actually very important for us to have to have counterviling in a defences. but let's just take that as a given from minute because obvisly, when we look at something like meter and facebook it, it's quite clear what the power and balances, you know, there's they they control all of our data, they they know what 're doing. so it's something like open they are in claued. so it's it's not as good as we thought it was because actually humans only to be involved, for example, they have all of our data right? and you might be working on some new innovative technology and you're using clad and you're sending all of your information up there and they can now copy you. i mean, what what ms of risks are you talking about to be more concrete?

Speaker 0: yeah now. i mean, so i was not talking about any of those things that's at the time. i was talking about this speculative question of what if i get incredibly powerbut.

Speaker 1: i mean, i now, for example, they they say that this is the new means of production. and and that seems completely hyperbullets to me. but like in your best estimation, now, if there are risks,

Speaker 0: what are that if there are risks with a current state of technology, i think i think some of them of the ones we've discussed, which is people and febling themselves by basically losing their ability to be to become more competent over time. that's the big risk. i worry about the most the privacy risk, it's there. but i'm not sure it's much more there than it was for google and microsoft. before you know, you used to work at microsoft, you know how much data they have about the average outlook offers six etera user did of a google, you know, the average google worspace or gmail user. um those privacy issues are real. although i think there are bigger privacy issues around these companies, which the government can outsource state election tools. so back in the day is to be companies like choice point in axim. nowadays, it's probably more companies like pentia. the US government is um actually prohibited from building large databases about US citizens, for example, but it's not prohibited companies are not prohibited from doing so in the governis, not prohibited from contracting things to those companies. so i mean, that's a huge worry, but i don't think it's one that II is uniquicly creating. it's tatainly like you're in the UK. as you know, in the UK surveillance has been universal for quite a while. now it certainly makes it easier to use that surveillance, but a sufficiently well resource organization could just through one thousand bodies at the problem. so yeah, not sure these are due privacy problems as maybe more common ones than i used to be jeremy.

Speaker 1: i have just noticed the time i need to get to the airport. Alright, this has been amazing.

Speaker 0: thank you. so thank you for coming. yeah. hope you had a nice trip. thank you so much.