Max Welling: I want to think of it as what I would call a sort of a physics processing unit like a PPU, right? Which is you have digital processing units and then you have physics processing units.

Host: So it's basically nature doing computations for you. It's the fastest computer known possible even.

Max Welling: Uh it's a bit hard to program because you have to do all these experiments also quite quite bulky. It's like a very large thing you have to do. But in a way it is a computation and that's the way I want to see it. You can do computations in a data center and then you can ask nature to do some computations, right? Your interface with nature is a bit more complicated, but then these things will have to seamlessly work together to get to a, you know, a new material that you're interested in.

Host: Yeah, it's a pleasure to have Max Welling as a guest today. Max has done so much over his career that I've been so excited about. If you're in the deep learning community, you probably know Max for his work on variational autocoders, which has literally stood the test of time or officially stood the test of time. If you are a scientist, you probably know him for his pioneer work on graph neural networks on equivarians. And if you're a material science, you probably know about his new startup, CuspAI. Max has a long history doing lots of cool problems. You started in quantum gravity which is I think very different than all of these other things you worked on. The first question for AI engineers and for scientists what is the thread in how you think about problems? What is the thread in the type of things which excite you and how do you decide what is the next big thing you want to work on?

Max Welling: So it has actually evolved a lot. Um in my young days let's prove I would just follow what I would find like super interesting. I have kind of this sensor I think many people have but maybe not really uh sort of use very much which is like you get this feeling about getting about very excited about some problem right like it could be you know what's inside of a black hole or what's you know at the boundary of the universe or you know what what is quantum mechanics actually all about and so I followed that basically throughout my career but I have to say that as you get older this changes a little bit in a sense that there's a new dimension coming to it and this is impact. Working in two-dimensional quantum gravity, you pretty much guaranteed there's going to be no impact on what you do relative, you know, maybe a few papers, but not in the in this world at this energy scale. As I get closer to retirement, which is fortunately still, you know, 10 years away or so, I do want to kind of make a positive impact in the world. And um I got pretty worried about climate change. Um I think we and um I think we should you know and politics seems to have a hard time solving it especially these days and so I thought better work on it from the technology side and that's why we started cosplay but there's also a lot of really interesting science problems in you know material science um and so it's kind of combining both the impact you can make with it as well as the interesting science so it's sort of these two dimensions like working on things which you feel is like, wow, there's something very deep going on here and on the other hand trying to build tools that can actually make a real impact in the world.

Host: So the the thread that when I look back look at the different things you worked some of them seem preconnected like the physics to to equivariance and yeah and uh graph neural networks maybe and that that seems to be somewhat related to cusp. Do you have a a thread through there?

Max Welling: Yeah, I think physics is the thread. So, having done, you know, spent a lot of time in theoretical physics, I think there's first very fundamental and exciting questions like things that haven't actually been figured out in quantum gravity. So, there's really the frontier, but there's also a lot of mathematical tools that you can use, right? In for instance, in particle physics, but also in general relativity, sort of symmetry space play an enormously important role. And this goes all the way to to gauge symmetries as well. And so applying these kinds of symmetries to uh machine learning was actually you know I thought of it as a very deep and interesting mathematical problem. I did this with Taco Cohen and you know Tacoan was the main driver behind this went all the way from just simple like rotational symmetries all the way to gauge symmetries on spheres and stuff like that. So and Maurice Wiler who's also here um when he was a PhD student with me you know he wrote an entire book which I can really recommend about the role of symmetries in in AI and machine learning that I find is a very deep and interesting problem. So more recently so I've taken a sort of different path which is the relationship between diffusion models and and a field called stoastic thermodynamics. This is basically the thermodynamics which is a theory of equilibrium. So but then formulated for out of equilibrium systems and it turns out that the mathematics that we use for diffusion models but even for reinforcement learning for shinger bridges for MCMC sampling has the same mathematics as this theor this physical theory of non-equilibrium systems and that got me very excited and actually uh when I taught a course in Ma it is South Africa close to Cape Town at the African Institute for Mathematical Sciences as I turned that into a book. So two years later the book is finished. I've sent it to the publisher and this is about the deep relationship between free energy diffusion models basically generative AI and stoastic thermodynamics. So it's always some kind of I don't know I find physics very deep. I also think a lot about quantum mechanics and it's it's it's a completely weird theory that actually nobody really understands and there's a very interesting story which is maybe good to tell to connect sort of my my PhD back to where I am now. So um I did my PhD with a noble laureate to he brilliant man I've ever met. He was never wrong about anything as long as I've seen him and now he says quantum mechanics is wrong and he has a new theory of quantum mechanics. Nobody understands what he's saying, even though what he's writing down is not mathematically very complex. But he's trying to address this understandability, let's say, of quantum mechanics head on. I find it very courageous and I'm completely fascinated by it. So I'm also trying to think about, okay, can I actually understand quantum mechanics in a more mundane way sort of, you know, without all the weird multiveres and collapses and stuff like that. So the physics is always been the threat and I'm trying to apply the physics to the machine learning uh to to build better algorithms.

Host: You are still very involved in understanding and understanding physics and the world.

Max Welling: Yes. So I would say I'm I'm not I'm not contributing much to physics but I'm contributing to the interface between physics and science and that's called AI for science or science or AI is kind of a super it's actually a new discipline that's emerging. Um and it's not just emerging it's exploding I would say that's the better term because I know you go from investments into like in the hundreds of millions now in the billions. So there's now actually a startup by Jeff Bezos that you know that 6.2 two billion C round right it's like insane like this is the largest you know startup ever I think right and that's in this field AI for science like it tells you something that we are creating a new bubble here right

Host: so so why do you think it is what has changed that has like motivated people to start working on AI for science type problems

Max Welling: so there's two two reasons actually the one one is that people have been applying sort of the tool the new tools from AI to the sciences which is quite natural and there's of course I think there's two big examples you know protein folding is or is a big one and the other one is machine learning force fields or sometimes called machine learning intra atomic potentials both of these have been actually very successful both also had something to do with symmetries which is also cool and sort of people in the AI sciences saw an opportunity to apply the tools that they had developed beyond advertise placement right or you know multimedia applications um into something that could actually make a very positive impact in society like health, drug development, materials for the energy transition, carbon capture, these are all really cool, you know, impactful applications. Beside that the science and the kind of the is also very interesting and sort of the I would say the um the fact that these sort of these two fields are coming together and that we're now at the point that we can actually model these things effectively and move the needle on some of these uh sort of science uh sort of methodologies is also a very unique moment I would say and people recognize that okay now some we're at the cusp of something new which is also whether That's where also what the company is called after. We we're at the cusp of something new and of course that always creates a lot of energy. It's like, okay, there's something. It's like sort of virgin field, right? It's like nobody's green field. Nobody's been there, you know? Um, I can rush in and I can sort of start harvesting there, right? And uh and I think that's also what's causing a lot of sort of enthusiasm in the fields.

Host: If you're an AI engineer... and you maybe don't have a strong science background, but are excited... how does somebody who is not a scientist on a day-to-day basis how do they get involved?

Max Welling: Well, they can read my book once it's out. But um this is basically saying that there is more. We should create curricula that are on this interface. So I'm not sure there is possibly already some universities actual courses you can take, maybe online courses you can take. These workshops where we are now are actually very good as well. And we should probably have more tutorials before the workshop starts. Actually we've I've kind of proposed this at some point. and it's like maybe first have an hour of a tutorial so that people can get new into the field. But yeah, there's a lot out there. Um most of it is of course inaccessible but I would say um we will create much more books and other content that is more accessible including this podcast I would say right so um I think you know it will come and you know these days you can watch videos and things there's a huge amount of content you can you can go and uh and see

Host: so maybe a followup to that... why should they get involved... what opportunities does AI for science provide them to make an impact to you know change the world that working in this the world of pure bits would not.

Max Welling: So so my view is that um underlying almost everything is a material. So we we're focusing a lot on LLMs now. Yeah. Which is kind of the software layer. But I would say if you think very hard underlying everything is a material. So underlying an LLM is a GPU and underlying a GPU is a wafer on which we would we will have to deposit materials... Underlying everything is a material. So I was saying you know there's the LLM underlying the element is a GPU on which it runs and then in order to make that GPU uh you have to put materials down on a wafer and sort of shine on it with sort of EUV light in order to etch kind of the structures in but that's now an actual material problem because more or less we've reached the limits of you know scaling things down and now we are trying to improve further by new materials. So that's a fundamental materials problem. We need to get through the energy transition fast if we don't want to kind of mess up this world. And so there is for instance batteries. That's a complete materials problem, right? There's fuel cells. Um there is solar panels. So that they can now make solar panels with new perovsky layers on top of the silicon layers that can capture, you know, theoretically up to 50% of the light where now we're at I don't know maybe 22 or something, right? So, so these are huge changes all by material innovation and um yeah, I think wherever you go, you know, I can probably dig deep enough and then tell you well actually the very foundation of what you're doing is a material problem. And so I think it's just you very nice to work on this very very foundation and ex also because I think this is maybe also something that's happening now is we we can start to search to this material space. This is this this has never been the case, right? It's like scientists were the the normal way of working is you read papers and then you come up with an hypothesis you do an experiment and you learn etc. So there's a very slow process. Now we can treat this as a as a search engine like we search the internet we now search the space of all possible molecules not just the ones that people have made or they're in the universe but all of them right and and and we can make this kind of fully automated. That's the hope right? You can just type it becomes a tool where you type what you want and something starts spinning and some experiments get going right and then you know outcome a list of materials and then you look at it say maybe not and you refine your query a little bit and you kind of do research with this search engine where a huge amount of computation is and and experimentation is happening you know somewhere far away in some lab or some data center or something like this. I find this a very very promising view of how we can sort of come you know build a much better sort of materials layer underneath almost everything and also a more sustainable materials or plastics are polluting the planet if you can come up with a plastic that kind of destroys itself you know after I don't know a few weeks right and actually becomes a fertilizer these are these are things that are not impossible at all these things can be done right and we should do it

Host: can you tell us what a little bit just generally about custoi and And we I have ton of questions.

Max Welling: Yeah. So, Kaspi started about 20 months ago and it was because um I was worried about I'm still worried about climate change and so I realized that in order to get you know to to stay within two degrees let's say we would not only have to reduce our emissions to zero by 2050 but then you know another half century or even a century of removing carbon dioxide from the atmosphere not by reducing your emissions but actually removing it at a rate that's about half the rate that we now emit it and that is a unsolved problem but and if we don't solve it 2° is not going to happen right it's going to be much more and I don't think people quite understand how bad that can be like four degrees like very bad so um so this technology needs to be developed and so this was uh my and my co-founder Chad Edwards um motivation to start this startup and also because you know we saw the technology was ready which is also very good so we're you know the time is right to do it and uh yeah so we now in in the meanwhile we've grown to about 40 people we've kind of collected 130 million investment uh in the into the company which is for a European company is quite a lot I would say it's interesting that right after that you know other startups got even more so that's kind of tells you how fast this this this is growing but yeah we are we are now at the So we've built the platform of course but uh it's it's for a series of material classes and it needs to be constantly expanded to new material classes and it can be more automated because you know we know putting LLMs in and so the whole things gets more more and more automated and now we're moving to sort of high throughput experimentation so connecting the actual platform which is computational to the experiments so that you can get also get fast feedback from experiments and I kind of think of experiments as something you do at the end. Although that's what we've been doing so far. I want to think of it as what I would call a sort of a physics processing unit like a ppu, right? Which is you have digital processing units and then you have physics processing units. So it's basically nature doing computations for you. It's the fastest computer known possible even. Uh it's a bit hard to program because you have to do all these experiments. It's also quite quite bulky. It's like a very large thing you have to do. But in a way it is a computation and that's the way I want to see it. So I want to you can do computations in a data center and then you can ask nature to do some computations right your interface with nature is a bit more complicated but then these things will have to seamlessly work together to get to a you know a new material that you're interested in. And that's that's the vision we have. You don't say super intelligence because I don't quite know what it means and I don't want to oversell it but I do want to automate this process and give a very powerful tool in the hands of the chemist and the material scientists.

Host: can you talk about your platform... explain kind of how it works and like what you your thought processes was in developing it.

Max Welling: Yeah, actually it's been surprisingly it's not rocket science. Yeah, I would say it's not rocket science in the sense of the design and basically the design that you know I wrote down at the very beginning. It's still more or less the design. Although you add you add things like I I wasn't thinking very much about multiscale models and come on our radar that actually multiscale is very important. In the beginning I wasn't thinking very much about self-driving labs but now I think you know we that's we are now at the stage we should be adding that. So there is sort of bits and details that we're adding but more or less it's what you see in the slide decks here as well which is there's a generative component that you have to train to generate candidates and then there is a digital twin multiscale multifidelity digital twin uh which you walk through the steps of the ladder you know that they do the cheap things first you weed out everything that's obviously unusful and then you go to more and more expensive things later um and so you narrow things down to a small number those go into an experiment, you know, do the experiment, get feedback, etc. Now, things that also have been more recently added is sort of more agentic uh sort of parts. You know, we have agents that search the literature and come up with, you know, actually the chemical literature and come up with, you know, chemical suggestions or for doing experiments. We have agents which sort of autonomously orchestrate all of the computations and the experiments that need to be done. you know they're in various stages of maturity and they can be continuously improved I would say and so that's basically I don't I think that part is rocket science you know the design of that thing is not like surprising but is is surprising hard to actually build it right so that's that's the thing that is where the mo the moat is in the data that you can get your hands on and the and actually building the platform and and I would say there's two people in particular I want to call out which is Felix Hunker who is actually you know building the scientific part of the platform and Alessandro de Maria who is building the sort of the SCA the kind of this the MLOps part of the platform. Yeah. And so and and recently uh we also added um sort of Aaron Walsh to our team who is a a very accomplished scientist from Imperial College. We're very happy about that. he's going to be our chief science officer and uh we also have a partnerships team that sort of seeks out all the customers because I think this is one thing I find very important in princ it's so complex to do to actually bring a material to the real world that you must do this you know in collaboration with uh sort of the domain experts which are the companies typically so we always we only start to invest in a direction if we find a good uh industrial partner to go on that journey with us

Host: over the evolution of the platform. Did you find... that human intervention... did you start out with having human feedback lots of steps and then like kind of figure out ways to remove you know

Max Welling: that that's is the second one. So you build tools. So you so it's much more modular than you think but it's like we need these tools for this application. We need these tools. So you build all these tools and then you go through a workflow actually in the beginning just manually. So you put them at first this tool then run this tool then run this. So so you put them in in in a workflow and then you figure out oh actually you know this this pores material that we we're trying to make actually collapses if you shake it a bit. Okay. Then you add a new tool that says test for you know stability. Right. And so there's more and more tools and then you built the agent which could be a Beijing optimizer or it could be an actual LLM you know maybe trained to be a good chemist that will then start to use all these tools in the right way and the right order. Yeah. Right. But in the beginning it's like you as a chemist are putting the workflow together and then you think about okay how am I going to automate this? Right. One very easy question you can ask yourself is you know every time somebody who is not a super expert in DFT. Yeah. And he wants to do a calculation has to go to somebody who knows DFT. And so could you start to automate that away which is like okay make it so user friendly so that you actually do the right D of for the right problem and for the right length of time and you can actually assess whether it's a good outcome etc. So you start to automate smaller small pieces and more bigger pieces etc. And in the end the whole thing is automated. So so your philosophy is you want to provide a set of specific tools that make it so that the scientists making decisions are better informed and uh less so trying to create a an automated process. I think it's this is sort of the same what you're saying because yes we want to automate. Yeah. But we don't see something very soon where the chemists and the domain expert is out of the loop. Yeah, but but it's a retreat, right? It's like, okay, so first you need an expert to tell you precisely how to set the parameters of the DFT calculation. Okay, maybe we can take that out. We can maybe automate it, right? And so increasingly more of these things are going to be removed. Yeah. Um in the end the vision is it will be a search engine where you where somebody a chemist will type things and we'll get list candidates but the chemist will still decide what is a good material and what is not a good material out of that list right and so the vision of a completely dark lab where you can close the door and you and you just say just you know find something interesting and then it will it will just figure out what's interesting and we'll figure out you know it's like oh I found this new material to blah blah blah blah right that's not the vision I have at least not not for you know a long time so for me it's really empowering the domain experts that are sitting in the companies and in the universities to be much faster in in in developing their materials and I should say it's also good to be a little humble at times because it is very complicated you know to bring to make it and to bring it into the real world and there are people that are doing this for their entire lives. Yeah. Right. And it's like I wonder if they scratch their head and say, well, you know, how are how are you going to completely automate that away like in in in the next 5 years? I don't think that's going to happen at all. Um, yeah. So, so to me, it's a increasingly powerful tool in the hands of the chemists.

Host: I have a question. You've talked before about getting people interested based on having you know sort of a big breakthrough in materials just incremental change. I'm curious... how are you chasing the big change...

Max Welling: We follow a mixed strategy. So we are definitely going after a big material. Again, we do this with a partner. I'm not going to disclose precisely what it is, but we have our own kind of long-term goal. You could call a lighthouse or, you know, um sort of moonshot or whatever, but um it is going to be a, you know, really impactful material that we want to develop as a proof point that it can be done and that that it will make it into the into the real world and that AI was essential in actually making it happen. At the same time, we also are quite happy to work with companies that have more modest goals. Like I would say one is a very deep partnership where you go on a journey with a company and that's a long-term commitment together and the other one is like somebody says I need I need a force field. Can you help me train this force field and then maybe analyze this particular problem for me? Um and I'll pay you a bunch of money for for that and then maybe after that we will see. And that's fine too, right? But we prefer, you know, the deep partnerships where we can really change something for the good. Yeah. And do you feel like from a platform standpoint, you're ready for that or what are the things that and and again not asking you to disclose proprietary secret sauce, but what are the things generally speaking that need to happen from where we are to where to get those big breakthroughs like that? What I find interesting about this field is that every time you build something, it's actually immediately useful, right? And so unlike quantum computing which or nuclear fusion, so you work for I don't know 20 30 40 years and nothing nothing nothing and then it has to happen. Yeah. Right. And when it happens it's huge. So it's it's quite different here because every time you introduce so you go to a customer and you say so what do you need? Right. Okay. So we work uh let's say on a on a problem like water filtration we want to remove PAS from water right so we do this with a company chimera so they are deep partner for us right so we on a journey together I think that the breakthrough will happen with a lot of human in the loop because there is the chemists who have a whole lot more knowledge of their field and it's us who who will you know help them with AI training and new methods and in that kind these interface these interactions something beautiful will happen and that that will have to happen first before this field will really take off I think and so in the sense that it's not a bubble let's put it that way so there people see that's actual real what's happening so in the beginning it will be very you know with a lot of humans in the loop I would say and I would I would hope we will have this new sort of breakthrough material before you know everything is completely automated because that will take a while and also it is very vertical specific. So it's like completely automating something for problem A. You know you can probably achieve it but then you'll sort of have to start over again for problem B because you know your experimental setup looks very different. You the machines that you characterize your materials look very different. Even the models in your platform will have to be retrained and fine-tuned to the new class. So every time you know you have a lot of learnings to transfer but also you know the problems are actually different and so yes I I would want that breakthrough material before it's completely automated which I think is kind of a long-term vision and I would say every time you move to something new you'll have to start retraining and humans will have to come in again and sort of okay so what does this problem look like and now sort of you know point the the machine again you know in the new direction and and then and then use it

Host: And for the non-scientists among us... Can you sort of explain... what is equivariance?

Max Welling: So equivariance is the infusion of symmetry in neural networks. So if I build a neural network, let's say that needs to recognize this bottle, right? And then he rotate the bottle, it will then actually have to completely start again because it has no idea that the rotated bottle. Well, actually the input that represents a rotated bottle is actually a rotated bottle. It just doesn't understand that. Where if you build a variance in basically once you've trained it in one orientation, it will understand it in any other orientation. So that means you need a lot less data to train these models and these are constraints on the weights of the model. So the so basically you have to constrain the weight such that it understand it and you can build it in you can hardcode it in and yeah this the symmetry groups can be you know translations rotations but also permutations like in graph neural network there are permutations and in physics of course there's many more of these groups

Host: to play devil's advocate why not just use data augmentation by your model is in all the different orientations

Max Welling: as an option it's just not exact why would you go through the work of doing all that where you would really need an infinite number of augmentations to get it completely right um where you can also hardcode it in. Now I have to say sometimes actually data augmentation works even better than hard coding the equarians in and this is something to do with the fact that if you constrain the optimization the weights before the optimization starts the optimization surface or objective becomes more complicated and so it's harder to find good minima. So there is also a complicated interplay I think between the optimization process and and these constraints you put in your network and so yeah you you'll hear kind of contradicting claims in this field like some people and for certain applications it's it works just better than not doing it and sometimes you hear other people if you have a lot of data and you can do data augmentation then actually it's easier to optimize them and it actually works better than putting the aggregants in. Do you think there's kind of a bitter lesson for um mathematically founded uh models and strategies for doing deep learning? Yeah, ultimately it's a trade-off between data and uh and inductive bias. Yes. So if your if your inductive bias is not perfectly correct, you have to be careful because you you you put a ceiling to what you can do. But if you know, you know, the symmetry is there, it's hard to imagine there there isn't a way to actually leverage it. But yeah, so there is a bitter lesson. One of the bitter lessons is you should always make sure you architect your scale unless you have a tiny data set in which case it doesn't matter. Um, but if you you know the same bitter lessons or lessons that you can draw in LLM space are eventually going to be true in this space as well I think. Yeah.

Host: Can you talk a little bit about your upcoming book and tell the listeners like what's exciting about it?

Max Welling: Yeah, they should read it. So this book is about so it's called uh generative AI and stoastic thermodynamics. It basically lays bare the fact that the mathematics that goes into both generative AI which is the technology to generate images and videos and this field of non-equilibrium statistical mechanics which is systems of molecules that are just you know moving around and you know relaxing to their ground state or that you can control to have certain you know be in a certain state. The mathematics of these two is actually identical. And so that's fascinating. And in fact, what's interesting is that Jeff Hinton and Radford Neil already wrote down the variational free energy for uh machine learning long time ago. And there's also Carl Friston's work on free energy principle and active inference. But now we've related it to this very new field in physics which is called stoastic thermodynamics or non-equilibrium thermodynamics which has its own very interesting theorems like fluctuation theorems which are which we don't typically talk about but we can learn a lot from and I think it's just it can sort of now start to cross fertilize when we see that these things are actually the same we can like we did for symmetries we can now look at this new theory that's out there developed by these very smart physicists and say okay what can we take from here that will make our algorithms better. At the same time, we can use our models to now help the scientists, you know, do better science, right? And so it becomes a beautiful cross fertilization between these two fields. You know, the book is rather technical, I would say, and it takes all sorts of things that have been done in stoastic thermodynamics and all sorts of models that have been done in in the machine learning literature and it basically equates them to each other. And I think hopefully that sense of unification will be revealing to people.

Host: Wait, and when is it out?

Max Welling: Well, it depends on the publisher now, but uh I I hope in April um I'm going to give a keynote at Icleair and I would be very nice if I have this book in my hand, but you know, it's hard to control these kind of timelines.

Host: Yeah, I'm looking forward to it. Great. Likewise. Thank you very much.