It's It's all right. Um, all right. Um, hope you guys have a great morning so far. Um, so I'm honored today to have uh a wonderful panel with me today. Um, in fact, all of them are my friends to some extent like for the last like many years. Uh I've known each of them for quite a few years so far. Uh so it would be great to have this opportunity to also reunion with all of my friends here.

So so we're going to focus on RL today and we all know that reinforcement learning serve is core to today's AI uh advancements. Um from earlier this year of deepse release of R1 where they have reinforcement fine tuning to really have um some sort of self-improvement on coding and math. uh to later this year where we see a lot of applications on math, coding, chemistry, physics, uh even embody AI uh type of applications on reinforce learning um that enable like really strong intelligence uh and we've seen a lot of RL companies that raise a lot of money as well. So we're going to dive into all of that today. But before that uh I'd love to have the panel introduce them themselves. Uh may may start from Lihon. Yeah.

Hi, sure. Yeah. Hi everyone. Uh my name is Lehon from Amazon. uh I uh I work on reinforcement learning and and language models for uh for a while and then I look I it's a pleasure to join the panel and thanks the organizer for inviting me here.

Jen, hi my name is Zwan. So I'm currently working at open AI and before that I was at deep mind. So I also work on reinforcement learning and actually I'm a big fan of Lehon's paper and uh also working on uh large language models and AGI in general. I'm very happy to join the panel today.

Uh good morning everyone. I'm Alsar. I'm a distinguished scientist at LinkedIn. Um mostly focusing on agental. Before coming to LinkedIn, I was at Meta where I was actually working with Bill closely and uh before that I was in Amazon Alexa.

That's great. Um, cool. So, so let's dive right in. Um I I think people if you guys know the recent advancements in reinforce learning uh one thing that has been like triggering a lot of interest so far is how how can we train RL agents uh in language models uh without actual u suice data? um that is that there's no human labeling there's no you know human uh expert annotated data and how do you train these models and one of the trends is RVR uh reinforcement from verifiable rewards um the idea is that you can literally use rules um and sort of like you know verifier systems from math to really verify whether or not the outputs of certain language models are correct.

So so the first question is um there's a lot of papers coming out recently on RVR and then this is sort of the almost like a consensus on on how how these things are going. Uh so where do you guys think like RVR can go before beyond like just current reinforcement fine tuning on math and coding? Yeah, I think um it's um it it definitely has a lot of uh uh great impact um in fars uh but on the other hand we also see that the many problems are not verifiable they don't have verifiable outcomes so I think that um there at some point there is a need to go beyond that to handle nonverifiable uh problems.

Yeah, I think um it's um it it definitely has a lot of uh uh great impact um in fars uh but on the other hand we also see that the many problems are not verifiable they don't have verifiable outcomes so I think that um there at some point there is a need to go beyond that to handle nonverifiable uh problems an example is that uh in when we are children we are learning in school we have tests that are verifiable that have verifiable results right ABCD choose one of them Uh but then when we actually go out to work and it's hard to say that you have to do this in order to to check the mark right to have there's a lot of high judgment call and nonverifiable outcomes that we have to explore and and so that's a much fuzzier place. So I think that's where uh we need to do more beyond verifiable rewards.

Yeah, I think for reinforcement learning is verified reward. My understanding is that this approach can be used in a very general paradigm which is you generate first and then verify and the reason is that usually it's much easier to verify than generate. I mean, if we think about whe or not it can go beyond the uh fine-tuning I think it can and actually there are already some work on site for example you can use this approach to develop stronger reasoning skills and you can also use it in somewhat different framework compared to uh fine-tuning like I mean uh reinforcement learning based on debate or reinforcement learning uh based on self-critic I think all of these go beyond fine-tuning.

I agree with all the points raised. I think u reinforcement learning kind of give you the capability to us go beyond of what a tutor or a mentor basically shows you and says hey this is what I want you to maximize and I think we are there on the cross puff how do we define this goodness or the cookie for the agent and verifiable rewards are definitely like a way to go because it's very crystal clear how to define them but many of the task I think requires a combination of both there are things that we know that has to happen if you're compiling code we whether the code should be built successfully but there are things that we actually don't know exactly how to define them like if you're writing a summary or if you are providing an innovative idea how innovative it is or how good is your summary and I think that's where we need to get inspiration from humans essentially to guide us to how to uh shape this reward function and there is a lot of litany of work of starting from like comparison and alignment DPO all of those algorithms all the way to now getting like richer richer using LLM to become I'm essentially a judge and fill the shoes of humans for those cases.

and there is a lot of litany of work of starting from like comparison and alignment DPO all of those algorithms all the way to now getting like richer richer using LLM to become I'm essentially a judge and fill the shoes of humans for those cases.

Yeah, I I think we're alluding to the next question basically where uh reward design itself might not just be completely verifiable. Uh it might be like designing a mixture of like verifiable reward with like human feedback and alignment type of rewards. But um one of the things that is puzzling a lot of researchers and also production people is that like what is sort of the golden standard here right because if you have verifiable reward basically you don't need data but then if you have sort of RHF type of uh information where you have human feedback as one of the judgment um then you actually would require a lot of data for human feedback and then for very complex tasks like you know trying to solve a level I like um sort of chemistry Olympics or physics Olympics type questions. If you guys don't know, like a single data annotation on a response uh from a language model takes about $20 uh from a PhD. So, so where do we cut like how much data do we need? Uh is there a way that we can handle this uh mixture of verifiable and non-verifiable reward without actually data? Maybe this time we can start from Elbur.

For very complex tasks like you know trying to solve a level I like um sort of chemistry Olympics or physics Olympics type questions. If you guys don't know, like a single data annotation on a response uh from a language model takes about $20 uh from a PhD. So, so where do we cut like how much data do we need? Uh is there a way that we can handle this uh mixture of verifiable and non-verifiable reward without actually data? Maybe this time we can start from Elbur.

I'm sure I I think we need both. Uh in reality, um we want to be cost uh aware. So humans judgments are expensive. as Bill mentioned as an example, $20 per sample and you need like gobs of data that would be like you know making your bank account break. Uh so we want to make sure that you are using it judiciously but we can't just also ignore that because for many of the tasks we actually don't know how to create this reward function. So it's more like a think of it as an active learning situation.

So you can create essentially a mechanism to gauge your LLM judges. So you can get like let's say four of us here decide about like a small set of data sets how we actually gauge those and measure those and if we all agree then this can be a rubric to then go and check LLM judges and see do they comply with our agreement and if they do then it can be a blessed now lowcost essentially nonverifiable reward that you can apply on the masses of data and then you can close the loop through like active learning if there are some areas we can occasionally go and see are you doing the right thing and if there is like some holes over there we have to go back it's like hey let's collect some data in this area make it better and then go back and apply it

Yeah, I fully agree. I think that usually pre-training is a type where you acquire the general knowledge. So you want your model to be expressive and roughly roughly understand the world and post the training is you want to adapt it so that can become a expert in some field. I also want to mention that I think the exploration uh is crucial in cases where the data acquisition is very costly because here you cannot really acquire a lot of data. So you need to want to uh doing exploration to make sure that you acquire informative data.

And another solution is obviously is to keep human in the loop and also that's why we do reinforcement learning from human feedback. I mean I think that if in this case exploration is very uh critical. The reason is that because it's so costly. So you want to identify the right expert to ask the right question. So this clearly request a very effective exploration algorithm.

Yeah, these are the great ideas and uh I guess I just wanted to add that there's another maybe another possibility of looking at the uh the stream and experience of the model right for where the model can rely on it own capabilities or external models capabilities to extract signals from this experience stream of experience and do learning right um and then at the end of the day I think there's something that's maybe too obvious to say it is that um the capability of the power of RL is limited by how good a simulator data is how good environment you use to train the system right so I think um depends on how uh uh all these are great ideas but I think at the end of the day we need to make trade-off on how to make the make sure the environment is of high quality

I might want to actually dive into it a little bit because um we mentioned a couple of really good ideas. One uh around exploration the other around like uh rewards that's like a mixture of different types of reward and sub problem like breakdowns. Um there's a bunch of ideas that surface. I want to like touch on exploration first uh and elaborate on that a little bit because uh I think everyone on this panel actually worked on exploration at some point in your career. Um this is one of the topics that um a lot of RO people are today not focusing on enough because you see uh the the PP algorithm, GRP algorithm, DPO algorithm uh they're fine-tuning an existing model that has already been trained in a collapsing modality.

And then exploration is really needed when you actually have like little information about the environment and trying to explore and understand like what's the most important part of the equation. So so the first the question for Jen is like um where do you think is the balance between pre-training and post- training when you actually really want to induce like strong exploration behavior?

I think we want to make sure that during the pre-training like when you're doing the pre-training you have a generally good model across a wide set of task. If you start seeing that you're getting better in some of these tasks but worse in some other tasks you're doing to too far. And I think that's the part that you have to delegate that to the post-training essentially uh stage. The main to from my perspective the main role of the pre-training is to have a model that has a good understanding of fundamentals of the basics and the key ingredients of how to solve these problems in a higher level and then have the post training you can fine-tune on a specific task you actually care about it and I think that's the breaking point to me it's like you can look at your evaluation across the board and see if you are start regressing on some of these model in these domains compared to the other ones.

Yeah, I fully agree. I think that usually pre-training is a type where you acquire the general knowledge. So you want your model to be expressive and roughly roughly understand the world and post the training is you want to adapt it so that can become a expert in some field. I also want to mention that I think the exploration uh is crucial in cases where the data acquisition is very costly because here you cannot really acquire a lot of data. So you need to want to uh doing exploration to make sure that you acquire informative data.

It's also important in cases that not every human or every radar can give you some useful feedback. So in that case you also need to explore to identify an expert from which you can get the useful and informative feedback. Um for exploration I think that the um today's exploration is very different from what we talk usually talk about exploration. uh and when it pre-training is done, I think the space of exploration is already very limited, right? I think the goal there is just to make sure the model has a good starting point and and to get a good trade-off between how good you train and how fast you can train a model in post training. Uh at least that's my understanding.

uh and then uh but I think for uh for real or more aggressive exploration it might be uh interesting to look at uh beyond the token level or something that has more uh high level and do exploration there and then combine that with token level exploration. I think that may allow the model to think out of the box or maybe more aggressively in the post- training process.

Yeah. Uh that's great. I think that alludes to both questions of like higher level abstraction and also hierarchal planning. Um so if you guys haven't heard Rich Sutton's latest talk about the oak architecture, it emphasizes a lot about an uh hierarchal um structure of options and then also sort of like lower level planning on top of that. Uh so the idea is that you can actually break down very complex tasks into like sub problems and then each sub problem can be solved by some level of planning inside a sort of world model and then outside you actually can sort of learn by experience.

So here I want to pose two questions. questions and then any any one of you guys can like answer any one of them u if you wish. Um so so one is on the abstraction and representation level how much should we go beyond token level if we want to really generate uh longer kind of um context intelligence uh and the second how much is it important to have an hierarchal structure u in reasoning uh on both questions yeah

Yeah, I I do think that it's uh uh very important that eventually we look at hierarchical reinforcement learning in AI the reason is currently I mean um most of the signal are just a token level and it's so low level and this actually cause a lot of problem I mean for example this cause a lot of problem of credit assignment and also data efficiency so I do think this is very crucial that eventually we go to a higher level go to a higher abstraction to look at something like subtasks plans modules and functions and do planning there.

I think that if we eventually want to look at a very uh long horizon, I do think this is very crucial and I also believe that if we do it correctly, it can significantly improves data efficiency and also better solves uh credit assignment problem.

Yeah. So I I also agree that u we we need something like that uh at high level and combined with the token level. uh there's there may be a distinction though in this in this uh in the question that uh when we talk about the inference time the execution at the end of the day when the model produce the output is still at the token level one by one at least the current in the current framework but I guess what I was trying to say to to express is that um this high level thinking or high level training may be needed in the post- training stage where we set up the reward model we set up the problem right

and that's how we give a signal to the model to learn in the high level how to jump from local minimum to another better better landscape of the policy space as opposed to doing this narrow or low code exploration at token level right but at the end of the day and then in the inference time I think it's still still probably we're still looking at the token by token level

So I'm going to I I actually like the direction that this is going but I want to take it even further so we as humans when we want to plan to go to Japan for example from here we don't think about the little muscle twitches that we have to make in order to make that happen which is like go through the door and then take a cab and then go to the you know take a take a plane we actually think on a higher level of abstraction we think about okay I need to get a cab from here then take the plane and then go to the hotel and then you start bringing those back into a lower level decision it's like okay if I want to take a cab how do I do it I call an Uber or a lift or something like that and I think this is what I'm thinking currently LLMs are missing.

Uh we are thinking in the token level still which is the muscle twitches of humans and I think as you start going into bigger and bigger problem you want to solve you need that level of abstraction in order to let you plan that far down the road and uh the one way of doing it which what Lee Leehung just mentioned is like when you want to do the backups you actually think about further away decision outcomes and then bring them back in rather than waiting for this to trickle through all the tokens. But one level higher than that is you actually have a policy actually thinks in a higher level because we think that way.

So we can have a policy to think is like what are the chain of highle things that I have to do in order to get to that outcome like I have to do like I have to call this tool and then get the results filtered by that thing and then do something else in order to get to the reasoning that I want. And I think that's the way currently there's not much work that I have seen out there, but I think that would be the direction to actually unlock the whole new era of reasoning available through these LLMs.

That's great. Um I want to add one question I didn't send you guys before. um uh it's exactly on this line where um so when we're talking about hierarchical RL um and then also especially when you're training LMS one of the things that people have tried a lot u in training LMS to is to add process based reward um and then if you want to do hierarchal RO to some extent you have to have process based reward like after you finish the whole you know like 10,000 token long sequence before you end there will be checkpoints in the middle like okay after I finish this task here's the reward next task another reward and so on so forth. Uh would you guys think like it is really hard for us to really sort of derive some sort of process reward um when we're training LMS in the next like few years?

Yeah, I I do think this is a very promising direction. I mean definitely there are some challenges but overall I'm optimistic about that.

Yeah, I think that's um interesting uh promising idea to try. I think at some point uh although right now I think uh outcome based reward seems to go a long way but uh I can imagine that in some complex problems we need some intermediate signals even for now I think when you call a long sequence of agent calls I think uh people some some recent work is finding that uh it's beneficial to have some intermediate signals right

and also another point want to make is that um this out process reward or intermediate reward. They don't have to to be final reward. It can be used as a uh a solution or a technique as part of a curriculum training. We use intermediate reward at the beginning to guide the solution to guide exploration and then eventually you take away the intermediate reward and only let the model learn from the final reward. So I think that's another possibility.

We actually recently looked at that problem very specifically. It's actually on archive if you want to read that paper. But it to some extent is it also depends on the model that you're starting with. Uh bigger model often have more power. So if you give them a problem to solve, they probably would find easier to a workable solution, not the perfect solution, but they can get to somewhere that I actually see this verifiable reward at the end. Because if your model cannot see that good cookie at the end at all, then you can't learn anything. But for the smaller models, they might not actually be able to do that.

So the thing that Lee Hung just mentioned, it is much more important for a smaller models to actually have access to a more enriched dense reward. So they can find their way out all the way to the end, a good end versus as you go through the bigger model, they might not need as much as massaging in the beginning for the reward model to make it work. But uh the interesting finding if you had is that one domain but a smaller models if you actually shape them well you can actually get about three to four times compute saving and get to the same outcome. So it's a balance of how how hard you can actually work on your work function to actually make it informative enough for the models to be able to tackle the problem. But you rip the benefits in GPU cost which now everyone has limited resources of.

Yeah. And that also brings like importance in exploration, right? If you have like very sparse reward at the end, um instead of just doing exploration, another way you could potentially do is probably adding more insightful intermediate rewards. All right. Um let's move on to a different topic. I I think we've talked enough about how to train RL agents on LMS, but I think we want to do a little bit more forward looking on where does RL really land us.

Um, so one of the things that OpenAI has brought up many years ago, probably not that many years ago because Chad GPD wasn't out for that many years, um, is the level of intelligence from, you know, just a chatbot to, you know, a reasoner, um, to an innovator and eventually become an organization. Um I think to some extent it's a it's an product play but uh to some extent as well it is sort of showcasing the capabilities of LMS uh and and large models in terms of their um sort of like capabilities u scientifically. So, so the question to the panel is like um can the route of RL eventually um give us the possibility of generating new knowledge and being actually an innovator um through pure RL reasoning or does that require uh a lot of additional data that that's human annotated for this to happen? I think we've tried all all orders so we're going to go back to the beginning. Uh Leo, maybe you can start.

Sure. Yeah, great question. Um yeah I think it's possible right for RL to create new knowledge from uh without human labor right um um and then as we have seen that uh I think there is a success story where uh you let the RL system or model to explore in math you can find new mathematical facts I think these are examples um but I just want to go back to an earlier point I I making that the the ceiling of RL is depends on how good the simulator is or how good the model is if the model we provide the RL system is in a form of the rules or just the uh the knowledge we know about human know I think the at the end of the day the RL can only explore wrong knowledge or a new knowledge in this framework it won't be able to tell to create new physics theory right or or new new discipline um I think to in order to go beyond that I think we need something different or something fundamentally different probably

Yeah, I do agree I feels like I mean as Lihon has just mentioned so I mean I is clearly depends on the model of the environment I mean which most specifically depends on the model of the dynamics model of the reward. So I think that for the means of fields where we have relatively accurate uh environment model and uh also suppose we do the compute in a reasonably correct way. I do feel that the IO paradigm can take us very far away. Uh but there might be some fields where that either it's impossible or too costly to do some very accurate simulation. I mean in such fields and probably I mean I is not the best paradigm to move forward and we need to either use some available data side of thinking another paradigm to move forward.

So first I want to answer I think AI has shown us already they can do discovery beyond humans. If you follow Alph Go or uh you you you know move 37, it's like a move that the AI made that everyone thought with the 3,000 years of Go history, they thought this was a mistake and then later on it turned out that this was actually the winning move. So it was something counterintuitive but it worked and it aligns what what both Lee Hung and Jen mentioned which is if you have a good simulator which in terms of go it was and also a really good reward function eventually it can go beyond what every human is capable of doing so and I think for LLM we are changing the domain we are going towards like understanding new physics and dynamics or like material but the problem is the same if you have a really good simulation that the agent can freely explore go beyond on what humans actually providing and also really really good rich reward function. I think it is very very much possible given the past history that we have seen in this space.

And I think for LLM we are changing the domain we are going towards like understanding new physics and dynamics or like material but the problem is the same if you have a really good simulation that the agent can freely explore go beyond on what humans actually providing and also really really good rich reward function. I think it is very very much possible given the past history that we have seen in this space.

Yeah. Um cool. So one of the things I I do want to mention is like um going beyond what we were just talking about innovation um um is like this goes sort of back to the the fundamentals of compute where uh verification is fundamentally easier than uh sort of generation or creating solutions. So um a lot of times that we're talking about verifications uh on these innovation innovative systems they are basically the simulator they want. Now the the additional question I want to add on here is is there a possibility that we can go beyond like numerical rewards.

Um because in a lot of situation like what Jen said like verification systems or simulators of RL systems they rely on a binary or numerical reward signal for us to be able to train RL system going that direction. So is there a possibility that we can go beyond numerical reward system um to be able to sort of like learn something smarter without like actually annotating with numerical rewards? This is a complete open end question. So feel free to skip if you don't have an answer for this.

Um I have a question for you actually. What do you mean by non numerical rewards? At the end of the day we're doing all this matrix multiplication at scale. So

right. So, so like you can think of like for example um the probability of distribution of words like on on the token level um they're not technically reward right they're they're basically probabilities can you leverage those kind of like probability distributions as a guidance for you to train our system instead of just like using like one final reward at the end uh as a reward signal.

So I I can start taking a crack at it. I'm Rich is a student. From Rich's perspective, everything has to come down to a singular numerical reward function. And uh there is there is a really interesting podcast with Rich. I highly recommend go listen to it. But other people say, "Hey, we as humans, we have a lot of things that we want to maximize. We have family, we have work, we have uh career inspiration, we have like I don't know fun, hobbies. How all of those become one single reward function?" And his answer is like maybe there is a single thing that we want to maximize and it just uh shows itself in various ways in our life. And his belief is that for the agents it's going to be the same thing. If you find that what should drive the agent all of these amazing behaviors would actually emerge from that maximizing signal.

But to your question maybe some of these can come from preference. Maybe we cannot put a point that is like how good this is is 7. We don't know but if you ask enough from people it's like hey is this one better than that one? We know that humans are much better in that job compared to give you like how good is this summary and maybe we can then walk backwards from that and for example extract that

J you want to take a quick stab at it.

Yeah, I kind of agree. I think there do have some sightings in which we think the reward should be a vector or a reward should be a probability distribution. But for the current paradigm, eventually I mean if that's the case seems you have not finished your reward model and somehow you need to convert the vector or the probability distribution into a scalar so that you can optimize. I do want to mention uh an exception is in multi- aent reinforcement learning you can think each agent has its own reward. So in this case you can think the reward is a vector and also in such settings you will deal with a notions like nash equilibrium I mean which is a little bit different than just doing optimization.

Cool. very briefly I think numerical rewards I think we need that something like but on top of that I think maybe multi-dimension of numerical rewards

cool yeah that goes to a lot to the generalized file function stuff as well um yeah I just want to throw one creative question out there so that there's actually a discussion then pre thought of uh answers all right so we're almost out of time so I want to to uh spend the last few minutes trying to like ask each panelist about where do you guys think uh the most near-term kind advancement that RL can bring us uh either on personalization scaling or just any AI system advance advancements you will see from RL in the next year or so.

Uh since I spoke a lot I think the abstraction is the way to go. I'm very passionate about it. I think it's going to give us a lot of goodies coming abraction abstraction.

actually I have the same answer. I mean I think hierarchical reinforcement learning and abstraction is the way to go.

Yeah. On top of that, another uh direction I think may be useful to look at is the uh or fruitful is uh how we can learn from more diverse data not beyond verifiable and LRM is a judge kind of reward signals.

All right, cool. Uh that's everything we have for today. Um thank you guys so much for the attention and hopefully hopefully you guys learned something uh today. Thank you guys.

Thank you.

Great. Thank you. Thank you, gentlemen. Thank you so much. Very very