Aryaman: I don't know why these things work. I attribute it to divine providence. Kind of crazy that we have to rely on people who have amazing intuition to make progress in the field.

Chen Qian: There is internal steps of reasoning happening in models that they don't say. If you have a really good language model, it will be able to predict the next word correctly there. To do that, it must have like a world model inside of like how people interact. We don't know what that world model is yet.

Aryaman: The goal of interpretability is like figure out what that is. None of our techniques are good at controlling the internal thoughts of a model. If your model is doing something bad, no one goes and edits the thoughts.

Chen Qian: Aryaman, welcome to a Valley 101.

Aryaman: Thanks for having me.

Chen Qian: Yeah, thanks for driving up to San Francisco and really I'm really excited we finally got to chat. Interpretability is something that people have been telling me to look into for a while and I thought it was pretty fascinating as well because I mean why wouldn't we want to know whether our AI overlords are plotting to kill us. To get us started, why don't you just like give the viewers an introduction of yourself.

Aryaman: Yeah, so I'm Arian. I'm a PhD student at Stanford. I'm advised by Chris Potts. And yeah, I have a background in linguistics in undergrad. Always been fascinated in languages, and then all this crazy AI stuff happened. And you know, given that I'm pretty scientifically minded, I wanted to study like, you know, why do these things work. So interpretability was a natural direction for me to work in.

Chen Qian: Nice, yeah. So I guess to put it simply, interpretability is just we are trying to understand what's going on inside a model. But this is a pretty broad question, and I feel like there could be a lot of different directions this could take you. What is your definition of interpretability research and what do you focus on specifically?

Aryaman: Yeah, so I would say there's a lot of angles that you can take for approaching interpretability. The perspective that most interested me is like, okay, like five years ago even, all the things we were doing in AI were not really successful, like especially in language models. People were using language models for a long time, but it was sort of like post-editing speech transcripts, this kind of thing. And then something happened such that they became the thing that everyone's working on, they invest billions of dollars into them, and so on. So, for me, interpretability is like trying to answer the question of like what changed, like why were things not successful before and now they are.

Chen Qian: One of the things that kind of motivated me to do this interview is Anthropic's release of this paper called J-space, or I guess a workspace for AI models. And I'm just going to summarize it on a pretty high level and you can kind of walk me through the details. But basically, it sounds like the researchers used a method to look into what's going on inside an AI model that it didn't say during the thinking process, and they introduced this concept of J-Space. One of the very fascinating examples that the researchers did in this paper is that they asked the AI model the number of legs of an animal that spins web, and the AI model answered eight, where the word spider did not appear anywhere. But then when the researchers went inside the model and changed spider to like ant, the answer changed to six. This is one of just a few examples of how it seems like we can manipulate how an AI model is thinking about things. So why don't you just walk us through this paper a little bit and kind of just tell us a little bit more about why we can edit a word that has never been said by an AI model.

Aryaman: Yeah, I thought that was a fascinating paper. And I think it adds to a growing body of evidence that like there is internal steps of reasoning happening in models that they don't say. I haven't read all like hundred and whatever pages of JSpace. But I sort of understand the idea. The broad idea is like we have this operator called the Jacobian, which you know is basically derivatives from calculus. We can use this in a model to tell us like what directions in its, you know, latent activation space, because you know everything's vectors. So it tells us what directions cause the model to say certain words in the future. So maybe like, you know, 10 tokens before, if the hidden representation inside the model is pointing some way, then it'll say the word spider later on.

Chen Qian: Yeah.

Aryaman: And one nice thing about the paper is that this is like a very elegant and simple thing that somehow no one had really exactly tried before. But I think we have a lot of evidence that like, yeah, these internal reasoning steps are there. So even if you look at Anthropic's earlier work on circuits, they had this thing called the biology of large language models, and they showed like, oh, when a model is doing something...

Chen Qian: Right? Like it's something that didn't really appear in the training data, but it kind of deduced it.

Aryaman: Yeah. In the context, if it's, you know, if you have a really good language model, it will be able to predict the next word correctly there. And to do that, it must have like a world model inside of like how people interact and like what the evidence was for, you know, all this.

Chen Qian: Oh, is it like we don't know what that world model is yet? Is that something that we're trying to solve right now or?

Aryaman: Yeah. So, I think the goal of interp is like okay, we know in order to do really well on next token prediction, you must develop a really complex internal model of like what's going on in a text. And like our goal is to like figure out what that is.

Chen Qian: Got it. Got it. Uh let's just take a little step back into this internal reasoning because it's it's kind of in direct contrast to what we now know as chain of thoughts, which is what the model spells out when it's syncing, if I can put it that way. And why is chain of thoughts not the best way to understand how like a model thinks?

Aryaman: I don't know if I'd make that strong a claim. It's definitely useful. And so it's also super obvious that like models that didn't have chain of thought like a year and a half or two ago, they're like worse. They're like strictly worse than models that are trained with chain of thought now. Therefore, chain of thought must be doing something useful, like something very useful.

Chen Qian: That's like when we switched to reasoning.

Aryaman: Yeah. When we started training models to reason before they say their answer. Yeah. I'm a big fan of like you should do the simplest thing first. This is like everything I've learned in research. If I combine it together, the most important thing would be like just do simple things first before moving on to complex things. So, I think like checking the chain of thought is super simple compared to solving interp, we should do it. And in practice, like you know, when Anthropic or Meta or OpenAI, they run their safety audits, they do check if the model's chain of thought is like saying sus things or or like planning to do weird things. And it seems to help them catch misalignment to some extent. So yeah, I do think chain of thought is like useful to analyze.

Chen Qian: I guess before we get into more examples of how this works, I just wanted to talk a little bit about kind of the end goal here because I think it's pretty clear why we should all care about what's going on inside a language model. But I'm wondering what do we do after we understand what's going on inside it? Like what are the applications, or we don't have to get into specific applications now, but like what is the goal and why should, you know, us normies care?

Aryaman: I think so there's sort of two sort of applications that I feel like, you know, solving interpretability would lead to. So I think one of them is like trust. Like to have trust in technology it's important for us to know how it works. If you think about, I don't know, like there are regulatory bodies and governments that regulate like oh how you manufacture airplanes, how you do like industrial machinery, or like, you know, oil fracking and all these things. It seems like to regulate, you know, impressive technologies that can be useful, we do need to understand the failure modes and like, you know, figure out how to assign liability and all these things. Yeah, I think right now it's really hard to regulate AI stuff because we sort of don't know how to prevent bad behaviors or like even know why something bad happened or or like who to blame, like because no one really knows, you know, where some behaviors come from. Like we have this, you know, massive datasets, long training process. It's really hard to tell what caused something. So I think sort of for improving societal trust it would be useful if we like understood why behaviors are happening even if we don't have a perfect understanding. If your doctor is using an AI system or something like that, you as well as your doctor will probably want to know like why it's producing some output, right? So that's one side of things.

Chen Qian: Yeah.

Aryaman: The other side of things is like I do think a lot of AI progress, like research wise, has been extremely empirical. And this is not a bad thing, like of course in our field, it's so easy to run experiments compared to like chemistry or biology or, you know, real world things. It seems like as stuff has gotten better, it's because some people, or like some really smart people with great intuition have come up with things, and they don't know why those things work but they work. There's a famous like Google researcher Noam Shazeer. I guess he's not at Google anymore, but he he invented a lot of the architectural improvements that led to modern transformers and he's on the original transformers paper. He has a paper where he introduced gated linear units, which are just one more architectural thing. In the end of the paper he's like, so he shows all his experiments, he shows they're better, he's like, you know, I don't know why these things work, I attribute it to divine providence. It's like a crazy line to put in a paper. He's like, sure, like God gave me the vision of like trying this experiment and it worked.

Chen Qian: So...

Aryaman: And I think that's like kind of crazy that we have to rely on people who have amazing intuition to make progress in the field. Like there should be something more systematic, or I believe there must be systematic reasons why certain things succeeded and others didn't. So I think if we solve interpretability, we could figure out why things were some things worked and some didn't, and maybe that will help us improve the models in a more systematic way or control bad behaviors in a systematic way.

Chen Qian: One of the things people say is that a model could be sufficiently aligned that at least it wouldn't lie to a user, and you know we've been talking about like interp and understanding what's going on. Kind of what is the relationship between that and alignment?

Aryaman: I do think a big motivator for many people working in interpretability is like we cannot trust what the model says, right? A model might on the outside, like on the behavior we observe, behave totally fine, especially in like a testing environment. Maybe it behaves perfectly fine and you think it's good to deploy, and then you deploy it and then some crazy stuff happens. You know, maybe the model is aware that it's being evaluated and then when you deploy it, it's like, oh, now I can do what I really want. Yeah, a lot of the interest in interpretability for alignment is motivated from like we cannot draw good conclusions always just from the behavior of the model. So I guess I could give like an example.

Chen Qian: Yeah.

Aryaman: So let's say like, you know, maybe there's some new person. You know, when you make a new friend, you do want to make sure they're a good person, right? Someone you can trust, especially, you know, before you start telling them your secrets or things like this. And an interesting thing about people is like we of course cannot know what other people are thinking at any time, but somehow society still works, right? So, if you make a friend, you know, maybe situations happen where like if they had taken a certain decision, it would mean that they don't have your best interest in mind. You know, maybe they have to protect your secret at cost to them and they choose to like take the cost to help you. And you know, from events like this, that's how we build trust in other human beings even without knowing what they're thinking internally, right? So like people's actions sort of reveal their character in the right situations.

Chen Qian: Yeah.

Aryaman: You know, alignment is somewhat similar where like with these language models or any AI system, one way to develop trust in them is to like test their behavior. But it's of course quite different because you can run a brain scan on an AI model at any point. You can get the exact values of the billions of activations inside the model.

Chen Qian: Right.

Aryaman: It's of course quite different from having like a human employee, right? Where you have some baseline trust that okay, even if they hate you, they're not going to put poison in your soup, because they have a lot of reasons not to. But with an AI system, you don't have any of those baselines. So it could decide in a microsecond to do something completely crazy. But because it has this transparent nature, if we knew what to look for, then we have so much more leverage than if we just test it behaviorally.

Chen Qian: Can you just elaborate a little bit on when you say like when we know what to look for, what specific are we like looking for now?

Aryaman: So I guess, you know, maybe we know that okay, you know, if this vector inside the model's internal representation is pointing in this direction, we know it's planning to do something bad, you know, something like this. If we knew this, we would just check, right? We don't need to do anything crazy.

Chen Qian: No. Exactly. And it sounds like because of that it's easier to do that on models instead of humans because in models you have like, you know, thousands of snapshots.

Aryaman: Yeah. Sort of. If maybe if you figure out one of them, that solves a lot of problems, right? If you can reverse engineer one model. Got it.

Chen Qian: Moving back to kind of like the general research, what Anthropic has done in that paper, as I understand, is not that different from kind of like what people are doing. It uses sparse autoencoders. Can you explain what a sparse autoencoder is?

Aryaman: Yeah, I guess it would be useful to like broadly explain what a sparse autoencoder is for. Anyone with some ML background will have heard of autoencoders. The idea is you train a model to reconstruct its input through a bottleneck layer.

Chen Qian: Yeah, that's sort of the idea. The vector is a nicer representation of the thing.

Aryaman: And so autoencoders are a classic concept. In interpretability, the idea was like, okay, we'll take all these hidden representations in the model. The neurons inside the model don't mean a single thing, they represent many different concepts in superposition. The sparse autoencoder works by projecting these representations into a much higher-dimensional space where only a few features are active at a time. This gives us monosemantic features that represent single, clean concepts.

Chen Qian: As in, the numbers are not clean enough?

Aryaman: Yeah, they still don't mean like one thing. And also depends on what data you train the sparse autoencoder on. Even if you train two different sparse autoencoders, you might find slightly different features.

Chen Qian: I feel like one of the essential problems with understanding how a machine thinks is that it rather than really think in natural language. So it seems like you have to translate the thinking of a machine into a plain language. Yes.

Aryaman: Yeah, that's exactly the issue, which is that you have to translate the thinking of a machine into a plain language. Yes. And a lot of the paper seems to be already doing that.

Chen Qian: So, what have people done looking at seeing you know how messy it is?

Aryaman: I think the classic example of this is the Golden Gate Bridge Claude example from Anthropic. They found a feature that corresponds to the Golden Gate Bridge inside Claude. Then they amplified the activation of this feature, doing what we call feature steering. The model became obsessed with the Golden Gate Bridge. Whatever you asked it, it would recommend the Golden Gate Bridge or try to connect it to the bridge.

Chen Qian: This is a very intuitive example showing we can edit what a model is thinking about.

Aryaman: Yes. But it also showed that it degraded the model's capabilities in other tasks. Because it was so focused on this one feature, its general reasoning and utility plummeted. It shows that editing hidden states is a blunt tool and can disrupt the balanced representations in LLMs.

Chen Qian: That's very interesting. So how is academic research different from industry research in this space? Is the industry leading everything while academia lags behind?

Aryaman: Yeah, so I wouldn't say I have a ton of industry experience, but yeah, my feeling with interpretability is that there is more sharing of results and progress. We have startups like Goodfire AI and Transluce lab. Goodfire has a large team trying to reverse engineer scientific discovery models, including DNA models and stuff. So, it would be cool to figure out how language models work, but we can also train models to model the climate or protein folding. Can we pull out human understandable algorithms from them? The scientific applications could be very big.

Chen Qian: Yes, extracting algorithms from them would be very significant.

Aryaman: Yes. Because ML models can process patterns in data that humans cannot visualize. If we can extract these into human-understandable algorithms, it would be a major scientific advance.

Chen Qian: Has interpretability research also helped improve the model architecture design itself?

Aryaman: Yeah. This is a great example. So the early state space models have been a thing for a long time, but they were always worse than Transformers.

Chen Qian: Tell me more about it.

Aryaman: In Transformers, interpretability research found that the reason they perform in-context learning so well is due to induction heads. Induction heads are circuits that look for patterns like [A][B]... [A] -> predict [B]. Early state space models couldn't do this. They were always worse at this kind of associative recall.

Chen Qian: They couldn't generalize rules?

Aryaman: Yeah, they were bad at associative recall tasks. When researchers at Stanford figured this out, they modified the state space architecture to simulate induction heads, creating H3 (Hungry Hungry Hippos). This advance led directly to models like Mamba and Delta. It's a prime example of interpretability diagnosing a flaw and enabling architectural improvement.

Chen Qian: So, is a model that's more interpretable necessarily safer?

Aryaman: I don't think so, because any useful technology is dual-use. If you know exactly how a model represents things, you can use that knowledge to audit it, but a bad actor can also use it to jailbreak the model more easily.

Chen Qian: Yeah, it's a paradox.

Aryaman: But I do think transparency is overall a net good. Right now, we trust our lives to these complex algorithms that we don't understand. The AI Safety Institutes in the UK and US are investing in interpretability. But it's still early, and we cannot confidently base legislation on these techniques yet.

Chen Qian: Another thing that came to mind is character training. It feels like these models can split into thousands of different personalities.

Aryaman: Yeah. Interpretability show that the training process shapes these latent persona trajectories. It could be a way to finely control character behavior with more guarantees than we have now, where we just train on data and hope for the best.

Chen Qian: Awesome. Thank you so much Aryaman for joining us for this podcast and thanks so much for your time. Looking forward to your research and what you end up doing after your PhD.

Aryaman: Thanks for having me. Appreciate it.