Token 流与新型价值网络：从支付欺诈到 AI Agent 时代的自适应防御

Alex: Hey, there's a new type of unit of value that's being streamed across the internet called a token. And over the next 10 years, the entire internet value chain was going to have to deal with the fact that like the more valuable tokens got, the more bad actors were going to go to try to get their hands on those tokens. And anytime you scale something and the payload gets more and more valuable, more bad things people try to get access to that value. Online payments, you know, is started roughly in the 80s and '90s, right? And grew to over a trillion dollars over the next 10 years, and we needed to build entirely new payment solutions to deal with online fraud. Where we are today is roughly there on tokens, but over the next even five years, we're expecting the token economy to get to like roughly $5 trillion. And over the next 10 years, I'd be shocked if we went to 10 trillion of token flow. We blocked 10x as much dollar volume last month as the month before, and the types of token fraud are diversifying quite a bit.

Host: Before we get into today's episode, I just have a small message for listeners. Thank you. We would not be able to bring you the AI engineering, science, and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content. We've been approached by sponsors on an almost daily basis. But fortunately, enough of you actually subscribe to us to keep all this sustainable without ads, and we want to keep it that way. But I just have one favor to ask all of you. The single most powerful, completely free thing you can do is to click that subscribe button. It's the only thing I'll ever ask of you, and it means absolutely everything to me and my team that works so hard to bring the Inspace to you each and every week. If you do it, I promise you, we'll never stop working to make the show even better. Now, let's get into it.

Host: Okay, we are here in Anj's house, which is where all great startups in San Francisco start.

Host: Howdy. And congrats on Cursor, Mist... god knows what else. You got so much stuff going on.

Anj: There's a lot going on. Well, OpenRouter is probably the most has been the most I would say one I'm excited about recently.

Host: Yeah. And we have Alex, first time on the pod, but you've been a few times. I appreciate every time you shown up for the community. Congrats. I just like what a journey. When I was looking back at your past posts, one of the earliest principles that I saw you write as a sort of product person is pub sub as a product principle. And I wanted to you to maybe explain how you think about what should exist in the world.

Alex: Yeah. The the pub sub piece which was early 2023. I didn't think about it until we talked like 10 minutes ago is about how there is like a way of thinking about products as an intersection between subscribing to data and publishing data and marketplaces are an easy easy example of this. You have suppliers that are publishing some kind of product to a SKU. And the SKU is kind of like a pub sub topic that a consumer is subscribing to and just going to like consume whenever they want. And humans consume in a very like discreet ad hoc way. It's not very scalable. You know, all their attention is on the topic when they're buying the thing and their attention is nowhere else when that happens. Agents and consumers of inference don't act like that. They're consuming continuously and they're changing the SKUs that they consume from all the time. So OpenRouter is sort of like a blend between a you know a normal API experience and a marketplace where we create model slug. We have the auto router. We have all kinds of like product SKUs that you can subscribe to and then you can like continuously add like derive value and make decisions based on those consumers.

Host: Yeah, this is something that was more consensus now but not consensus when you guys started which was that there is such a demand for swapping models and changing things out and that people would not use the native SDKs. I guess for each of you, what was your sort of realization moment that this would this would be it? You've given a talk at AIE about Alpaca as like one of your inspiring moments?

Alex: Alpaca I can like rehash the Alpaca moment for a sec. Like the very beginning at the end of 2022, OpenAI was the only game in town. There was like OpenAI, Cohere, and then a smattering of early attempts at open-weight models. When Llama came out in January of 2023, it was like, "Wow, really exciting. This is really big. It outperforms GPT-3 on one or two benchmarks." But you can't chat with it. It wasn't actually an engaging model. But it seemed like someone just needed to fix a couple things and do some RLHF on it to get it all the way there. And Alpaca was the first model that I saw that did that. It only took $600 to do. A team at Stanford generated a bunch of synthetic data, fine-tuned Llama, and made Alpaca 7 billion parameter model—or was it maybe it was 13 billion parameters—and it was so good, like I was just like on an airplane using it. In many cases, you could not discern a ChatGPT versus an Alpaca result. And I figured if it was this easy to make a model, one, we have a whole new way of monetizing data for the first time. You can just like take really valuable data and turn it into a service in $600, and that cost will probably go down over time.

Host: When you say—so sorry—when you say monetizing your data, as what eventually will become an HTTP endpoint or as training data for a model?

Alex: Yeah. Training data for a model, like an abstract way of saying like, "Hey, I have this data..."

Host: Compress it into a model.

Alex: Like it makes sense for me in my product, but like I could repackage it in the form of a model and sell it. And so it's just a whole new business model for the economy. It also of course provides like you know a way of following what frontier labs are doing, but in a way that like a single developer or a small team of developers can roll on their own. And so whenever you have an example of that—like a breakout app that's doing really well, and then some kind of framework for imitating it with your in your own flavor—you have an immediate ecosystem that should arise, because there's just a huge gap between the decisions that the single company is making and all of the variations in those decisions that like a wider ecosystem can can create themselves. And so then you know you need a marketplace to like discover all of those services and all of those products. There wasn't any place on the internet that was like a home base for LLMs in terms of seeing how much they were being used and seeing who was using them and why.

Host: The closest would be Hugging Face.

Alex: Hugging Face was the closest.

Host: They just started Hub like a few years before before that.

Alex: Yeah. And Hugging Face also didn't have the closed-source models.

Host: Yeah.

Alex: And they didn't—you couldn't use the models at the time. And there wasn't data about who was using them. There are like a bunch of differences between OpenRouter and Hugging Face, and those differences felt really critical to me, especially when I was just trying to learn about LLMs and like why people are choosing like these different little ones that are emerging over time.

Host: Got it. And then Anj, no stranger to wanting more model diversity at the time. You know you're a couple years into your Anthropic journey, which we covered in a previous podcast as well. What was your introduction to Alex?

Anj: Well, the introduction was I think 13 years before that. Oh.

Alex: But the OpenRouter handshake actually happened right over there, if you remember. Yeah.

Anj: Which was Alex and I met, I believe it's sophomores now if I remember correctly, Review...

Alex: Meeting for the first time.

Anj: I think so. Yeah.

Host: Yeah. So Stanford Review was the libertarian newspaper on campus at Stanford that Peter Thiel had started back in the day. And forever for...

Former Discord Head of Platform: whatever reason I you know Alex and I both showed up to one of the meetings and I remember um the editor-inchief was a mutual friend of ours. Lisa was really a really great editor-in chief. You know part of part part of an editor-in chief's job is to assign responsibilities to people and make sure the work gets done. Um, and I I I may be misremembering the details, but I I remember wanting to it was kind of surprising to me that at the time there was no dedicated technology section in the newspaper.

Alex: Um, you know,

Former Discord Head of Platform: because it's political, right?

Alex: Yes. Yeah. States and things.

Former Discord Head of Platform: Yeah. But it it to take us back in time, you may remember this, but um there was there was this technology uh kind of legislation that was being debated called uh the net neutrality act. And net neutrality is like inherently this political concept, right? It's it's about the regulation of internet broadband access. And so there was a community of us who are kind of technologists but also debating the politics of the technology. And I thought the review would be a great place for to like write about that. And I was working on I think a net neutrality article and I remember proposing well maybe you should start a technology kind of section and Alex was one of the only people who said yes that would be cool and said I I forget whether we ended up writing stuff together but that's when we first met um was 2011 or 12 I forget which year it was. It was one of those.

Former Discord Head of Platform: Um it was at Old Union if I remember correctly. That's where we used to meet. But, you know, along the way, Alex and I have had a chance to to to hang out often. And probably the the the time when we had the most professional overlap was when I was running the platform of Discord. Um, and it had become this explosive kind of platform for crypto.

Alex: Yeah.

Former Discord Head of Platform: And NFDs in the middle of the pandemic,

Alex: which also, by the way, you were in charge of safety and security as well, right?

Former Discord Head of Platform: I was the head of platform which meant all of the crypto DAO and NFD launch security debugging fell on me and

Former Discord Head of Platform: the fishing the the the social engineering attacks like a ton of DOS that we were getting hit by um around the time I started teaching security at Stanford CS53 and Alex was in the at openc at the time and I was trying to figure out how we we could defend against all these attacks that we were like and at at peak I forget if you remember how much NFT volume was running through

Former Discord Head of Platform: discord but it was like a meaningful amount of like se it's like several billion dollars in NFT volume of GMV so to speak running through the platform and it was all coming from openc it was these like buy sell trade

Alex: I mean the server the D and D is discord

Former Discord Head of Platform: yes and so that's when I think we had hung out professionally but a year after that openai gave Discord early access to GPT

Former Discord Head of Platform: sorry GPD3 no it was GP3.5 actually yeah GPD 3 which is the RL version of GPD3 and that's around the time we we made a Discord bot with um OpenAI for internal deployment and that's when I I realized we would need like since I since I was part of the deployment team what was the use case there were two that were and there there's actually a post now called Discord is your place for AI with friends that somebody sent me recently that I wrote um and published in 2023 but there were two use cases one was Clyde which was that in like a first party friend inside of Discord that could help you set up your Discord server and talk to you about onboarding and get your friends to hang out more. Um, and then there was content moderation and one of the realizations we had with content moderation was the it would refuse to moderate like it would just refuse our prompts because the RL the post training was we were very early in the post- training era and it would just our prompts would trigger it. uh it's like guardrails. And we told OpenAI, hey guys, we need access to the weights because if we're going to be doing content moderation at scale, we had 250 million monthly active users. We need more reliability that the model will do what we need it to. And they said, well, sorry guys, that's not how this works. We're a closed source company. And so that was my first realization that we needed open models and the enterprises would need more control o over capabilities and then ultimately would need some kind of control plane or management system to orchestrate these open models. But there weren't no good there were no good open alternatives until maybe 6 months later when Llama came out and 6 months after that I led the series A into Mistral which was started by Giam and the Llama team and I that around that time is when I remember hearing about Alex launching open router and going these worlds are going to collide and I don't know when it'll make sense to team up but Alex was so early and could see I think he was totally right about this ecosystem starting with Llama uh that then needed like a an easy layer to to manage for especially for I was approaching it from the enterprise perspective because I'd been that like the as the VP of platform at discord it was my job to ensure that when we deployed models to like 250 million users they did what we wanted them to and that was very hard um cuz if you outsourced it to the labs and they controlled the the guardrails and their guardrails or their safety policies forbid the model from responding to your prompts that was quite catast catastrophic.

Alex: Yeah. But well, you know, a moderation is a thing that they want to support and obviously beyond that they would work open would work with you uh you know presumably to give you a moderation endpoint which they offer for free.

Former Discord Head of Platform: It was an interesting use case um that they so they did give us a moderation endpoint. However, as you guys know, every Discord server is like a mini deployment of itself. And so the use case was instead of having human moderators that have to interpret the norms of the community, you just give the they often like every you know subreddit discord servers public ones have their own rules that the user the randomly space discord in yeah

Former Discord Head of Platform: and then humans used to read those norms and then enforce it every day manually like observing each message in these communities and these communities are like millions of users. So we had a 5,000 plus person team globally in the on the Discord content moderation team. These are outsourced contractors who had a really tough job. And so the idea was instead if you could give the norms of that server to the LLM, then the LLM would do custom moderation for that server. It's almost like a like in context moderation for that server. And many of those servers norms just violated OpenAI's rules. And so that it it was like we had our own custom eval. So we used to like each server had its own custom eval but this at the time OpenAI's eval so primitive in our thinking about how to deploy these LMS that often the um the post training prompts were super heavy-handed. It said oh anything about Harry Potter anything that has trademarked content you know don't refuse. And it was a if it was a fan Harry Potter fan community this is a real use case that had a content moderation the LM would just refuse.

Alex: Yeah. And that was just not precise enough.

Alex: Another one that that we heard was like if someone was trying to write like a detective story and there's one chapter with a lot of violence, like maybe someone like kills someone, the LMS would just refuse to like help with that part of the story.

Former Discord Head of Platform: Yeah.

Alex: And then they like the user would be like, "Okay, this this is not like structurally inherent to LLM. There must be like some choice out there so that I can like switch to another model." um when I'm getting like a refusal or a bad result from the the main one that I have and and that like tension also drove me for a marketplace.

Former Discord Head of Platform: Yeah, I think that is well accepted now. What was it like back then when you were raising or you know starting this? Um did people get it? Um you know what was the some of the struggles? Basically I like I like getting stories out of him about how other VCs don't get it. So like anything anything you want to you want to uh talk about now you

主持人: Know now that let's let's call it the, you know, that the early journey of OpenRouter is done, right, you can obviously talk about some of the early days stuff.

Alex: Uh well I was going to say that like the the biggest objection we got is is big model win which is all the value...

主持人: Scaling laws.

Alex: Yeah, scaling laws, um and natural network effects are just going to kind of accrue to one company which will be like it'll be a Google-style monopoly, just like how Google won the search market, um by a large large margin, um and you'll just be fighting for it scraps at the end basically. That was probably the biggest objection we got.

And it is interesting that Google won the the the search engine race with such a huge margin. Um, you know, I think like had there been more interesting benchmarks or had like search engines been, you know, a bit, you know, have people like seen them a little bit more like LLMs where they're services that you can build companies on top of, that might not have been the case.

Um, but LLMs don't merely have a user interface. They're also like ways of building entirely new businesses. And you know a Google-level monopoly would be like the Dutch East India Company times you know quadrillion in magnitude, cuz the whole economy ends up like depending on the one monopoly as well. So it didn't seem like you know a like would be a really crazy outcome if that happened. And it's also less likely because the the economics of like creating good competitors are are much like much more decentralizable.

投资人: Everything Alex said is true and I came at it from a completely different perspective which is...

主持人: Yes this is why we're here.

投资人: Um the scaling laws were never like in my mind were always a feature not a bug for why OpenRouter would be very valuable, because I was one of the first investors in Anthropic and it was obvious to me that other researchers in our friends group... and I went to grad school for machine learning and I just had a lot of friends in the ML community who it was a very obvious to us that the bitter lesson holds. And so I was like oh like fantastic now we have at least two pro proof points that compute scaling works. It was OpenAI and Anthropic.

Um, and by the time I think we decided to team up on OpenRouter, I had already invested in Mistral and Black Forest Labs and Luma. So there was multiple model companies and teams that I was uh working with.

主持人: But you did other modalities whereas this is different modalities.

投资人: Exactly. And it was so obvious to me that an ecosystem of different kinds of models were being created, um and that this whole do narrative of like only one company will dominate like Google was... um well like may be true but one I don't believe that, but two there was so much extraordinary innovation happening across several different research teams. But the shared problem I was noticing across all of them was often, you know, the research teams were fantastic at figuring out how to reason about new capabilities, they think in terms of capabilities, but never like are not developer mindset oriented, like what happens after the training is done and the checkpoint comes out. Like you'd be shocked how how like similar the early pre-training teams at OpenAI... uh sorry Anthropic, BFL, Mistral uh were in in in their like default approach to taking their research out of the you know lab and kind of scaling their impact, which was often oh the checkpoint is done put it out as an API done, and then there'd be crickets.

Um in in the case of Claude, the first Claude checkpoint, and it was actually done a year before they released it internally. And then ChatGPT came out and we decided, okay, yes, it's a good idea to to release a Claude version externally. And they had no plan, like no no no plan for how to get developers to actually try it out. And so if you go to the Claude 1 blog post, you'll notice they like three kind of developer examples for users of the API: and one is a Discord bot, and the second is Vivian my wife's startup called Junior Learning, because and then there was like Notion, um because these are all friends of like the Anthropic team. Because that's how like last minute the planning was around hey once the model's done training how do you get it out to the world.

There was no distribution platform that understood what developers needed: all the the key management, provisioning, like simple like you know endpoint management, versioning control, like all these things that the scientists and researchers go, "I mean that's plumbing, I don't detail, right?" And instead, Alex came at it from that perspective.

And so, you know, it was so obvious to me that like every single lab I was funding would would spend like literally sometimes billions of dollars into training and then a checkpoint would be done and there'd be crickets like doing early access cuz they're like, "Oh, that's right." Like it's hard to use a checkpoint to make anything. You actually need a whole bunch of plumbing around it to make it usable by a developer. And so by the time I think we it was so obvious to me that a distribution platform like OpenRouter was critical to have in the ecosystem if we wanted there to be competition to Google. Like unless, you know, with Google DeepMind is done training a new checkpoint, and then they push a button and it gets blasted out across all their surfaces from Google Docs to you know...

主持人: Everywhere even if I don't...

投资人: Everywhere you want to know about like on Android like like overnight they can deploy a new checkpoint to like a billion devices right, and that invisible infra advantage distribution advantage most people don't realize. But until OpenRouter showed up, you had to think about all of that yourself as a model lab. And it was very daunting, you know, at Anthropic, I think it took more more than 12 months to get to our first 10 million in revenue. And in contrast with with Black Forest Labs, I remember the early days, you you guys had a conversation with the BFL team and uh it it was so simple for OpenRouter to say, "Oh, no problem. Like the day you launch, we can send a million developers to you." You know, that that was crazy. That was like a step function change in in like power.

主持人: Is that a real number? Million.

Alex: I I I think today it's like million. How many developers are on OpenRouter today? We over 10 but over 10 million but um but like it's it's hard to I don't know how to we do a lot of like you know account deduplication work but you know no one...

投资人: If you could get a thousand developers, just to put in context, if you get a thousand developers to actually try the model on day one after you release it and just like do inference and give you feedback, that's a thousand more developers than they knew how to get to on their own.

主持人: Well, you know, BFL had a reputation.

Alex: Yes.

主持人: They had one with Stable Diffusion.

投资人: Yeah.

投资人: And with Mistral, uh I don't know if you guys remember, but the first checkpoint they they released was like torrents. It was like torrent weights.

Alex: Yeah. They just put up a magnet link.

投资人: Yeah. There was no API.

主持人: Cuz they didn't they weren't in for people, you know, like, okay, download these weights and you guys go.

Alex: He has a story on his side. Yeah.

Alex: Yeah. I I mean in addition to the like building a really good developer experience around it, the marketing that we do uh like for different models is totally different and perceived totally differently from the marketing that a model lab does for itself.

投资人: Yes.

Alex: We are like a, you know, neutral layer looking at this market like it's a big dark room with all the corners completely obscure to users and users were walking into the room and like feeling around and trying to figure out what objects to grab off the tables and like build into uh their companies. It's an insane way of working. Like models are not products where you can just enumerate all their features onto a web page. They're all black boxes, including the openweight ones. So, you need to like shine lights on all corners of this room um so that people can see what makes this model good. And you need the company shining that light to be a neutral third party, which is what we specialized in. So the like it's in addition to developer experience there's also like a very important like...

Alex: ...marketing and product packaging component and a way of like routing and discovering models becomes critical to your go to market as a provider or a model lab or a server tool and more in the future.

Investor: And this value to your earlier point about how many VCs, you know... One of my biggest frustrations is venture capitalists, many of them just don't have any operating experience in the field, you know. So unlike a traditional investor who's just maybe come up through the ranks as an associate working on financial modeling, or maybe hasn't been a real operator in the field for more than 10 years—which is a big part of the industry now.

I had just arrived at a16z like a year after running the platform [at Discord], and so I knew what the challenges were of building a real developer experience and actually being able to create a working piece of software with a model. And there were a few—I won't name names—but there were investors who were looking at OpenRouter and felt at the time, when I would compare notes with people, that it was just, quote, "just a marketplace."

Host: Yeah, just a thin layer, just a proxy, just...

Investor: ...a wrapper or whatever on other people's APIs. And I was like, you have no idea how strategic the value that OpenRouter has created by being able to orchestrate even three APIs in production. The amount of both engineering work and community design that goes into getting that actually live and running in production at the scale the OpenRouter team had started just doesn't happen by default.

You know, and that was one of the things that stood out to me about Alex from the earliest days. He just understood from a systems perspective how you get these flywheels going. That stood out to me with OpenSea when we were working together on the NFT integration at Discord. Alex had a level of community systems thinking around how you get these flywheels going that most scientists and machine learning people just don't think of. We often think in terms of pre-training, mid-training, post-training...

Host: It's a linear stage. It's this linear pipeline. There's no loop.

Investor: Yeah. It wasn't until much later that the modern context feedback loop cycle really got standardized in the industry. But at the time, if you remember, machine learning was like...

Investor: ...mostly we did a lot of ML, like when I was in grad school, on a laptop. So you just download a dataset, ran some ablations, and you looked at the loss curves and you're like, "Great, I made AI." And the idea that you have to deploy those capabilities, collect feedback trajectories, then put those into a continuous loop came much, much, much later. And it was very counterintuitive to the traditional AI science mindset.

I do remember during the investment phase for OpenRouter, I just didn't try and re-educate a bunch of other VCs on why it was not just a marketplace. I was like, you know what? I'm just going to invest...

Investor: ...and I'm going to take the opportunity to partner with Alex. And if no other VCs get it, that's totally fine, 'cause at the time it was not obvious to several other investors that OpenRouter was not more than just a wrapper around APIs. And that infuriated me. And I was like, you know, I don't have time to debate you, we're just going to invest.

And then I think like a month later Matt Murphy marked it up by 10x our—I forget what the exact post-money was and so on—but to his credit, Menlo Ventures realized, okay, there's actually much more strategic value here as well. Maybe you didn't hear all these conversations behind the scenes, but that frustrated me a lot.

There's a lot of this opining about wrappers. And if you're like, "Oh, an app is just a wrapper on a model," and "OpenRouter is like this wrapper on top of other APIs"—this is the most stupid, reductive framework. It's clearly somebody who has no experience deploying.

Host: It's the thing you dismiss other things with, like, "Everyone's a wrapper on everything," right? And at some point, some wrappers have value.

Investor: I mean, investors are wrappers on LPs, right? Like venture capitalists. So I mean, yeah, it's all wrappers down—all down to bare metal, I guess.

Host: When I started the whole [AI] Engineer—I guess the coining in 2023—that was the number one pushback, is that "This is no value. You should actually just train models, right?"

Host: And yeah, I mean, obviously you guys are one of the testaments to the fact that you can actually build very valuable wrappers, but also very valuable model companies.

It's so hard to be... the day a model launches, the fact that you have an OpenRouter endpoint for that model frequently at the top of Hacker News on day one—people don't realize the amount of work that goes into accomplishing that. And OpenRouter did that over and over again. And I remember going: people have no idea how hard that is.

Host: Yeah, we've covered some of the inference engineering that goes behind some of that with Baseten and all those.

Host: Well, today you have all those cool code name things, and people guess what "Oxy Alpha" is and all those things. But I guess one of the things that you're teasing is: how do you get that initial flywheel going, right? Because today you have your scale and your reputation, all these things, so obviously you drive immense distribution. But when you're early on, when it's most...

Host: The bootstrap, yeah. How—what is the bootstrap like?

Alex: I mean, to bring it back to early Discord days. I think we initially connected—this is an OpenSea story technically—but we initially connected when you were at Discord and we talked about the Axie Infinity server.

Investor: Yes. Yes.

Alex: This server was like the biggest server at the time at Discord.

Investor: That's right.

Alex: And you were kind of like constantly bumping up the limit.

Investor: The limits on the server—for those who don't know, like 10% of the Philippines was actually on that server. I was on that server.

Host: It was like a meaningful crypto game. But...

Host: There's like a Pokemon breeding thing.

Alex: Similar. Yeah. There was battling, there was breeding, and then there was a marketplace for trading...

Host: Play-to-earn as well.

Alex: Yeah, play-to-earn. And the graphics were really cute and fun, and you get kind of emotional about your Axie that you make.

So to start a community like that—which we had to do many times at OpenSea with basically every early project for us to create a marketplace for it—we need to make sure that the community actually wants it. And it's kind of like building something that people want and going and telling them about it. You can do that on a one-on-one basis, but it's way higher leverage to do that in a community where everyone can talk to you at the same time.

So we spent a lot of time building things that the community really wanted. We did the same thing for OpenRouter. The Axie community was one of a zillion communities we did that with. And you saw us doing it because you could just see people sharing OpenSea links constantly in that Discord. Users sharing links is a really clear indicator that something important is going on.

So we spent a lot of time first figuring out what the gap is in the technology that people care about—like, what was the actual problem that needs to be solved. In early LLM days, it was OpenAI refusing to finish the prompt or complete the task. It was also the inability to customize models. And so there are communities that are just completely blocked on that issue, and those are the communities that are most useful to sort of learn about, dive into, and explore.

Investor: Something that really struck me at that time, as I was just hearing your talk... I remember noting how—you may not remember this, but we had these working Zoom calls that we were doing a sprint...

Head of Platform at Discord: ...around for like this OpenC integration with Discord. Um, and you know, we we'd get it would it was myself, my engineering team, I think you were there. And I remember, you know, um, Alex in the middle of one of those calls just like there was like silence, uh, you know, we were we all like, "Oh, yeah, this totally makes sense. Let's do this." And then there's some like everybody aligned and Alex was like, "No, this makes no sense to me." And everyone's like, I remember going, "What? What?" Like that it works. like you click on a link and this then it bounces you out to like open C and he was like it's not a good user experience. Yeah, we should not do this. And I remember going, you know, he was the only one person out of all of us to actually raise his hand and go, yes, it made sense from a technical implementation perspective, like we were bouncing the user out into the into OpenC. And so it kind of checked the box of the product manager requirements on both sides. But Alex went one step further and was like, you know... And not one person on the call, like seven of us who had met like, you know, week after week.

Host: And it's the guy who doesn't work for Discord.

Host: Like technically you benefit if they bounce.

Head of Platform at Discord: Exactly. And that was like adversarial to keep the user inside of Discord would be adversarial to OpenC. And yet Alex put that user experience first. And I was like, that's special.

Host: Wow.

Head of Platform at Discord: Cuz it's very hard to have somebody who's technical like Alex and understands the developer flow, but also understands the best user experience and wants to prioritize that. And that's two sides of the fly. that you can get spinning like is often hard to stop and you just reminded me like that one was one of those moments where I go I went I realized I got to be better at user experience cuz I should have been the one who came up with that and I didn't and I learned from you and um I think that went into one of our case studies for the PM training program at this I don't know if it's there because

Host: you need an Alex conclusion

Head of Platform at Discord: yeah yeah you need an Alex and and this is why I'm not you know nobody should be surprised why Stripe decided like they had to buy open router because it's a really rare combination of people who understand the machine learning community, the developer experience and the end user experience and putting all that together has resulted in this extraordinary scale that very few other marketplaces have been able to achieve over the last you know 5 years.

Host: Yeah. Well, we should talk about the other reasons for acquisitions uh which you've written about. Uh I want to sort of proceed somewhat chronologically as well. So that there there is a point that you know one of the questions that uh Dave from HFZ sent in was when did you know it start really started to work and you brought up Mix Draw I don't know if you want to bring up that.

Alex: Oh yeah

Host: which obviously you overlap with so

Alex: yeah thee was I don't know when I mean the there's no like one moment where I was like oh this is you know officially starting to work. It was like moment increasing like really super early on.

Alex: Oh yeah. Well, yeah. So, before open router, I wanted to like explore a bring your own model experiment and um

Host: anyone familiar with crypto is like you know phantom and all these things.

Alex: Yeah. Yeah. So, it felt like doing a meta mask analogy for AI would be kind of a fun way of exploring that. And at the time there were no AI apps. There were probably as many AI apps that were like hitting AI via like hitting an LLM via an API call as there were like games just doing it in JavaScript. You basically like there there was a there was a moment in time where it could have been the case that web apps call LLM through the browser like through some kind of desktop managed app that is controlled by the user. Um, and of course there are like I think many reasons that that did not happen, but back when the when the days were that primordial. I built a a a Chrome extension called window AI and

Host: with plasma which I had come across early on and I was like who's going to actually use this? You did

Alex: plasma had a couple like I think Phantom was using it. Um, there were some other like like real companies basically

Host: react for Chrome extension. It compiles to all these kind of like Nex.js for

Alex: and yeah built window AI on top of it. The creator of plasmo like started contributing code to window AI and uh in GitHub and that turned out to be Lewis Vichy who is the co-founder open router.

Host: That's you have told me this is how you met Lewis.

Alex: Yes. Okay. So, um, that that allowed users to kind of like configure which model they wanted to use for a web page in their browser and then like the the app would just call out to that model when it needed to do things. You know, not the right form factor for LLMs, but you know, it's like fun experiment. you learn a lot and like I you know open sourced it and uh and you know the main learning is like okay this has to be an API and it has to look a little bit like there has to be more of a developer experience here and more of a discovery experience as well like I don't know where to use these models and a little Chrome extension is not going to help me discover it's not enough real estate I need more space I need visuals I need graphs I need you know examples I need images I need to like I need to be able to like explore both as a human and as an agent. So that's kind of how how open Rider came to be.

Head of Platform at Discord: You know, a meta point that

Head of Platform at Discord: I think is underappreciated, but Alex is reminding me is that we were quite lucky that we were so we were like adjacent to the crypto community in those days because in hindsight, crypto ended up being kind of like a dress rehearsal for generative models, right? If you if you think about the the AXI experience, uh you know, Alex is totally right. There were not that many AI apps at the time. And while I was dealing, you know, my job was to be the head of platform at Discord, which meant to be a general purpose place for communities and friends to create uh for developers to create apps and bots and you know, other services that could be deployed across Discord. And while 80% of the attention of the time was being spent on crypto because that's where all the NFT volume was, there was like 20% of my time of my time I was spending with a friend u who would get hotbot with me and ask me for we would play Magic the Gathering on weekends. Um and he was working on a little Discord bot that could take a text input and turn it into an image and it was called Midjourney.

Host: You know, was that David?

Head of Platform at Discord: That was David Holtz. He was a good friend and David and I have both been sort of failed ARV VR founders, you know, in the last before that. And um I remember this you midjourney was one of the fastest growing communities we had after axi infinity started to peter off and many of the the like the abstractions and the infrastructure decisions we made to scale Axi happened just in time because you know Axi did this and then fell off a cliff and then as Midjourney was taking out we like explicitly decided to help David make the server the midjourney server as the primary place for interaction with the with the model because it was very hard for people to understand how to use the model if they couldn't see other people using it and copy them. And so the single player Midjourney web app on its own like majour.com had like terrible retention cuz people would show up they'd see this empty field. It's kind of like Dolly 2 um and they would type in like cat or dog and it was like paralyzing for them to have this blank canvas that they had to fill because they never used an AI model before. but instead in a discord server you could see other people using it and riff off of their prompts and the engagement was off the charts and so scaling you know midjourney from zero to like 10

Speaker A: million monthly actives was a much smoother approach post Axie Infinity and so

Speaker B: don't forget the best of four pictures, which is the feedback loop, the RLHF feedback loop. Which, by the way, separately like Tom Brown, David, and I used to play Magic: The Gathering on weekends, and so it was one group of friends would hang out. These concepts were all being discussed all the time. But you know, I think there were few of us who bridged both the crypto worlds and the AI worlds. And compared to crypto, where the question was always "what's the use case for this technology?", there was never any need to ask that for AI, because the use case was so visceral. It was like: I can create now anything I can imagine, I can write novels, I can code. And the infrastructure that those of us who believed in the distributed systems value of crypto—like the censorship resistance part—found this use case that was explosive. And I think between Midjourney... Claude was a Discord bot pre-launch that we were using internally as an LLM; ElevenLabs had a TTS model that we had on Discord as well. Discord became this petri dish for early apps to innovate. And I don't think it's a coincidence that they found a home there before OpenRouter gave the world a public home store or storefront. Discord was this petri dish storefront that had kind of piggybacked on the...

Sean: So then my question is: how come you were... my perception is OpenRouter is not that Discord-centric, right? You have a Discord...

Speaker B: Yeah.

Sean: And you use it to engage your community, but it's not like Midjourney where that is like the primary way people experience OpenRouter.

Speaker B: Yeah. Midjourney, like it really helps to see visually really quickly how people are using the model and how to prompt it.

Sean: And I think that is partly why the server was so critical. It is the user experience. It actually adds a ton.

OpenRouter Team: Yes. Um, and you can go the whole mile with just like prompting via Midjourney, like via the Midjourney Discord server, getting your images and then sharing them and having fun. For OpenRouter, for LLMs, you need a lot of user experience around LLM to make them really usable. And seeing the examples of other people is also not as useful because it's a lot of stuff to read. It takes a long, long time. You need codebase integration, not possible to do in a Discord server. Or technically it's possible, I shouldn't say that, it's just not a great developer experience. You need governance: at the point where you got codebase integration, now you need governance for managing the LLMs that have access to it, the data policies, which teams—all that stuff needs a lot more than a Discord server can provide. So it's just like...

Sean: It's not the right...

Speaker B: Well, in addition, you're not wrong, but also there's the very important distinction that Midjourney was an end-user application.

Sean: Right.

Speaker B: And you know that's why Discord, which just has 250 million monthly end consumers, made it make sense for Discord to be a host for that application experience. What I knew was going to happen soon after Midjourney found explosive product market fit—because when Midjourney launched, from launch 100 million revenue run rate was less than eight months. And shortly thereafter Stable Diffusion launched. And all of us used to hang out in the Discord server, it was the...

Sean: The Stability Discord?

Speaker B: It was the LAION...

Sean: Yeah, LAION community that Stable Diffusion...

Speaker B: And so when Stable Diffusion came out, I realized...

Sean: Oh, now other people can build their own Midjourney.

Speaker B: Because until then Midjourney did not have an API, so they were a full stack company, right? They were training their own models and they were deploying them as an application. But if you want to build your own Midjourney, there was no API of that quality. And I think DALL-E 2 was still quite primitive, like Midjourney actually had great quality. And then when Stable Diffusion came out, suddenly there was this new capability in the world, which is a developer could create their own Midjourney. And that I think created the need for something like OpenRouter. Because then you need an API to... if you had the kind of creativity of David Holz and you had Stable Diffusion as the model and you wanted to put these things together, how could you do that without having to figure out how to host the weights? And what the shape of OpenRouter enabled is that right when you have open models, alternatives to closed sort of applications, OpenRouter's value in the world becomes extraordinary, because now any developer can just show up and...

Sean: Model... did you just say "the shape of OpenRouter"?

Speaker B: Oh no, I'm misaligned now, I've been overtrained. I've been using Claude way too much, haven't I? "Claudish" is what people say.

Sean: "Claudish", oh god, I got to untrain myself. Okay. And I just want to cap off the Mistral side. My TL;DR is there was a Mixtral price war is what they called it, right? Roundabout NeurIPS 2023 or '24, they launched the Mixtral 8x7B, and like the price went down like 80%. To me that's very positive because it's like the first real competition to host Mistral. Is there more?

OpenRouter Team: Yeah, that was... I'm like trying to remember all the things that happened. We saw that model come out...

Sean: And immediately saw people say that it was the best model in the world. Like this was, to my knowledge, the first time an open weights model was called that in real seriousness.

Speaker A: It's hype, right? Is it, you know...

Sean: It was hype, it was also hype from AI influencers at the time. And there were many examples where it was like outperforming GPT-4. So people really wanted to try it out and see: is this going to be true for me too, and if so, at what price? And the inference landscape was really messy.

OpenRouter Team: Yes, we cleaned it up. It allowed providers to compete on price so we could give users the best price in one spot. And so it was, I think, the first clear example of a provider marketplace working in a way that adds value to developers.

Speaker B: Sean, you may not remember this, but I think we met for the first time a few days after Mixtral came out at NeurIPS at a luncheon.

Sean: Yeah, that's where I also met BFL as well. Yeah.

Speaker A: And Guillaume was there. I was at NeurIPS at that time.

Speaker B: You were there too. And we had just announced the Mistral investment, and I remember Guillaume was over there, and I remember turning to Guillaume and asking him like: "How are you feeling after the launch of Mixtral 8x7B?" And you know him, in his typical French fashion, was like: "I mean, it's an okay model, it's not that good." And I was like... it was so in contrast! But I remember him also saying that part of the reason he felt a lot of people thought that it was better than GPT-4 before was because of the speed. It was an MoE model that they had absolutely figured out how to make super efficient. It was on the Pareto frontier. And this is an important thing with LLMs, right? Sometimes when they're faster, you think they're smarter. Even though if you did N of... these common eval tries—and I don't actually remember, I think we should go back and figure out what the data says, but I wouldn't be surprised if it turns out on an N of 7 attempts, GPT-4 was smarter on evals, or correctness would be smarter or more accurate. But from a human preference perspective, people felt that it...

Alex: was faster because it was smarter because it's so fast.

Host: Um,

Alex: and actually most queries do not take that level.

Host: Don't take that, right? This is the start of humans as router which then eventually becomes open router as router of like the auto you know what I mean like because humans are the routing mechanism like I will ask the fast model first and then if like oh not good enough I'm going to upgrade manually

Investor: but then he's going to auto it

Host: I didn't I hadn't thought of it that way but that that I mean that makes sense

Alex: which then there's there's a lot more techniques like fusion fusion is a thing that we should talk about before I move on to those things I just want to close off the sort of early early years uh one thing that I observe which you are also an investor in arena Right.

Host: And we talked about midjourney having that that feedback loop of ABCD and choosing that very being being very important. And you understand the flywheel. So how come you didn't build arena and how come Arena didn't build open router?

Alex: Well, Arena started before open router, right?

Host: They had they had the school project and then they became a

Host: Marina. Yeah. So, but and I know you had some Arena experiences like the the heads up comparison type things, but you never really went as hard as Arena did

Alex: and doing heads up experiences

Host: and and LMS actually did have a router project based on Alam Marina ELOS uh which they never commercialized.

Alex: It's hard to do a company that does both because one company is taking data and selling it and the other company really can't by default. So you know I think there there is like a branding reason that there are two companies here. Um like when you set up open router there's no training or no prompts like aside from what your provider policies set like open like open router can't see your prompts or completions. If you want to see that as an org, you have to opt into it and enable it. And so we're like pretty conservative and careful about data policy and security and privacy. And Elm Marina is like their business model is is like oriented around the labs and and

Host: because they give it for free, right? You don't give it for free, they give it for free.

Alex: Yeah. But I mean, we do give some we like have free endpoints too, but like those free endpoints, I think we're not collecting any of the prompts. We're not like monetizing the data unless you you know opt into it for some reason.

Investor: Th this this comparison I mean you're not the first person to ask me this and Alex knows this but I you know I was the the inter like the founder like first C CEO of Arena for the first 5 months when we were helping Anastasia and Whan kind of spin out of Berkeley and uh I did invest in that before um open router but it was it was very strange to me the comparisons that outside you know folks would make between the two projects because the missions were completely different. the founding entity for Arena, we called it the AI reliability institute because it was it was actually there as an eval service like the data so to speak that they they were originally um kind of offering the labs was how do you make the evaluation of models more reliable than kind of like the state-of-the-art at the time which is like really just finger in the wind. Um that that's kind of what Anastasio and and Whan's PhD work was as as scientists at Berkeley was on statistical methodologies for sort of uh correcting you know eval estimates um based on like intrinsic biases and how you collected the data

Investor: um and style control style control and stuff like that and which is very much like a hey how if if you're a scientist and you're trying to kind of um the highest expectation customer for Arena was always like a a post-training and uh like a a researcher at a lab. Whereas the highest expectation customer from from my perspective that that Alex like really understood and and was the mission was to serve was was like a developer, right? Who then takes the result of the research and then produces an application that's deployed to the world. It's actually a completely different problem and person that these two teams were focused on. And so from the outside in actually I don't know if you remember this but I have a distinct memory of a few weeks before we did the term sheet uh together for open router I'd given you a call because we were trying to get a pool's data set together from open router and from arena to uh create like an open- source repository of prompts.

Investor: I mean these projects were so kind of different in their goals that it was totally normal to me to be like oh yeah let's call Alex and see if you'd want to team up on on pooling data cuz they're so different. we need we we actually don't have that kind of data at all. We like we didn't have API prompts. We we didn't have like what developers want to do with the models which is very different from what researchers inside a model lab want to do before releasing the model.

Alex: Yeah.

Investor: Does does that make sense? And so to this day I I think you see that this difference even though at a 30,000 foot level you could I guess you could kind of conclude that arena and open router are adjacent but uh the road maps the missions and so on at the time at least were like in very different sort of directions.

Host: The ideal customer I get I I totally get that as a founder I want to own everything right. So like this is clearly the adjacency then I'm like I'm going to explore that. uh own everything meaning like you don't know what to do yet so you want to like make sure you catch PM I think what he say you you want to own the entire infrastructure space and so you'd kind of expand to whatever demand

Alex: yeah I I think that's that's hard you know in reality because serving multiple customers is is

Host: clearly you know this is the only one of focus right

Alex: yeah I I still think even in the age of AI like focus is is underrated and critical Not just because you end up with a better product by focusing your humans on it, but also because the world knows what your focus is.

Investor: The world can map like, oh, I have this issue. Which brand out there is going to help me with that issue? This is the brand that's known for that focus. So like if I want real attention on this issue, like this really matters to me, I should go with the brand that cares the most about it. to underscore Alex's point about how important focus is. In the early days of Anthropic, it it was not easy to like people think that the early days of Anthropic were like super easy because they were on the GPT3 guys who left. But it was actually very competitive. The company was starting $10 billion behind OpenAI,

Alex: right?

Investor: And so to get to the frontier like the big question was what what do we want to be known for? What's the mission? And the mission was AGI pair programming. And so to the um exclusion of all kinds of other things that were really shiny at the time like image models and video models that were getting lots of you know momentum the anthropic team was like we just got to focus on coding like that is the core capability that we're focused and today you can see the results right it's a trillion dollar company within 5 years and that focus I think like the high the focus on who your highest expectation customer is and how you exceed their expectation because exceeding anyone's expectations is hard and doing it for multiple like customers is so even more difficult um is part of the reason why opener succeeded and entropic as well.

Host: Was the focus on coding that early though or did it come later?

Investor: Literally from day one it was AI pair programming is responsibly commercialize an AI pair programmer was the seed memo. That was when I invested.

Alex: Right. We we actually kind like refine that memo a lot.

Investor: Well, you got to ask Dario and Tom for permission on that. Um but it's an extraordinary piece of writing that they had put together and AI you know commercializing responsibly commercializing AI pair program was the mission you know from day one. And I would say there was maybe like a couple moments in the company's history where like they did experiments to kind of see if like little detours

Founder: made sense like a general chatbot like Cloudi when chat GPT was really taking off but at the end of the day especially once um they got their significant pre-training computer online I think like the all the main evals at the company for example have always been coding evals long horizon agentic programming I mean from day one that was always

Co-host: when like flawed instant came out and Claude 2 came out. Yes,

Sean: I remember the marketing mostly being focused on pros like this model

Co-host: write better and

Sean: long context

Co-host: long context um

Sean: this directly affected me cuz I told something on that

Co-host: what did you make

Sean: uh small developer which was my Devon before

Co-host: oh yeah small

Sean: yes uh and uh you know so I think like there's there's all that really like good like focus is another thing that is a question that people do want to ask uh you know you could have built any any other things like and obviously open roto was working working working uh were there other ideas that you wanted to pursue that you turned down you know just the paths roads not taken

Founder: we made a couple prototypes for things that we didn't launch one was a fine-tuning model as a service

Sean: lot of that open pipe and all those things

Founder: but it was it it kind of was in a very consumerry form factor where you would give us a YouTube video or two or three we would then extract all the transcripts from it and try to fine-tune model to talk like the person in the YouTube video or the people in the in the videos that you sent. So like a really really easy way of creating a fine-tuned model based on like some kind of videos that you like.

Sean: That would be so useful.

Founder: We we we made it too. It was like

Founder: I it was and nobody used it. was we didn't actually like test it with that many people because the model marketplace was our main our main focus and it and it was like growing and we're building more conviction in it over time.

Sean: Um

Sean: just just as a creator I've been pitched many like uh you I have 500 hours of recorded voice of myself

Sean: make a thing of you charge access to it. Uh works for only fans doesn't work for for us as regular people. I think this is mostly uh it's just a glorified rag bot whether it's in the weights or it's outside the weights doesn't really matter. You're just doing rag on the videos and people ultimately always just want to find the source video uh that directly answers it.

Founder: My use case was mostly to practice with from with myself cuz I often like to see what like the way I practice for a job interview or if I'm hiring a candidate or public speaking or whatever is I I wish there was like a good mini me

Founder: that I could like critique cuz it's kind of hard to pull yourself out. I would never get I would never offer it to other people service like pick your top five mentors that then talk to them instead of talk

Sean: that would be cool too. Yeah,

Co-host: that was that's a replica

Founder: and that was the use case we were aiming at.

Co-host: I see.

Sean: It's like you want to create an experience

Co-host: like AI Steve Jobs and

Founder: AI AI Steve Jobs was the the initial use case

Sean: that's a even though it's not allowed

Sean: that's a that's a common prototype. you know talking about adjacencies fine-tuning as a service as part of the router service is something that I would typically think about as well right like like why don't you do that because if people are running already their inference through you store everything log everything uh fine tune to a smaller model that is cheaper faster all these things that that's within your control right uh you didn't do that but like other people would have pitched that in the general state of of infra startup

Co-host: I think you were just maybe a little bit early because today that's an extraordinarily fast growing segment like you know from astral where they do a lot of enterprise deployments I mean it's often fine tuning is you know custom models for ASML or whatever often

Sean: but not as a router they're just like I come to you because I like your ML models I want custom model right it is not I want uh to run all my openi prompts uh get store all my results and then just move off of openi right they're not doing that

Co-host: uh as a as as like a way to export off of dependency on a on a frontier lab I have not seen that yet which

Sean: which was your kind of

Founder: decision to do I mean we we decided really we like leaned into our focus and figured that like there are like we just saw the ecosystem develop over time. All these inference providers that that do want to help companies do that. Um be like like it makes sense for us to partner with them and to like give users lots of choice and to like you know figure out what makes them um what gives them competitive advantages. It's it's a whole new business basically and there's there's value in being a neutral marketplace that just kind of like works with those companies. Um

Co-host: could you share a little bit um to to Sean's point like how you prioritized what what are some ways you prioritize features cuz you've always done it so elegantly. I never you know it just happens and you make all the right decisions that always have product market fit from the outside looking in. consistently you seem to have prioritized you know a lot of hit features that worked and maybe I have a sample set bias or whatever but Sean

Sean: can list what you think hit features worked well like

Co-host: oh the leaderboards like

Founder: leaderboard okay

Sean: yeah you know like from day one

Founder: the feedback

Co-host: charting B okay

Co-host: but like he had like plugins uh you know he had like uh and I think there was a whole thing I want to get into about like completions versus

Founder: yes

Co-host: check completions versus completions and then also uh let's call it like the the rise of the reasoning models and how you deal with multimodality all those all those things by huge one

Founder: there's one like I think it was in early 2024 very early 2024 we thought it might be interesting to fuse the results of multiple models together

Founder: and we launched a prototype called mom mixture of models that let you like pick a couple models we'd pick them for you and then it would fuse the results together at the end and it would show you all the intermediate results in this like big conbon board looking product. What does the fusion at the end? Another model.

Founder: Another model. The the smart the smartest of of the three of the set.

Co-host: So, this is like a council idea.

Founder: It was a model. It was like a very early LLM council.

Sean: This is a multi- aent swarm as as like they would call it at one of the frontier labs uh in the early days, you know.

Founder: Yeah. Like some of those ideas are like going the right direction, but the devil's in the details. There's a lot of like product refinement needed to make them really work. um they take your focus away from right you know whatever else you have going on and there's a lot of like community building and learning that you need to do and the technology might be too early so there like all kinds of reasons they might go wrong and in our case the technology was a little too early in other words the fused result was a little bit worse sometimes the same as the best model that was being used to fuse because the best model was so far ahead of options two and three at the time. You know, over time, the top three or four LLMs have gotten closer together. Still neurode divergent, but like all capable of inserting like pretty interesting ideas. Like RL has basically like expanded the surface area of creativity for machine learning researchers within each lab and so they can, you know, diversify the reasoning power of different models more effectively. At least that's my my theory for why

Sean: uh Fusion is it like works better than it used to early 2024.

Founder: And um so the technology was a little bit too primitive. The form factor was was not right and and so we would have had to go through a couple more iterations. And so we decided to just delete all the code. And uh then years later, middle of 2026 um or early 2026, we're like let's bring it back. like the research is looking kind of promising for fusion. The models now have like two, three, four top frontier models that are

Alex Atallah: all really good and like I'm frequently trying to consult multiple models to get the best results. And then I ran a little personal experiment where I was like: I'm going to do an architecture plan for a code change. I'm going to give it to all the models. I'm going to fuse the result, and I'm going to ask all the models if the fused result is better than the individual result each model came up with. And they all said yes, that the fused result was better. And this happened a couple times, and I was like: okay, spot check pretty good. We should benchmark this, and that's how we built Fusion.

Swyx: Yeah. And it came on your fable. So you were like this is fable level.

Swyx: Yeah. Yeah. Let's start leading up to this year, which we haven't gone to yet. Can you mark out the main milestones in the journey? I think it seems like your promise was routing. You decided the business model very early: you take a cut.

And what are the major milestones that inflect the growth, right? Like you're growing like 9% week-on-week now, is that the official number?

Alex Atallah: In terms of token volume, I think that sounds about right, yeah.

Swyx: Yeah, so can you mark out the sort of brief history of OpenRouter up to the acquisition? Let's call it—we're just talking about people having your birth moment with the Mistral stuff where people are really competing. You have your State of AI thing where it's very cute, you have 100 trillion tokens haha, because now you're doing 10 a week—

Alex Atallah: Uh, you know, um—

Alex Atallah: We're doing 10 a day.

Swyx: 10 a day now?

Alex Atallah: Yeah, more.

Swyx: So yeah, you do this in 10 days! Like what are the major points there? There's a smooth curve, but you feel the inflections.

Alex Atallah: A lot of this is kind of oriented around model launches. We had a huge focus on prose all the way up through May of 2024, because coding was just not there and no apps were able to build much on top of it. So, a diversity in models, but not a wide diversity in use cases. Dream Tavern was one of our top apps at the time. The creator of Dream Tavern now runs product at Cognition Devin.

Then in the middle of 2024, we saw Claude Sonnet 3.5 come out—an incredible leap forward in coding. And we saw the dynamics of apps building on top of us change. We saw a huge surge in volume in users using OpenRouter, and this is when I think people started to look at the money that they were spending and get a little bit like: whoa, what's going on, I might need to think about more cost-efficient but equivalent models. And shortly after that—I think it was after Sonnet 3.5—Mixtral 8x7B came out, and everyone was like: what, this is the model, the open weights community delivered! And so it was really good timing from Mistral.

Swyx: Basically all of LLM companies are just helping you out.

Alex Atallah: It takes an ecosystem to grow an OpenRouter, you know.

Alex Atallah: Yeah, that was the early ecosystem. It was like a swing action where model labs would come up with some sort of frontier innovation, usage would surge, then users look at their invoices 30 days later and like: whoa, what's going on here? And then open-weight models would deliver cost-effective options two to three months later. We saw that happen several times.

Swyx: One thing you also did with the coding agents was that you broke out which are the top coding agents, and they love that leaderboard—the Cline versus Roo Code versus what have you.

Alex Atallah: Yeah. Like Cline was like the top of our leaderboard at the time. Skipping forward a little bit to the end of 2025, there were quite a few coding apps on the leaderboard, but they were all IDEs or terminal-based agents. And at the end of 2025, we saw OpenClaw appear.

And OpenClaw was particularly interesting because, one, it was a new form factor that brought in a new type of user—not just a developer, but a productivity or sort of an internet creator came to AI for the first time. And it also had an interesting architecture where it was calling your chosen model for these heartbeats to see if it was still alive, in addition to actually using the model for real tasks. And the heartbeats—you don't want to pay a lot for a heartbeat. So the auto router that we provided was really, really useful to this wide range of users all of a sudden. And so we just saw it rocket exponentially, and then we saw OpenClaw just blow up, and a couple other apps lean into that new paradigm and do something similar. Hermes came out and really leaned into things like the auto router, built a really good community, and leaned into skill management and making it really easy and effective for people to set their memory in the agent and build really good skills.

Swyx: Which is another thing you never did: memory, skills, sandboxes, all these adjacent things you could have done.

Alex Atallah: Could have, but I think like—

Swyx: It's hard to bet.

Alex Atallah: They're also things that really matter for the developer use cases that were coming out at the time. Developers wanted to architect those things.

Alex Atallah: Those were kind of critical to building a good user experience. It's been hard for companies to find abstractions that work for all developers on the memory layer. There are some—like Mastra has done a pretty good job, for example—but developers have lots of varied preferences for them. And then, the way our leaderboard has changed over time is kind of like a movie of how the AI space has changed over time. If you just go to the Wayback Machine and look at the rankings leaderboard and the apps leaderboard over time, it shows you what's happened in AI over the last couple of years.

Swyx: To me, the coming-of-age moment was Andrej Karpathy being like: "I no longer read LocalLLaMA because I just go to OpenRouter's leaderboard."

Swyx: Which I remember that. I think he probably said like: "Sorry guys, I'm going to send a bunch of traffic to you." So I also want to bring it into the Stripe thing. How does that kind of conversation start?

Alex Atallah: We had this longstanding relationship with Stripe though, from many different projects that we had worked on with them. We invest a lot of effort in countering abuse—

Swyx: Token fraud.

Alex Atallah: And token fraud.

Swyx: Can you give some numbers just so people understand?

Alex Atallah: I think I posted about this. We blocked 10x as much dollar volume last month as the month before. And the types of token fraud are diversifying quite a bit. You know, there are fraudsters going after typical stolen credit cards, but there are also people trying to resell traffic against the terms of service. There's hacked accounts. There's people whose whole company is compromised and they don't even realize it, and we help them regain control and detect it. There are accounts that are reselling inference on the side. There are accounts that are dealing with an accidental runaway agent and they don't realize it—not a hack, but it's something that blows up and the company doesn't want it. And so our trust and safety team works a lot on all of these categories of problems and helps block it and detect it. And so we've built models around them. We worked closely with Stripe for a while on this. And I think it's going to become a huge problem in—

Alex Atallah: ...the ecosystem. Like we're already seeing a lot of companies start to see these fraudsters like spread and look for other ways other, you know, other than OpenRouter to other fraud vectors. And if you're making a gateway or or selling like generalized inference, you are a target for fraud. If you're selling very discreet like intelligence products, intelligence products that are like doing something pretty specific, but not like, you know, just reselling inference with some added capability, then you're way less likely to get these fraudsters.

Alex Atallah: So it I think we'll see companies also move away from just reselling inference with some sort of like added capability and move towards sort of like discrete tasks and charging for those tasks and charging for those enhancements, and letting people bring their own inference like in a third-party way.

Host: Whoa. Okay. And yeah, obviously you would power that. Uh but you people pay uh for outcomes or per task?

Alex Atallah: I think people will pay um you know I think like the the the Datadog pricing page is a is a good look at like the future to come. It's like companies like infrastructure companies will like charge for different types of events that they're providing, and there'll be lots of like continuous pricing models that look like that.

Alex Atallah: And of course there will be like if you go down uh towards consumer apps, you know, simpler pricing, more subscriptions, um you know, fewer events to worry about, um and ones that like are not focused on just adding a markup on top of inference, not just because fraud is hard, but also because the pressure from the labs and from inf like good inference providers to like do a commit and then bring your inference elsewhere is going to be very high.

Host: Any comments?

Guest: Uh two. One, you know, I I think Alex has done a very eloquent job of describing something, you know, counterintuitively I knew would be a thing at scale like four years ago because of Discord. And the particular experience that taught me this was, um you know, as we started scaling Midjourney, you know, the one of the primary ways that we used to give away or like get people to try Midjourney early on to get to their first 10 generations, cuz you know 10 generations of 10 images generated was roughly the magic moment activation point we found. Like once you've done 10, you were like this is extraordinary. Um but for that we so we had a free trial with Midjourney.

Guest: And one day I woke up cuz they had a platform and had to monitor, I had all these dashboards, um you know, I had like three missed calls from David, and it turns out like they had there had been this flood of new users overnight, and we were like this is great, and he was like no, actually we shut down the free trial. And I was like why is that? Um and he said, "I want to look at the geolocation IP addresses." And basically somebody in China had started to resell Midjourney free, you know, subscriptions with the free trial as a way to like basically, you know, it was fraud abuse, right?

Host: Even for a specialized model like Midjourney?

Guest: Yeah, and that was actually an application. So this idea that I I think the big picture realization I had back then was, hey, there's a new type of unit of value that's being streamed across the internet called a token. And over the next 10 years, the entire internet value chain was going to have to deal with the fact that like the more valuable tokens got, the more bad actors were going to go to try to get their hands on those tokens. And anytime you scale something and the payload gets more and more valuable, more bad things people try to get access to that value. And and so it, you know, it was very obvious to me back then. And so look, to this day, I don't think there's a free turn, like I don't think Midjourney's ever actually turned on the free trial since then, because it was really not an easy problem to solve in terms of trust and safety.

Guest: That's why I started teaching the class "Security at Scale" at Stanford. Like this, like one of the that and the Anthropic learnings, to me it was clear that the need for security at scale was going to be enormous a few years from then. Cuz if you just do the math, right, think about like if where, you know, online payments, you know, started roughly in the 80s and 90s, right, and grew to over a trillion dollars over the next 10 years, and we needed to build entirely new payment solutions to deal with online fraud. Um where we are today is roughly there on tokens, but over the next even 5 years, we're expecting the token economy to get to like roughly $5 trillion, and over the next 10 years, I'd be shocked if we went to $10 trillion of token flow. And so if we were starting to see such aggressive abuse and fraud at subscale Midjourney—remember Midjourney at this point was like less than $300 million revenue run rate a year—I I just realized we were going to need like entirely new like systems to deal with the fraud that was going to happen for trying to get into the token flow.

Guest: So um my I I, you know, when I I forget the board meeting it was when you brought up that, you know, Stripe wanted to to partner up, and it made so much sense to me cuz Stripe Radar—when I was a Kleiner 10 years ago, we invested in Stripe, and the whole pitch that, you know, Patrick and John communicated so eloquently was like, hey, unlike traditional payment tools like Braintree that do a 7-day verification like KYC and email to get get the fraud out of the way, we actually just bite the fraud cost up front as a customer acquisition cost, and tell a developer like just use five lines of code and we'll start accepting your payments in 5 minutes. And what'll happen is, over time, we'll collect all this data on the developers. Cloudflare model...

Host: ...is the Cloudflare model, right?

Guest: And and they did. 5 years later, they launched Stripe Radar, and Stripe really today is a security company. That's the real—people think it's a payments company. No, the reason—there's lots of other payments providers today that give you like cheaper payments transmission, but the reason Stripe keeps, you know, being the dominant one here in the US and Europe is because they have extraordinary fraud detection that they've built, you know, over the years.

Host: The same story with Elon and Max Levchin and...

Guest: ...and Affirm. Yeah. You know, I think the story shows up over and over again where every time you have value streamed across the world in large amounts, you need new protection and security infrastructure to fight to to keep the bad guys out and allow the good people to like have their transactions happen really fast. And so I think, you know, this is why the from my perspective, like the Stripe and OpenRouter story is a security story for the internet ecosystem, for the frontier AI ecosystem. Without a partnership like that, it becomes very hard to defend the quality of experience and the speed and all the good stuff without letting the bad guys get in the way.

Guest: Um, the second is that, you know, there's this underappreciated thing about like the fact that you need to like these all the bad things that that Alex described as being perpetuated by humans right now is going to be perpetuated by AI agents over the next 10 years,

Host: right? So, think about the like recursive scale we're about to see of bad actors.

Guest: It's not just bad human beings. It's it's all the bad agents that are going to be attacking the token flow. And and there's it it's very hard if you're a researcher and or an AI lab to reason about that problem, because the only data you have is how agents you're training are going rogue. But that's just a fraction of all the bad behavior on the internet that we're going to see.

Guest: And so what you need is is defenders, new sheriffs in town with cowboy hats, um that can can see all the bad behavior from AI agents across the ecosystem, from different model labs and different post-trained deployments and different developers, and take all of that data and say, "We're going to build a shield for the entire token economy." Because without that, you know, the amount of fraud we're going to see of this $10 trillion in GMV and global GDP growth is like a huge percentage of that, I think, is going to be fraud, abuse, and we might never get there if people just...

Speaker A: Don't trust tokens, right? And I don't think this infrastructure exists. So, you have your work cut out for you at Stripe, but I don't think people have realized the scale at which agentic fraud—like bad behavior perpetuated by AI agents—is about to hit us like a tsunami.

Host: Yeah. I mean there's a lot to dig into there. I want to give you the last word. We do have to wrap. What can people expect from OpenRouter and Stripe?

OpenRouter Representative: I mean I think this is a really good way for us to accelerate go to market and to go up market more quickly. It's also, you know, as eloquently described, there's a really clear better together story here when it comes to improving trust and safety and making it really easy to accept tokens and let people bring their own inference to your app and to help developers just build on top of inference. Going forward, we have a really strong brand with OpenRouter and we're keeping the brand. OpenRouter as a product and the roadmap and the name and the brand is staying the same. What you should expect in this next 6 months is that most things will be what we would have done had we been independent, except everything will be moving faster, and that's kind of our near-term goal. Longer term, hopefully I can comment on it soon, but I can't now.

Host: Okay. Well, we'll hopefully do a follow-up at some point. But thank you for being so generous with your time and congrats on the partnership. I mean, this is one of the most beautiful bromances I've seen in a...

Speaker B: ...just starting from Stanford to here.

Speaker A: Lots more to do. Lots of sheriff policing to do of the...

Speaker B: ...of the town, the token economy. We need new sheriffs for sure.

Host: Yeah. Awesome. Thank you.

OpenRouter Representative: Thank you.