Sean Lie: what used to be considered fast at like 1,000 uh 100 or or 200 tokens per second is quickly becoming the new batch mode. And so I think that you know there's a place for that. It's great for prompt processing. It's great for very very parallel workloads. Um

Host: but my emails take 20 hours. I can hit a button and run it in two hours.

Sean Lie: Exactly. Exactly. Right. But but quickly, you know, being able to do prompt processing isn't isn't quite enough. Right. That's a big part of the market, but that's not quite enough.

Host: Before we get into today's episode, I just have a small message for listeners. Thank you. We would not be able to bring you the AI engineering, science, and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content. We've been approached by sponsors on an almost daily basis, but fortunately, enough of you actually subscribe to us to keep all this sustainable without ads, and we want to keep it that way. But I just have one favor to ask all of you. The single most powerful, completely free thing you can do is to click that subscribe button. It's the only thing I'll ever ask of you. And it means absolutely everything to me and my team that works so hard to bring the Inspace to you each and every week. If you do it, I promise you we'll never stop working to make this show even better. Now, let's get into it. Okay, we're here at Crisis HQ with CTO Sha Lee. Welcome. Uh

Sean Lie: thank you. Thank you for having us.

Host: Yeah. No, thank you for having me.

Host: Uh and it is the day after hot chips. Lots of things launching. Even like Apple launched the M6 stuff. Uh but you guys obviously had we were at Supernova. You uh talked about CS4. We're going to talk a little bit about CS5. OpenI talked about Jalapeno. All the all the hot stuff in hot chips. What's your take this year having been in this industry?

Sean Lie: I think I think uh well, first of all, thank you for having me. Um and uh and and you're absolutely right. It's a super exciting time right now. Um hot chips is a is is a is really good, you know, time when the whole community comes together. Um, and you know what what used to be, you know, I still remember going hot chips 10, 20 years ago when it's a bunch of like, you know, computer architect, you know, geeks, you know, geeking out on, you know, speeds and feeds and and now it's like this is where, you know, industry changing hardware is is being revealed. And so, um, you know, it's come a long way. It's really exciting, you know, community. And I think one of the main takeaways that that that I had after, you know, after leaving Hot Chips yesterday is like what an exciting time it is for the the chip design and the hardware industry. It's uh you know, this is really the golden age of uh of hardware right now. Um and you know, like you said, I've been in this industry for some time. Um this is a very very unique time not just because you know AI is taking off and you know and there's uh you know obviously a lot of uh people doing their own hardware but the amount of innovation that's happening now across multiple fronts we've never seen in the history of uh of the semiconductor industry this level of innovation across all the levels right in the chip uh the interconnect the system design the software the optics the you know the methodology the tooling people are pushing across the board and so like you know as a technologist and and you know a computer architecture geek at heart it's like really really fun to see this community and this industry at this stage right now and when we and we felt that like at at hot chips and everybody I talked to it's like that that buzz is there it's amazing

Host: we want to go through the chips let's you know talk about your chip first what's what's most exciting uh CS4 came out some crazy numbers here 30x faster the GPUs but uh you know walk us through what's new with the new CS4.

Sean Lie: Sure. So, you know, we we we we designed uh our next generation CS4 architecture um with a new brand new system platform with the goal really um to make wafer scale uh mainstream to make it uh uh to bring it to hypers scale and to solve uh a lot of the the the high density um data center problems that you know we're facing every single day. So we've designed this uh modular platform uh that provides twice the amount of power to the wafer than we have in our previous generation. Um twice the amount of uh interconnect bandwidth, half the latency. Um and all of it is done at the system uh architecture level. And what you see here is the result is that by being able to provide significantly more power to the to the wafers by you know packing more density into the rack we're able to drive up the performance even further. Um you know we like to say that Cerebrris is uh with our current product already the undisputed leader in you know ultraast inference. Um and and what's amazing is this CS4 product will actually push that frontier even further by taking it another two times faster. Um and this is a a perfect example right we here in this in this demo that we gave uh at hot chips. Um we're showing uh GPTOSS uh running at over 4,400 TPS which is just mind-blowing. It's like it's it it it almost feels like it's fake, right? Um and uh and and we think that this is going to really revolutionize the entire you know industry because you know not only are we able to you know continue to push the frontier on how fast these models can run um it will enable all sorts of new applications um you know uh uh user experiences start to become extremely different right what used to be batch and offline applications start to become real time um And you know all of this you know growth in new agentic flows and agentic uh frameworks now also mean that you know you're you're sitting there waiting for these agents to go over and over and over uh uh in their agent agentic loop and all of a sudden if you're running uh your model at over 4,000 tokens per second. Now the you can do you know more agentic uh uh loops you can do more reasoning ultimately you get significantly more capable more intelligent agents. And so this is what really excites us about this. Not just the fact that you we show these really really big numbers which is also really cool. Um but but the fact that you can you know really start to do things that you can't really do uh otherwise and and and start to enable brand new capabilities. Um that that's that's what makes me so excited about this this reveal that we had yesterday.

Host: Yeah. I think it's also very rewarding that you guys started on this journey the big chip journey. uh you know wafer scale everything everything that you've always said it just like people didn't take it that seriously until they had to and then now they're they're really really taking it super seriously right

Host: um I I I do feel like as you know co-founder CTL the the architect of this whole thing how are you guys approaching the new generations post like AI boom like I imagine CS123 was a bit more uh sort of uh calm

Host: now like literally open AI is launching soup the the ultra fast mode with you guys and like it it is a matter of like I mean

Sean Lie: No, absolutely. I I feel like the main shift that has happened over the last few years um for us is that you know the the the major pivot that we made was from you know historically thinking about this as a giant training machine to now really realizing that this architecture has a huge fundamental advantage for you know low latency ultra fast inference.

Host: And so maybe we'll talk about yesterday.

Sean Lie: Yeah.

Host: Uh previewing CS5. Uh can you recap for people who are maybe not not yet caught up on that?

Sean Lie: Yeah. No, no problem. So I mentioned that our CS4 system that we just uh launched has this brand new platform and CS5 is basically taking that exact platform and putting our next generation wafer scale engine on that platform. And what that means is that you get another 2x performance increase over CS4. So, you know, we're already running at 4,000 tokens per second. Now you're talking about running closer to 10,000 tokens per second with CS5.

Host: What's the rollout process like? So, you have a deal with OpenAI over the next few years, but when do we get access, you know? So, everyone wants ultra fast right now. Sure. When do we get it?

Sean Lie: that's a great question. So right now uh we are basically you know sold out of everything that we can make. The demand is super high. And so we are working as hard as we can with our manufacturing partners like TSMC to really scale up the production. In terms of timelines, you know, CS3 is obviously shipping today and powering a lot of this ultra fast mode. CS4 is sampling now and will be ramping into high volume production next year, followed closely by CS5 on that same modular platform.

Host: by the way you know the way that you framed it I realized that they didn't have a hardware play of their own that was public until now and so people didn't know what they were doing.

Host: nobody knew that that's an interesting like business decision almost I don't know

Sean Lie: I would say I would say that um you know obviously I can't uh you know explicitly comment about their business strategy. But what I can say is that when you look at companies of that scale, they have to look at multiple ways to solve the problem, right? They need massive scale for general workloads, and they also need extreme performance for ultra fast experiences.

Sean Lie: um, you know, in in the ultra fast space as well. I mean they're balancing a lot you know most recently with what they presented at Hot Chips with Jalapeno.

Host: we got to talk about this as well we got to we got to go into them so what are we thinking prefill jalapeno vs decode?

Host: what are your takes

Sean Lie: I think that's I think that's uh that's a very you know rational um you know conclusion I I think of their work, right? When you're operating at their scale, separating prefill and decode makes absolute architectural sense.

Host: I guess to to double click on that, you're the expert in chips, right? There's people see token per second and they think everything is uniform. But prefill is compute-bound, matrix multiplication heavy, whereas decode is memory-bandwidth bound. So for Jalapeno, they didn't actually specialize jalapeno for low-latency single-stream decode, right?

Host: for they didn't actually specialize jalapeno for right

Sean Lie: they did not specialize jalapeno for that but they specialized it for throughput right and so they get high throughput prefill. Prefill is very compute dense—you're doing massive GEMMs on large prompt contexts. But decode is fundamentally memory bandwidth and latency bound. When you generate token by token, you have to stream the entire model weights through memory for every single token. On traditional architectures with HBM and multi-chip interconnects, you hit a memory wall and a communication wall. Because we have wafer-scale SRAM with petabytes per second of memory bandwidth and uniform on-wafer latency, we excel at single-stream ultra-low latency decode. So they're addressing different parts of the pipeline.

Host: I think one thing that people are talking about like you know I'm trying to get to the disagreements in the industry. The SRAM versus HBM debate. You guys have been here the whole time championing SRAM on wafer.

Sean Lie: been here the whole time. We

Sean Lie: we we've been talking about it for a long time. And it's amazing to see, you know, the the industry really converge on recognizing how critical on-chip SRAM bandwidth is for fast inference.

Host: do they feel different post acquisition? I mean, you've been competing with them for

Host: a while.

Sean Lie: Uh, to first order, no. I think it's really awesome to see that, you know, SRAM architectures are being recognized. But the fundamental difference is scaling. If you build a small chip with 200-300MB of SRAM, to fit a 70B parameter model you need hundreds of chips connected across circuit boards, cables, and racks. The moment you go off-chip across copper or optical networks, your latency shoots up and network topology dominates. What Cerebras did with wafer-scale is keep all that SRAM and hundreds of thousands of cores on a single continuous piece of silicon, communicating via on-wafer wires at speed-of-light latencies. That architectural difference is night and day.

Host: I mean, there's a separate Reuben, you know, strategy.

Sean Lie: Well, so there's there's there's Reuben, but you know, the LPX itself

Host: that that was supposed to be where it was.

Sean Lie: It was supposed to be Reuben LPX together, right? Um, if if you guys recall, I mean, the Jensen spent

Host: GTC

Sean Lie: GTC like half an hour explaining attention runs here and and you know and and the movies run here and KV cache runs there. When you have to partition an architecture into so many disparate pieces, it's an admission that a monolithic general GPU is struggling with the diverse requirements of modern frontier models.

Host: Yeah.

Sean Lie: Right. And so when you start to think about it that way, it's like, well, is it surprising that they have to stitch these complex hierarchies together?

Host: so they're going to gradient this into this.

Sean Lie: Well, I I think I think it's I think it's the other way, right? I think what's what's going to end up happening is that as long as you are constrained by standard reticle limits and package boundaries, adding more specialized chips just adds latency hops and interconnect bottlenecks.

Sean Lie: All right. Um, you know, if you if you have that limitation in your architecture, then I I think that you're constantly fighting physics at the package boundary. With wafer-scale, the entire fabric is uniform, eliminating those artificial inter-chip boundaries.

Host: Yeah. And look, the market's large, right? You have a different market than them. And you know that there's room for multiple winners.

Sean Lie: No, absolutely. I mean the the the market's large there's a lot of different opportunities for you know different architectures. Batch processing and massive parallel throughput will always have a place, but for real-time interactivity, conversational voice, and autonomous agent loops, ultra-fast low latency is the key differentiator.

Host: yeah. Frontier ultra fast.

Sean Lie: Exactly. Exactly.

Host: Um, I we was going to go into some of the other companies that were, you know, top of town this year, but it gives me this gives me opportunity. I want to follow up on one thing which is how do you think about the relationship between hardware design and model architecture co-design?

Sean Lie: Well, so I I think there's probably two different views here. Um you know, one is the the product view and the other is the long-term architectural view. When we design a chip every single day we're deciding like am I going to hardcode a specific mathematical primitive, or do I make it flexible and software-programmable? If you over-specialize for today's Transformer architecture, what happens when researchers invent new architectures like Mamba, RWKV, or hybrid attention? So our philosophy is to provide massive general-purpose compute, ultra-wide memory bandwidth, and flexible routing, allowing the software compiler to adapt to any model evolution.

Sean Lie: modular it's modular and when we design a chip every single day we're deciding like am I going to use this silicon area for fixed functions or programmable cores? By keeping the cores programmable and connecting them over a fine-grained mesh, we can support Dense models, Mixture of Experts (MoE), and future architectures without redesigning the silicon every time the AI algorithms pivot.

Host: How much of that plays into model design, model architecture? So working with OpenAI, you get to co-optimize how the model is structured to take maximum advantage of the hardware?

Host: It's not enough. There's also voice diffusion. Uh what what if what if there's more interactive modalities coming down the pipe?

Sean Lie: Absolutely. And I think as as we start to get into more interactive use cases that we enable through ultra fast inference, voice, real-time video, and diffusion models become practical. In human conversation, latency over 300ms feels disjointed. To achieve sub-300ms total conversational loop including audio transcription, reasoning, and speech synthesis, the core LLM must generate text in tens of milliseconds. That requires thousands of tokens per second. Ultra-fast inference is not just about reading text faster; it is the foundational enabler for natural multi-modal interaction and continuous agentic thinking loops.

Host: Yeah. And we we don't have time for this because we want to move on, but uh the people who understand this are really excited about what this unlocks.

Sean Lie: Yeah. No, absolutely. I'm I'm a huge believer in that for sure. I think before getting into other hardware architectures, this paradigm shift in latency is the single most important concept to grasp.

Host: I think that uh Nvidia is doing exactly what we all expected Nvidia to be doing, right?

Sean Lie: Nvidia is executing very well on their roadmap, doing what they've always done best—scaling massive parallel throughput across GPUs. But when it comes to single-stream low latency, they are fundamentally bound by the physics of HBM and board-to-board interconnects.

Host: but my evals take 20 hours. I can hit a button and run it in two hours.

Sean Lie: Exactly. Exactly right. But when you take that from 2 hours down to 2 minutes, it stops being batch processing entirely and becomes an interactive development loop. That completely changes how software and models are built.

Host: Okay. Um, spicy spicy chips. There's this page that you have on your website that uh this company Etched just doesn't have. No, no numbers, no benchmarks, any takes? That's the talk of the town.

Sean Lie: Well, so they have said very little about what they're doing. Um uh but what they have said has changed over time.

Host: which they acknowledge

Sean Lie: which they acknowledge and and you know the the environment is changing a lot and so that makes sense. But in hardware, talk is cheap. You actually have to tape out the silicon, bring up the boards, build the systems, run the full models end-to-end, and publish verifiable benchmarks. Until you have real silicon running in real data centers, it's all theoretical.

Host: Yeah. But at the same time, like massive valuations now, it's like, you know, not a pushover.

Sean Lie: in the best case I would love to see them or others frankly push the boundaries of more innovative system-level architecture, because real AI scaling requires full-stack innovation across silicon, packaging, networking, and software.

Host: the rack node,

Sean Lie: the rack, the node, the package, the

Host: the the interconnect,

Host: the factory,

Sean Lie: the the the factory. Um, they may be doing something like that. It's hard to tell, right? But the real moat is the full-stack engineering execution required to make massive systems work reliably in production.

Host: scheduling, pipelining, all those algorithms,

Sean Lie: all of those things, right? And and so, you know, of the reveals yesterday, it's the entire infrastructure—power delivery, liquid cooling, compilers, and collective communication algorithms—that makes wafer-scale practical.

Host: I was going to say like your backpack stuff reminds me of what the memory people are doing. Is that a similar convergence?

Sean Lie: there there's a there's a little bit of of that, right? Because you know what we're doing with our power delivery and backpack integration is bringing power conversion and regulation as physically close to the silicon as possible, minimizing resistive loss and maximizing power density delivered directly into the wafer.

Host: Yeah. Uh let's leave some hints for people. A couple other names that stood out in memory...

Sean Lie: along these lines. I I think you know Samsung has some discussions around their ZHBM right um

Host: all memory guys

Sean Lie: all I mean look in the end there's only three main things to to building a you know a an AI chip right: compute, interconnect, and memory.

Host: and then there's memory

Sean Lie: right and so um and and more and more importantly as we all know for at least in the low latency space, memory bandwidth and latency are the absolute dictating constraints.

Host: and power and cooling not as major still.

Sean Lie: No, absolutely. I mean, you just did a big redesign. Power and cooling are massive physical constraints.

Host: Those are those are what's powering all of this, right?

Host: Yeah. I'm just saying, you know, like physical limits,

Sean Lie: but that's actually a really really good point, right? I think that um when we when we originally started Cerebras, people told us it was physically impossible—they said you can't deliver thousands of amps to a single continuous wafer, and you can't cool a 20kW piece of silicon. We had to reinvent liquid cooling from scratch with micro-fluidic manifold cold plates and design vertical power delivery architectures. Scaling AI hardware today is fundamentally an applied physics and mechanical engineering challenge just as much as a semiconductor challenge.

Host: Huawei and what what have you

Host: hyped outside of knowing what it was on

Host: just model is good doing good in you know open router came out not even running on US chips

Host: so the TDR is are people talking about it uh what's what what we doing

Sean Lie: I mean I think that absolutely I mean of course people are talking about it right because like the top open models are flourishing globally.

Host: not 100% but

Sean Lie: almost okay 95% right most of the most of the big models most of the big open models that are out there are demonstrating tremendous architectural ingenuity. It shows that algorithmic breakthroughs are happening everywhere globally, and that hardware platforms must remain agile to run whatever weights and architectures emerge from the global AI research community.

Host: so I I imagine that should take place as at Hot Chips. It's like the secret room of all you guys. You guys are pushing the envelope.

Host: right? that you would be the guys to

Sean Lie: I mean we we have absolutely been been pushing for this and we've been very supportive of of you know the entire open source ecosystem and running any frontier model at maximum possible speed.

Host: Okay. Uh you got to go but you've been very generous with your time. Congrats on all your success. It's been an incredible journey.

Sean Lie: Yeah. Yeah. Yeah. Sometimes I still have to pinch myself to remind me that we accomplished all this from an ambitious idea, but we're still in the very early innings of what AI hardware will become.

Host: still early.

Sean Lie: That was not bad. That was not bad.

Host: Uh no congrats uh and uh look forward to meeting up with you in the future year.

Host: We got to ask you know give us ultra fast. Give us bigger give us more.

Host: No no I mean give give the people ultra fast too. Like

Sean Lie: we but we but we can probably get you in line at at OpenAI.

Host: I hope you didn't bias OpenAI too much to, you know, just keep it internally. No, people need it. We need better models and faster access.

Sean Lie: It's OpenAI's decisions. We're we're just providing the infrastructure.

Host: Hey, but give us give us GLM. Give us Dec.

Host: Just give us our own rack and then we'll run it. Uh, okay. Thank you again for coming. Really really appreciate it.

Sean Lie: Yeah.