Speaker A: I think it's interesting, right? It helps people understand the capability of what you can ask and delegate that often times people don't realize until they try or someone shows you and then you're like, oh okay, okay.

Speaker B: I see. Yeah. I think it's also there's also like a light security issue where like basically you're the permissions layer. Like yes, I could query everything that you query and I could get an automated response, but maybe I'm not supposed to see it. Yeah. And then yeah, there's no way I would know because I'm not supposed to know what I don't know.

Speaker A: Especially as like, you know, with try to be work for for asking you to connect your plugins and you know, it's pulling from your local files and stuff like that. Like the the amount of context that the agent has access to is like deeply personal and like that's something that like we need to preserve. So that'll be definitely a challenge.

Speaker B: There's Excel, there's PowerPoint, there's Docs, you know, the grand trio of work. What other formats of work do you do you think about? You know, like obviously you worked on Airtable. Is there a future where there's like OpenAI Airtable? Like, you know, like what what what does that look like if if you ever ended up doing it?

Speaker A: It's a really good question. I think um I mean, one that you didn't bring up was Sites and I think that was a big part of this launch. There's one one side of Sites that I think people commonly talk about especially on Twitter and stuff or X of like, you know, this sort of like prototyping tool. Actually like we saw that happen with this launch even the the model slider that you guys were referencing earlier, like that was developed almost fully in a site. Like, you know, the the collaboration between design and engineering and product on that was like on a site where we could play with, you know, the the the affordance and and and figure out how it feels and and all of that. Um but the other aspect that that I think is a little bit less talked about is like Sites has like an an artifact for for knowledge work. I was actually talking to someone the other day who was on like our our corporate finance team and like we were mentioning how like now when they have these reports that they're they're working on as a team month-to-month, historically those things were in in slide decks and in spreadsheets and now they're just in sites. Like sites is the mechanism that they collaborate across the team. And the reason is cuz it's like it's like somewhat higher bandwidth. Like, you know, at some these tools like PowerPoint and Excel are like infinitely flexible, but at some point you reach the boundary of like either as a human you may not know how to use some feature or something or the product itself doesn't support it. But in the case of a site you can do anything you ask for anything and you can get that. I mean, once people see that magic I think it's been really valuable.

Speaker B: Yeah, let me show you my my case study. This involves all the hot topics including ChatGPT work but also 5.6, token billionaires and token maxing, and sites and auto research. I'm a fan of this game called Strata. It's a basically it's like a little board game that you that you play with a physical blocks that come on top of it

Speaker A: like that. So over the weekend I took like 30 photos and just threw it into ChatGPT. 1.7 billion tokens later outcomes this site with a fully playable thing with 3D block placement and everything. Because it requires physical blocks and I needed friends to train on it so they can get better so I can play against them. But also I could also do things like train an AI on it. And that's your auto search. That gets into auto research. So you want to train your own AIs and then make sure they self play against each other. I need to set both the AIs. So this is AI versus AI and they're going to self play. Obviously that the AI start out bad and then you want to define a loss function and get good. I wasn't going to supervise all this. I was down in San Mateo attending a conference. What I ended up doing was auto researching on this and creating benchmarks. And there was just way too many parameters for me to read. So I started asking it for a site and it's created this lab panel. Where is there a shortcut for a site that it's created?

Speaker B: Uh you should be able to go in the sidebar to sites, top of the sidebar. Uh left sidebar.

Speaker A: This one? Oh, left?

Speaker B: Yeah, just scroll all the way to the top.

Speaker A: Oh, oh, it says sites.

Speaker B: Yeah.

Speaker A: Oh, there you go. Yeah.

Speaker B: Ooh.

Speaker A: So, it create it creates the sites. Um I don't think this is a it is exactly what I wanted. Um but let me show you what it popped up, right? Like I think as a research artifact, um it is very important to communicate exactly what is being done. Outputs this thing which I eventually started publishing. So, I moved it off of sites because I wanted more database and infrastructure than sites afforded me. Uh but this is like research output that you can start to mess with and like try to think about like what hyper-parameters are you tuning for training your AIs. Uh and like I was trying to make like scaling laws and everything and doing all sorts of like game optimization stuff. Uh and the fact that you can just kind of throw this up as a research artifact, like I no longer need to read ChatGPT output. I read site output. But then there's also huge sprawl. Like look at how long this thing is. There's so many numbers. Uh it is pretty overwhelming. So, then I have to start putting in from there. But it's an interesting transition from markdown

Speaker B: Yeah, actually that you're putting out to you're you're putting a whole functional site.

Speaker A: I think markdown just isn't that optimal for people to read, right? Might as well just write HTML website and I don't know. I think you can do a lot with customizing this, right? You have your skills that explain what you want. Like I noticed they're quite verbose. I don't need a lot of this information.

Speaker B: verbose.

Speaker A: So, and then the nice thing of having a site side by side is, you know, you just iterate on what you want and what you don't, right?

Speaker B: Yeah, I don't know if that any that triggers any stories for you of how it's run internally. Am I doing this right?

Speaker A: Yeah, I mean, I think that this is like a workflow that we're seeing like all different types of teams use where like the canonical artifact that was previously a DAC or something is now becoming a site and like with a site you because it's just HTML you can like it's infinitely flexible and so you know if you want to give more prominence to a certain thing that like in a slide deck would you know feel like it was braid like you can do that you can have it be like the hero image right and so I think that like um people are starting to see that um there's obviously more work to be done to make these things like much more easier easy to collaborate on um you mentioned that they're very they're long and verbose could be broken up I'm sure there's something we can do about that.

Speaker B: long.

Speaker A: Yeah yeah but I think we're starting to see that like there there is this aspect of this is a really interesting format um for for people to use um that's like much more flexible than than what they were had before.

Speaker B: I think your job also comes becomes kind of meta you're not designing the products you're designing a product to make products and I'm curious how you manage that.

Speaker A: I I think one thing that we've been like when we look at the UX like that we've that we've been thinking a lot about is how can we balance like simplicity with capability like if we if we're designing a product like you said that like is is made to make a build out of the things right you can build so many different things but we can't put that all in front of you because you'll get overwhelmed.

Speaker B: Yeah.

Speaker A: And so we had similar problem or similar challenges even with ChatGPT but especially now like when there's so much that can be done I think the balance that we're constantly trying to strike is like how can we give the user enough of a UI surface where you know they can be expressive they can tell the the agent what they need they can verify that it's using the right tools it's pulling from the right sources etc. but then it gets out of the way and then how can we build the right system such that we can show them instead of telling them what can be done? Because so much of this is going to be like how do they discover the next use case and the next one after that if they really want to you know to to be super powered by the AI.

Speaker B: Yeah.

Speaker A: It's interesting. I feel like everyone also just has a different way to do it, right? I made a similar version of this, same game. I didn't take any pictures of board or rule game. I threw in at goal, 18 minutes 53 seconds later, a lot of tokens later I've got a similar version. Obviously not with all the auto research and whatnot, but you know um

Speaker B: You got to do all the latest trends.

Speaker A: And yeah, I I did it with uh did it with codex not work, but it's interesting, right?

Speaker B: Yeah. And this is obviously GPT image generating the avatars. Very good for game design. Like a lot of game designers were like really into GPT image.

Speaker A: I I will say like the broader takeaway probably is the reason that we do this is more so just to test the tools, right? Like um this was also a test before 5.6 came out. I had done the game on 5.5, right? The ability for me to no longer need it to I had to feed it the rules. It's a pretty niche game. It couldn't find how to do this on its own.

Speaker B: Oh, yeah.

Speaker A: Uh 5.6 auto distribution. That's is why I was also very keen on testing the 5.6 capability.

Speaker B: But you know, this is just as as work comes out as new things come out. These are just our sideways to test things, right?

Speaker A: Yeah. I It's some kind of private email, I guess. Which that is not all that private. But also valuable cuz now you can send this to your friends and I mean I learned about this game through seeing this.

Speaker B: Hard game. He's very good.

Speaker A: [laughter] Uh uh it's good to win when no one is competing with you. Uh but yes, it's a classic RL problem of like self play bootstrapping your game AI. Um yeah, you see how easily work becomes personal and personal becomes work because the thing I do for personal, it actually directly informs people I work with cuz I showed it to them. They were like, "Oh, you can do that with GPT?" Which like I I imagine is the growth strategy.

Speaker B: Yeah. The show not tell is a big piece that, you know, I think we've not still not fully cracked of like, you know, showing people all the things that they can do with the product versus like trying to teach that to them through like, you know, articles or onboarding or whatever.

Speaker A: Yeah. So, meeting them in the moment.

Speaker B: It's a career risk for me uh because I used to be in developer relations, right? Where your job is to show. And then, you're like, "What do you mean it you you don't you don't need uh it actually your job is to tell." Mhm. And then and then but the product people are like, "Well, we don't need you if our product is intuitive enough." So.

Host: Yeah, I mean, that's the magic of the models. So, you can tailor the telling or the showing to like specifically what the user needs, like what what they care about, what they've done in the past, exactly where they are on the adoption journey. So, I think that's like going to be a super big opportunity.

Host: Seems easier and easier now to tailor custom showing, right? People have different use cases. As much as you said you don't want to segment different people into different buckets, right? It's also not that hard to for people that are in different categories. But, the question I guess is you said your team is more broadly on What What was the term you used? Productivity?

Reid: Productivity.

Host: Productivity, which is now work, basically. Is it work? Is there another distribution that we're not hitting? Is there a group of people that will have something different than ChatGPT, Codex, or work? Is there Is there more that the mass isn't targeting?

Reid: I see it as like a sequencing, like, you know, the the vision is like bring useful agents to everyone. We started with like developers. Like developers historically are like early adopters. They're willing to put up with more friction, set things up, etc. Like that's where, you know, Codex started. I think the next opportunity is like sort of we call it general knowledge work, you know, all the other functions around developers. I think when when you go from developers to this segment, like there's inherent challenges, obviously, with like, you know, this this show not tell thing that we're talking about, um making the product more understandable, um bringing in new capabilities that matter more for for this cohort, the matter for developers, things like artifacts, things like computer use, etc. And then I think like the the same learning is like similarly how we took the learnings from developers and brought it to, you know, general knowledge work, the next stage will be like taking the learnings from general knowledge work and bringing it to everyone, no matter what they're doing in their lives. Um and we're already seeing that a little bit, like this this game example that you have is, you know, something that's it's like on the border of like fun and personal life to to, you know, your your professional life. I use ChatGPT at work full-time at home for everything, like for for whatever I'm doing. I used it the other day to come up with a meal plan and like, you know, save that on um on the like computer environment that it has and something that I can continue going back to. Like, is everyone doing that yet? Probably not because of things that we work on it, but eventually, you know, we want to get people there.

Host: ChatGPT life.

Reid: Yeah, exactly. ChatGPT cooking. Um but I think there's a lot of uh there's a lot of opportunity there, but I see it as like, you know, we're we're built we built the foundation in in in in software engineering and we're going to take this same learnings that we take from software engineering to knowledge work to knowledge work to to everyone.

Host: Do you have any power user advice? I feel like um there's a group of people that will live it, use it for everything, stay on it 24/7. Yeah. Um and then there's a bit of a gap between that crew and people that, you know, okay, I use it for work, I use it occasionally, sometimes I pipe questions. Uh any advice, any learnings, anything you recommend or just, you know, takeaways that you found that help bridge that gap?

Reid: I think a couple things that I've seen is like one that it really helps to broaden your imagination of what's possible. And this has been a learning even for me, like, you know, the technology has progressed so fast that, you know, there's something that like even 3 months ago like, no way, no way the models can do this. Like, now it's like, wow, it's like it actually can. Like, um We're going through right now that are like review cycle internally and you know people always talked about this as like kind of a a thing that the the models are good at like you know there's a cliche of like okay like I don't want to be writing reviews and like we just use AI to do it but I mean and then it needs to be evaluated as well. Yeah exactly. In all seriousness before it was like just like slop basically and like I I think it was helpful but you know not super productive. Now I found that like the model can do a much much better job than me especially in this environment of like pulling contacts on like what people are up to how they've like the things that they've done to make a difference highlighting like you know wins that they've had that like I might may not even have seen you know it has access to like everything right like the code like you know things that they've caught reviews slack everything and so it's like incredibly powerful in that domain and like just like six months ago the last time we did this cycle like I didn't even I tried using it but it was not at all helpful and this time it's been like incredibly helpful and like so I think continuing to push the the frontier of imagination what's possible even if you tried something before I think is maybe that my biggest piece of advice. The other I guess thing is like the more the more you put in especially in this environment where like you know the model has access to to everything on your on your computer or in chat GPT work like you can create you know artifacts over time and save them in your library and like the model continue having access to those like the more information you give it about whatever domain you're in whether it's your life or your work the more valuable it becomes and it'll become more valuable in like ways that might surprise you like it might pull from contacts in a way that you know maybe proactive in that you might not even have thought about but it needs to have access to those so that those tools or that context first.

Host: One thing I was just wanted to talk about the review stuff because I still that's a very sensitive thing and you're you're founders manage people you've hired people as manager myself I'm very reticent to put out any LLM generated things especially when it comes to people cuz it feels like you don't care. Presumably at Open AI people are obviously more open to being basically rated by GBT. Uh but are there any unofficial rules around this? Like what's the etiquette?

Reid: Oh, I mean I think the etiquette is that like I would never write something via like only via AI and like present it as like a review for someone. What I was talking about is more like gathering context.

Host: Yeah. That's the place where it's

Reid: So it's just search. It's agentic search.

Host: It's like agentic search, but you know, that you can tailor and steer much more capably than you could before. And cuz like the the thing is it's all there's sort of a flywheel happening, right? Because of Codex people are able to do them because of ChatGPT more people are able to do so much more now than ever before. And if you're able to do so much more, it's easy to miss things as well. Um and so like I think we need to use these same tools to keep up with all the impact that people are having. And and and understand, you know, where where we can be helpful.

Host: I think that the the thing like obviously I I run a small company, so easy to search. But at the scale of OpenAI, with the mono messages you guys put in Slack, do you think that it misses things?

Reid: Probably, but I think that I also miss things.

Host: Like it doesn't matter, right? Like it's it's as it needs to be human level.

Reid: relative, right? Yeah. Sometimes it's nice when it finds things you wouldn't, right? Like right now my Codex system prompts they're set up in such a way that every project I have has a secret separate notes MD. And it just writes learnings to there. And then the the global one can pull from all these. So sometimes it'll be like, "Oh, there's this project you did like 4 months ago. Here's a note that we had." And it it randomly pulls it back in the context that I would never do I haven't thought about. And I'm like, "Okay, this is quite superhuman, right? Like stuff that would And you know, it'll save

Speaker A: like hours on chunking of stuff or find something that's already been done. I'm like, as much as it might miss stuff I would do, but it's very useful when it finds stuff. And I have like a very, you know, non-super engineered solution to this. It's just markdown files that get pulled whenever they want.

Speaker B: Yeah, I actually have a funny anecdote about this. Like recently gearing up to this launch, you know, the team has been, you know, really cooking on it for for for a couple months and over that time like there's so much conversation and chatter going on in Slack and Docs and elsewhere. And uh one one of the members of the team set up this um scheduled task like automation to like look at everything that's going on and like come up with the best memes and then post in one of our shared channels. And like there are two cool things about this. Like the first is like I think the models are you know, over time like actually starting to become like funny. Whereas like you know, a year ago like that was not at all the case. The second is that it was what you were saying like they find things that in surprising ways that you may not have not have thought of and like create connections that you may not have thought of and that really helps with like the meme generation because then you can see something that you know, genuinely surprises you and then and um is funny in that way. So yeah, I mean obviously that's like not like the most productive uh use of this of this technology, but it does it doesn't cover this like this capability that's emerging which is just like defining information that you otherwise would not know.

Speaker B: Talking about the the launch, I I think uh I have pretty much said this is the most successful launch in a long time. I think even more successful personally than 5.0. And you're announcing 10 million users. Does it feel different? You've been through a lot of launches. I think it feels like a culmination.

Speaker A: Well, I think two things. One it feels like a culmination like I was mentioning earlier like this like vision vision that we've been on for a long time. Like I said, we saw the magic of Codex internally and then we're like extremely excited to bring this to many more people and to see it working to like see us reach you know, the distribution goal in um numbers that you mentioned like I think that's like huge and and and super exciting. The flip side of that is like there's so much more to do too. Like that's also really exciting. Like you know, ChatGPT as a whole like this product that you know, everyone almost equates to AI and like loves you know, has hundreds of millions of users. And so like 10 million is really cool, but like we we need to get this to everyone. Like we need everyone to feel this magic. Um and so that's the next step from here, but yeah, I think extremely pumped about how it how it's going so far and then the opportunities.

Speaker B: Uh awesome. Uh I did want to also, because I have I've I've been tracking the the number closely, it transitioned at some point from just Codex users to Codex plus ChatGPT work. Obviously, because the same harness, the whole point is that you don't you you can't uh comment separately. Do you have roughly a billion um ChatGPT users? When did it just jump to 1 billion right away? Like, isn't that the default on ChatGPT or no?

Speaker A: We don't default you into ChatGPT work if you're on ChatGPT or free. Yeah. It's also only available to it to paid users right now and I think there's like a process if you know educating users of what is the value of this product, having them try it, learning from their feedback, and making it better over time. But, I mean, the goal is to, you know, get as many of people who who love ChatGPT today to like feel the feel the power of ChatGPT work, but I think it'll be a a journey.

Speaker B: Yeah. And Codex will will still be alive as a brand for the foreseeable future.

Speaker A: Yeah.

Speaker B: Um and we'll just toggle between them as as as needed for UI stuff.

Speaker A: Yeah, I think it's even even stronger point than that. Like, I think we fully intend to like, you know, treat developer like developers have been, you know, core market for us for so long and like there's there's so much more that we can do to make Codex great specifically for um software development and we'll continue to do that. This doesn't take away from that at all. If anything, it should increase the utility of something like Codex because now like you can move seamlessly between writing a def to creating an artifact or, you know, doing a search over your code.

Speaker B: I do wonder how much this terminology leaks to the non-technical user. Like, do they have to learn to say artifacts if I want artifacts or, you know,

Speaker A: It's funny like we call artifacts internally cuz that's what the teams call them, but like externally like no one says that. No one calls it an artifact. I But, I think that people like often like describe things whatever they're used to, right? So, if, you know, ChatGPT work is good at creating slides, they'll say ChatGPT work is good at creating slides, and that's actually what we want.

Speaker B: One big another, I mean, it's July of 2026. One big thing that also happens in for opening AI was open claw, and that's I think a lot of people's first time really maxing agent for personal stuff, but also crossing over to work in in that sense same way. As far as as far as I understand, open claw is still independent, but did you go through your own open claw moments? Were there any lessons you took from open claw to Codex or back, whatever?

Speaker A: I think there's a lot of inspiration. I did go through my own open claw moment. I um

Speaker B: Yeah, tell the story.

Speaker A: Me and my um my my wife like set up an open claw to like try to manage everything in our house. Not that there's like a ton, but it was like actually quite useful. It gave it a calendar, started, you know, creating events for us and stuff. At some point the the laptop they were running on it died, and then they never got a chance to to pick it back up, but there's a lot of inspiration there. Like, um you know, in ChatGPT work in web and mobile, like you you get access to this like persistent computer environment where, you know, you can store files, and those files stay around between sessions. And the idea is to be able to enable use cases like this. Um one of the members of our team actually uses ChatGPT work for what they used open claw for them before, and I feel like it has like completely transitioned, which is like uh work out planning and like meal tracking, um which again, it's like a worky thing, right? It's like not work necessarily, but it's like in this personal productivity space. But it has all the same primitives. Like it has scheduled tasks. It has the ability to store files on a file system. It has the ability to like reference those things over time. Um and so you start to see the same types of use cases emerge, which has been really cool.

Speaker B: Is there a point that ChatGPT work completely replaces open claw? Obviously, they're independent, so

Speaker A: Yeah, I mean, I'm I'm I'm not close to it, so I can't speak to the the open claw road map, but I I don't think so. I think that there's going to be, you know, there's always a need for like this like incredible like open source technology that that team has built and I think that we can draw inspiration in the product and you know, chat GPT I think many more people have like heard about and used chat GPT than have you have used open claw and if we can take the magic from open claw and bring it to them, I think that'll be a success. I think that like one thing on the chat GPT work side that we feel strongly about is that like the core experience is that you come to this product and you have a conversation, start a session, whatever you want to call it with this agent and the magic of the product is that you can do anything in that moment and we would like to create a product where you don't have to click a button or to go to a different place, whatever and you can get whatever functionality exists in you know, your your finances app or or any other product like in this in this one place and so that's the goal. It's like it'd

Speaker A: We want an extensible system with plugins where you can connect to the tools that you need in order to be able to accomplish like a financial task where you can, you know, if you're doing like science work like we have an ability to like extend the system in such that you can like write the tech and and it performs well. There'll always be like products that we support that are best in class at those things, but we want as much of the magic as possible in that core experience, you know.

Speaker B: Do you think that you can do everything you used to do with Wolfram's in chat GPT finance?

Speaker A: I actually tried it. I mean, like chat GPT doesn't yet custody cash and and assets for you, so so that part no, not yet, but I I mean, there's like a whole component of like retirement planning and um sort of like financial planning and budgeting and stuff that that um we were looking into when I was there and like with the finances plugin like that's all possible with chat [laughter] GPT today, so um I feel like at least that component's replaced for me.

Speaker B: I haven't really plugged it in yet. Um I'm somewhat scared to look at the answer.

Speaker A: [laughter]

Speaker B: Like that's honestly like the same reason for health and finances. Like, I'm like, I don't know. I don't know.

Speaker A: [laughter]

Speaker A: It's really good. I mean, it's it's really cool how I mean, we were talking about like the agentic search aspect a little bit earlier, but like it's really cool how like, you know, in in conventional UX, like if the more power you want to give to a user, the more like knobs and bells and whistles you need to add. Like, you know, for like these finance and budgeting apps, like there's always like a bunch of the different filters and like search bars and stuff like that. But like now like with the right connect connectivity to the right data, you can have whatever you want. You can ask any question you want and and into that box and and get the answer and I think that's super powerful.

Speaker B: I think it's also nice to just have it centralized in one space, right? You have different health apps. I have one for a smart scale, a watch, all these different things. It's just nice to centrally collocate it.

Speaker A: Which is, you know, part of the whole thing of Open Claw, right? Like that that you would have a personal OS, which presumably ChatGPT wants to become. I do think that just relying on like just-in-time pulling of data for, let's say, through via MCP, CLI, API, whatever you whatever you do, still not enough. Like I I like I come from a bit bit of a data engineering background. Like you still want like a data warehouse or some kind of caching or semantic layer. Um, do you [snorts] feel that or do you already have that?

Speaker B: I can't speak to like all the details on how everything works, but I think it depends on the access pattern, right? Like, if you want to answer immediately, then yes, it's very difficult to do that if you need to pull from all of these sources. But, a lot of the like use cases that we want to enable on ChatGPT work aren't necessarily something that you need immediately. It's more like a task that you want the agent to go and do. And that that's going to take a certain amount of time. And, you know, with things like programmatic tool calling and stuff now, like some of that time and sub agents and stuff like some of that is also parallelizable. And so, it's possible I I I think it's very possible that there's a the ceiling on what can be done, you know, with MCPs and like calling out to these third party services has been raised substantially. So, we're really excited about that.

Speaker A: You mentioned sub agents. I I I got to double click on that. Ultra is a new mode. Um you have special affordances in ChatGPT itself to show off the the agents. Can't really do much with them to be honest. I just just just watch.

Speaker B: [laughter]

Speaker A: Um what have been your what have been your experiences? Any design issues that you would call out to other builders building with sub agents?

Speaker B: I think it's sort of goes back to the balance that I was raising earlier. Felt like, you know, showing builders the power of the tool, but also creating enough of an abstraction to not overwhelm them. I think with sub agents the thing that we wanted to show is that you can take a task that, you know, has many parallel tracks or um is is complicated in a way that, you know, sub agents can handle and this product is for you. Like the model can can accomplish those goals or try to accomplish those goals. Um and so like that's the point of like showing them in the product and and that's where we we've gone with the design. There's another, you know, iteration of this where like you can see exactly what they're doing and and things like that, which I think is like, you know, could could verge on like overwhelming um with information. And so this is like the double edged trade off that we made for now.

Speaker A: you you you do display quite a lot of transcripts.

Speaker B: Right. Right.

Speaker A: I think it's you want to display more than that?

Speaker B: No, no, it's fine.

Speaker A: Some some people could want more. So, I'm one of those people that will basically throw a lot of stuff at goal and pretty much every goal I'll tell it to use sub agents. Seems redundant, right? But every time I'm like, "Okay, use sub agents where possible." And I have a lot of people a lot of friends that recommend and do the same. Whereas I'll sometimes talk to people that are like, "Okay, this is where I want you to use sub agents for this sub task." And I'm sure they would appreciate seeing into how they're being used. For me it's primarily like two things, right? One is net time efficiency. So, span out across sub agents. Two is probably cost, right? Uh don't use big expensive model, offload to a lot of smaller, cheaper models, and some people want that level of control. So, if you have repetition in what you're doing, right? Say I want something built where I want it to consistently do this every day, I might want to go in and fine-tune subagents here, subagents there, so you can see both, but I think if I'm not mistaken, it's hidden by default. There's a drop-down that goes a lot where I'm like, "Okay, I'm just going to keep it on."

Speaker B: You can You can change the model that they use?

Speaker A: I know I tell them to be steered. I'll say my I know Anthropic offers this in Claude code. You can tell Fable to use Sonnet or Opus to use Sonnet as a subagent, so pretty trivial thing, you know, you tell it to span out subagents with Sonnet, you know, it's cheaper, faster. I would assume if it's not there, it could be built there, but I think there's a side of

Speaker B: Too many toggles.

Speaker A: Mhm. It's not a toggle, actually. It's just a You tell it in chat. The way I do it is prompt it, right? And I think this is something that gets abstracted unless it's something you built for repetition, right? So, if I'm building something, say that's a podcast prep, right? Research into people, do a very, very deep, extensive research, um that I might want to configure to cheaper, faster model just for web search, right? I can see a world in which you want both. I think the default is actually pretty good right now, where it's hidden, but you can drop down and get some more info into what's done. I know people talked a lot about it on 5.6's launch. Uh this thing loves to use a lot of subagents and causes the ChatGPT app to just crash because it's so processor heavy, but um

Speaker B: So, what is your personal experience? Yeah, I mean, yeah, you know, I haven't had it crash from subagents.

Speaker A: I I haven't either. I have We both have big laptops, but I know I know people brought it up. There was a topic of discussion that we didn't see the same, but it is another vibe eval, right? People are like, "Okay, the amount of subagents all this running is crazy." And I'm like, "I think this is okay. I think it's good, but just stuff people bring up.

Speaker B: I think when we launched the product too, we we weren't as opinionated about like who is Ultra 4 and like when when should they be using it and since then we made some changes to like, you know, require you to turn it on and and and find it in the advanced setting cuz that's who it is

Speaker A: for. It's for like power users who understand what's going to happen because it also, you know, depending on your use case can can use more of your your limits as well.

Speaker B: Yes.

Speaker A: So that's where I think a lot of the the feedback was coming for us.

Speaker B: Okay, reset the limits.

Speaker A: [laughter]

Speaker B: Always reset the limits.

Speaker A: Well, it's you know, today we're resetting because it is I want to change topics to one last piece of the harness, memory. A lot of people are commenting on memory recently. ChatGPT's new memory system used to suck is not very good. And then this guy also basically the same thing. And Samir who you presumably work with talking about memory. What can you say there?

Speaker B: I think that, you know, Samir and the team have made a ton of and and and the research teams have made a ton of updates and and improvements over time. I think when I talk to friends, family members about what they love about ChatGPT, like the fact that it knows them, that they feel like their ChatGPT is is their ChatGPT, I think comes up probably

Speaker A: Yeah.

Speaker B: number one. And ChatGPT work in the in the cloud like by default all conversations like inherent from ChatGPT memory so you'll know they'll know context about you. They'll also be able to write back to this memory.

Speaker A: With the like a like a like a small text right. Like you tell me when you're writing, right? Is it

Speaker B: No, it's part of the same like memory V3 system that that we we launched.

Speaker A: What do you mean V3? Yeah.

Speaker B: So I think that's been really powerful because, you know, going from ChatGPT to ChatGPT work feels like an extension of what I've already been doing with the product for sometimes many years. Um so that's been awesome and it's awesome to see that like people are recognizing um the the improvements here.

Speaker A: Is there So it's it's basically a retrieval problem, right? Like are you retrieving the right things? Are you over focusing on the wrong things? Is there like uh more false positive or false negative, you know, if if that makes sense? Like what's the bigger problem?

Speaker B: So, I don't work on memory directly, so it's hard to say what the bigger problem is with like certainty, but I I think you're right. I think that like, you know, the there's two sides of it. It's like, you know, making sure it knows things about you, but then also having the EQ to like bring those things up at the right moments, proactively or surprising you in ways that are positive, not negative. Um so, I think it's a very challenging problem, but something that I I think we feel very is a huge opportunity get right, which is like why we made like big investments in it.

Speaker A: How do you see the side of, okay, when you're building ChatGPT for work different than the regular chat app, different than Codex, managing memory across different projects, um collaboration and whatnot? How do you see the the side of what's separate from the harness, right? So, if I have four threads on one project, um any learnings on how to build memory systems there, you know?

Speaker B: For background as well, I guess, to steer it a bit is when you do chat style applications, I'd say you have a lot of one-offs, right? When you switch to work, it might be something you're doing for a month, something you do a lot, right? Now, as I add more sessions, there's a lot more than just single-threaded, right? And there there might be memory there.

Speaker B: I mean, I think first I'd challenge that like the depth of the memory or the like value of it is like fundamentally different across chat and work. Like, it it is true that like, you know, there are a lot of like shorter sessions on chat, but I think, you know, the ChatGPT the product has had like a ton of longevity, um and you know, as long as this this technology has been around, and and people use it for worky like productivity-related things already today. And so, I think we've found that there's a lot of value. I mean, I found this even in my personal usage, like all of these one-offs add up over time into something like quite durable and like like quite a good representation of who I am. I know like from time to time something will go viral on on X about like, you know, ChatGPT telling you everything it knows about you, and people are always surprised like how how deep that is.

Speaker A: The fun roast me, you know.

Speaker B: Exactly. So, like I think like the That's all to say that like I think there's a lot of depth there in the existing, you know, ChatGPT product, and so that's why I think we think it's valuable to bring into the work product. But, the other reason I brought that up is because I think, like, hopefully we can use some of the same fundamental primitives and systems to extend memory here as well. And I know this is something that the the team that focuses on this is like working working through right now.

Speaker A: I wanted to bring up one element of memory, which I honestly don't really use much, and I'm curious if you do, Chronicle, which was is up on screen right now. Uh it's kind of a super memory, or like what what is it?

Speaker B: [laughter]

Speaker B: I mean, the idea is that, like, it can learn from, you know, how you're using your computer, and like it's an another input source um into memory. And um I think it's, you know, experimental right now, and something that, like, isn't default off, but I'd recommend that you try. I think that it's like quite interesting how it goes back to a conversation we were having earlier on, like, you know, you were asking, like, does it Can ChatGPT miss things? Like, does it, you know, on Slack, when it's searching, does it miss things? Cuz there's such a volume of stuff, right? And like it's I You can ask the same question about, like, everything that you're doing on your computer. Like, is it going to know everything that you're doing? Is going to capture the intent and stuff like that? Probably not, but like it probably will find things that you might not know about. And then, if it can surface those to you in relevant times in proactive ways, like when you're doing tasks, then I found, at least, that it can be quite helpful. So, it's worth trying.

Speaker A: So, mostly for insights and longer term?

Speaker B: Yeah, exactly. Like, insights and it builds context that that makes that can make you more productive on certain tasks, but it's it's it's hard to describe without feeling it. Um

Speaker A: [laughter]

Speaker A: I will say you can feel it pretty well. Like, the idea of what they're saying here, right? Just check through my memories, or check through my logs, and add skills.

Speaker B: Yeah.

Speaker A: Pretty underrated, right?

Speaker B: But, that's that's automations. You you can repeat that using a cron job.

Speaker A: Checking through your memories and creating skills.

Speaker B: Yeah.

Speaker A: But I think the creation of the memories from Chronicle itself is like what's different. It's like you have much deeper memories because you have Chronicle on.

Speaker B: It's there. I don't use it much, but maybe I just I need more examples. I I imagine you guys use a lot of it internally, so I'm always fishing for use cases.

Speaker A: Yeah, I would just try turning it on and then like it just auto works. Like it

Speaker B: Yeah, and seeing like where where it might start helping you. I think you'll be surprised.

Speaker A: Yeah, amazing.

Speaker A: I think that was the about it in terms of like the the overall coverage of ChatGPT work. I think there's been a lot of like good progress and discussion on building and all these things. There's a lot of like ex-founders in the in the community and in OpenAI as well. Do you think that things have changed a lot? I I guess like your overall reflection of building pre-AI and post-AI.

Speaker B: I mean, I think things have changed a ton. I think it's it's like super exciting to see how quickly you can go to from idea to something real today. Yeah. Um whereas like even before like I think you know, 5 10 years ago like it it was fast if you were scrappy and you know, like willing to build the the minimal viable thing, but like now the extent of what you can build is like much much broader. And I think that also like what we've seen internally building is like that gives you an opportunity to validate much more quickly, to talk to users, to talk to to internal doctors, etc. and like make sure you're on the right track.

Speaker A: And like that loop I think has been has become more closed than ever before and that's like a win for product development. I think it's a win for for consumers and users too cuz ideally that means they're getting much more better much better products out the gate.

Speaker B: Does it mean your team's a smaller?

Speaker A: I think there's much more to do now. Um so I think people can accomplish more individually or in a small team than they were that would require more people within before, but there's at the same time there's also more to do. So I think the teams are much more ambitious.

Speaker B: You seen any changes in scopes of roles and building teams and how we used to have teams say a few years ago versus what ideal teams look like now?

Speaker A: I think we've seen a blurring in the lines between like the typical product development functions like between like you know, EM, PM, engineer, um designer, etc. like

Speaker B: Yeah, when I bring up this quote, there will be uh only four jobs left in tech. There's AI uh slop cannon, the people who just like uh they'll burn a bunch of tokens uh and then there is there is SRE uh who people who are more responsible. There's grown-ups who sell things and then there's hot people.

Speaker A: [laughter] It's an interesting take. I think my my suspicion is that there's everything everyone will be like T-shaped in a way and not like AI will enable everyone to become a generalist like you know things that like I I never would be able to like come up with a design before and like even now I don't have maybe like the visual taste required but I can iterate on something with the help of AI. But then people will have a specialty and that's like the the like it's a straight line in the T or the the upward line in the T. Um and so like you have a specialty that you're interested in with the help of AI you can go deeper and become better at over time but then you'll also be a generalist and so with that foundation the what you can accomplish is like almost limitless.

Speaker B: What are you bottlenecked by in terms of specialties? Like do you need more designers? Do you need more slop cannons? Do you need more hot people?

Speaker A: I think the bottleneck some becomes like sort of like ideas and taste I guess. Um I think because anyone can can build now, I think um it really is the era of like bottoms-up ambition and because there's so much to be built, like you're always going to be bottlenecked by you know the amount of ideas and amount of things that you're doing at any given time.

Speaker B: Do you think models help solve that?

Speaker A: Models?

Speaker B: Yeah. I mean I have the example of like I have a front end design skill that's like they give me four drastically different examples of what this looks like. You sure it burns a lot of tokens, but you know and then I'll mostly just condense down. Okay, I like this part, I like this part. Let's draw these together and it's like yeah, I had a vision, but like I don't know. I would say that the one automation that I would love to work and it doesn't work is bring me new ideas, right? Uh somehow LLMs are just not it. One interesting part about ideas is like they're not like in a vacuum. It's like not they they usually come from somewhere and like you know in in product development like they're coming from talking to users or reacting to you know friction that you're seeing or feedback building on some foundation that you already had planned out before whatever. Um and so I think that's where like I think there there always be value in in these like generalists that we talked about like you know closing that loop and and and and having coming up with those ideas that are grounded in in that feedback or talking to users or whatever it is.

Speaker B: Cool. Uh you were going to you lead the productivity team. How do you define productivity?

Speaker A: I think our mission is to make it possible for people to do things that they weren't able to do before. And right now we're thinking about it from the the perspective of knowledge work. And so when looking at knowledge work I think about people are no longer siloed by their roles, they're no longer siloed by maybe the the um background or training that they have. Like no matter what function you're in you can suddenly build things and suddenly get access to data that you otherwise might not be able to interpret etc. Um and then I think that extends to your personal life where we want to give you leverage at the end of the day. Like we want the models in the product to be able to give you leverage so that you can you know create time for yourself to do the things that you love.

Speaker B: Does that also translate to a way to measure productivity? Like what is how do you measure leverage?

Speaker A: I think we haven't figured this out yet. Part of the reason is it's so diverse. Everyone has different goals and really the true measurement is like their ability to achieve that goal. Did we help you or did we not? Yeah. And it's very difficult without knowing what that goal is up front and also tailoring it for every individual.

Speaker B: And the thumbs up and thumbs down from ChatGPT doesn't give you anything, right?

Speaker A: Right. I mean, you don't know if they're thumbs downing the the content of the answer, the vibe of it, whether or not it helped them with their goal. I think that's difficult. Um but it's something that I think we will need to figure out and the industry at large will need to figure out because, you know, that's how we we measure success if this is what we're for.

Speaker B: Do you think it's changed productivity and how you measure it? Basically, you said there's a lot more work that can be done, a lot more scope. Um has it changed?

Speaker A: I think it was always true that what you really wanted to measure is like you know, was your team, was the individual, were you personally able to hit the goal or are you closer to hitting that whatever your goal is, right? But I think previously we used proxies for this. So like, you know, code commits or lines of code or whatever. Um

Speaker B: Story points.

Speaker A: Yeah, exactly. Story points. And like

Speaker B: Coming back, by the way.

Speaker A: [laughter] May maybe. I mean, but but that is for part of the change and like I think with with AI now, those proxies starting to fall apart. Like you you know, the number of tokens you use or the number of pull requests you make or like no longer like maybe as hyper correlated with the that is your team able to hit the goal or are they on track to hit their goals. So I think we'll need to come up with with new um measurements.

Speaker B: For the managers listening, uh give them one thing to try.

Speaker A: I think for me, what's important is like at-bats. Are we as a team building the muscle to have not just quantity of at-bats, but quality? Like, are we able to go all the way from like generating an idea, building it out, getting the feedback, reacting to that feedback, actually validating or invalidating the hypothesis, going on to the next idea? Are we able to do that really efficiently? Like that goes to like, you know, the actual like code that's being written or the designs that are being made or the specs that are being written whatever, but also the culture of the team. Like do we have the humility and and and and um are able to like go through that process many many times and stay motivated and excited throughout that. Um so that's the thing that like I think is important now, especially when we're on the frontier of this technology and like there's so much to build, there's so much to do. That's probably the most most important thing that we look at.

Speaker B: Any traps people fall into around measuring productivity with your team work on? I feel like there's a lot of okay, we added a lot of LLMs, we have dashboards for this and that, but not much has changed, right?

Speaker A: [laughter] That is the trap, yes.

Speaker B: And you know, the broader source of the question is for for the managers and teams building, you know, how how should they approach this?

Speaker A: I think maybe the trap is like conflating motion and progress. I think motion is much easier now than ever before because of the tooling that we have. But progress requires you to be like very prescriptive and deliberate about like what you're

Host: Actually, that's what we're trying to achieve. And it goes back to our question of measurement, right? Like you were we were talking about like can we OpenAI like figure out how to measure productivity for our users? That's that's a very hard problem because of the diversity, but like as a team like you should have a really prescriptive and deliberate view on like what progress looks like for you and for your team. And if you don't have that, then it's very easy to conflate these two things. I think that batch is a really great thing. I'm I'm really glad I I like the discussion between motion and progress. I think that's a quote that we're going to feature in the write-up. You've been very generous with your time. Thank you so much and congrats on 10 million.

Guest: Yeah, thank you for having me.

Host: The next to a next one at 100. In two months.

Guest: [laughter]

Host: Thank you.