OpenAI全新Agent架构剖析：Computer Use、Decisions API与超低延迟推理

Host: Computers today are getting faster at completing tasks, and for average humans, that might often be enough. But I think the next frontier challenge is for computers to provide superhuman service—running software just as fast, or even faster, than experienced power users like us. If that happens, it will be profoundly significant and exciting, because we will suddenly be able to build products that offer true real-time experiences.

Host: Just four weeks ago, this was not the case. But recent breakthroughs energized us. OpenAI still maintains an incredibly intense hacker culture where people are deeply passionate about everything. When this new reasoning type emerged, someone from the infrastructure team said, "This is insane. We have to be there and hack on this." They built a functional prototype, and now the focus is entirely on slashing latency and bringing it to market as quickly as possible.

Host: Welcome to this OpenAI DevDay special edition podcast. We have Ari here, who leads product for the computer-using agent team. Ari, could you briefly summarize what was announced today regarding Computer Use?

Ari Weinstein: Yes, it has been absolutely thrilling! We just finished the keynote announcements. The fundamental difference from traditional approaches or other products is that instead of just getting a sandboxed cloud browser, you now receive a dedicated, full Linux computer in the cloud.

Ari Weinstein: This means the agent can execute complete desktop applications alongside a web browser. The immense power of Computer Use—and what excites me most—is that virtually all software in the world was built for human interfaces. Now, agents can utilize the exact same software stack, allowing you to delegate virtually any task on your computer.

Host: It truly opens up endless possibilities. People spend hours on repetitive digital chores. Delegating flight bookings, online shopping, or even routine gaming and data tasks to an agent can save enormous personal bandwidth.

Ari Weinstein: Exactly. A personal example: I subscribe to a meal preparation delivery service to manage my diet. Adjusting the exact grams of chicken and carbs across meals used to take me two hours of tedious clicking because of their complex interface. I finally had a computer-using agent handle it; it finished the entire task in 15 minutes. It saved me two hours each time, performed it eight times, and was faster and more accurate than me doing it manually.

Host: As a content creator, my number-one use case is automating YouTube creator workflows. YouTube offers very limited APIs for things like multi-variant thumbnail A/B testing or community tab interactions. Without public APIs, having an agent operate the web UI directly in a virtual environment bridges that massive gap.

Host: Some popular AI podcasts argued that computer use hadn't made meaningful progress over the last two years. Having spent years in this domain—from building Workflow, joining Apple, founding Sky, and now leading this at OpenAI—how do you see the evolution? What drove this massive leap?

Ari Weinstein: I believe anyone claiming stagnation would change their perspective 180 degrees seeing where we are today. Early models could initiate tasks but failed brittlely whenever an unexpected modal or latency hiccup appeared. They lacked self-debugging capabilities.

Ari Weinstein: Over the past year, models developed profound debugging capabilities. When something fails, they can diagnose why and try alternative strategies. Furthermore, instead of sequential single-action mouse clicks, the agent can write and execute JavaScript and Playwright scripts directly in the environment, bundling multi-step actions into a single cycle, which dramatically accelerates execution.

Host: The keynote highlighted a 7x speedup and major benchmark leaps. Internally, how do you measure these improvements? Is the gain primarily coming from the model itself, or from the harness surrounding it?

Ari Weinstein: It's a combination of both. In our benchmarks, we see steady compounding gains from both the raw model intelligence and the surrounding harness architecture. The Pareto frontier has shifted dramatically, offering substantially higher success rates at significantly reduced token cost.

Ari Weinstein: A major breakthrough was moving beyond pure pixel screenshots. Previously, an agent had to screenshot, inspect, scroll, screenshot again, and repeat. By integrating Accessibility APIs and direct DOM tree access, the model perceives the full structured context of an application alongside visual cues, eliminating massive amounts of latency.

Host: That resembles the AppShots feature in Codex and the macOS desktop app when you double-press Command. Users often assume it's a basic screenshot, but under the hood, it captures the entire accessibility tree and structural metadata, right?

Ari Weinstein: Exactly. A visual screenshot alone cannot reveal target URLs of buttons or clipped text in a calendar invite. AppShots extracts the structural accessibility representation. We spent extensive engineering effort optimizing token efficiency so that this rich context fits compactly into the model's prompt.

Host: Looking ahead two to three years, where are the primary bottlenecks holding back fully autonomous computer agents? Is it model intelligence, or the physical latency of environments and application loading?

Ari Weinstein: As models become faster, the execution environment increasingly becomes the bottleneck. In tasks like ordering food on DoorDash, the agent spends most of its time waiting for web pages and network assets to load. Shifting from synchronous polling to event-driven architectures where the agent reacts instantly to DOM mutation events is crucial for closing this gap.

Host: For developers building on this newly released API, what pitfalls should they avoid? How should they balance expansive agent capabilities with safety boundaries and user trust?

Ari Weinstein: First, maintain human-in-the-loop safeguards for high-stakes actions like financial transactions or destructive data modifications. Second, enforce least-privilege scoping: sandbox the agent's environment and restrict network access strictly to the domain whitelist required for the designated task.

Ari Weinstein: One of my absolute favorite use cases is closed-loop software engineering. Previously, engineers generated code with Codex and manually tested the UI. Now, a coding agent can build a feature, while a computer-using agent launches the application, runs end-to-end tests, inspects UI regressions, and reports back fixes, completely automating the development feedback loop.

Host: Thank you so much, Ari! Next, let's welcome Nikunj Handa, Head of API Products at OpenAI. Nikunj, could you introduce yourself and walk us through the major architectural advancements introduced in the API today?

Nikunj Handa: Great to be here! I'm Nikunj and I lead the API product team at OpenAI. Our mission is to bridge post-training research breakthroughs into robust, low-latency, scalable developer primitives.

Nikunj Handa: One of the biggest architectural upgrades is native asynchronous tool calling and mid-stream steering. Previously, function calling was strictly synchronous and blocking. Now, models can dispatch long-running asynchronous tasks while continuing reasoning, and developers can inject steering instructions into the active trajectory to correct or pivot agent behavior dynamically.

Host: This must require major communication protocol changes. Is this built entirely on bidirectional WebSockets, and how does it relate to the ultrafast reasoning capabilities being discussed?

Nikunj Handa: Exactly. We leveraged our WebSocket foundation from real-time audio to create a unified duplex transport for agentic APIs. Full-duplex connections allow bidirectional streaming of tool outputs, interrupts, and steering tokens with minimal transport overhead.

Nikunj Handa: This becomes indispensable when paired with our ultrafast inference stacks. By aggressively optimizing Time To First Token (TTFT), the model can produce tactical decisions almost instantly, matching human conversational and operational cadence.

Host: Speaking of speed, let's address the elephant in the room: the Decisions API. The community has seen a massive surge of interest around fast decision models. What was the genesis of Decisions API, and is it a completely new foundation model?

Nikunj Handa: Credit to the open-source community for pioneering incredible ideas in this space. Internally, our customer operations team needed ultra-low-latency ticket triage. Our infrastructure engineers realized we could optimize our frontier reasoning weights to serve rapid, deterministic classification tasks at breakthrough speeds.

Nikunj Handa: To clarify: version one is not a ground-up pre-trained model. It leverages our proven frontier weights, but strips away decoding overhead and heavily optimizes the inference stack specifically for structured outputs. This delivers deterministic categorization decisions within tens of milliseconds.

Host: How should developers architect solutions using Decisions API alongside the higher-level Agency API? What are the prominent internal use cases at OpenAI?

Nikunj Handa: The most immediate application is high-throughput routing, automated triage, and real-time safety classification. Traditionally, using an LLM as a judge incurred significant latency and token costs. Decisions API turns real-time moderation and arbitration into a zero-latency utility.

Nikunj Handa: Furthermore, our newly released Agency API serves as the underlying backbone for OpenAI's own first-party agent products, including our interactive meeting assistants and collaborative editing tools. It natively encapsulates long-term thread state, sandboxed runtime management, and tool routing.

Host: Agents inherently generate massive historical context over prolonged trajectories. How does the Responses API evolve its Prompt Caching infrastructure to mitigate cost and latency?

Nikunj Handa: Prompt caching is existential for long-running agents. We completely re-architected the caching layer in Responses API. Beyond our standard 30-minute transient cache, we are introducing an extended 12-hour cache SLA for enterprise workloads.

Nikunj Handa: We also added a Cache Warmup API. If you anticipate a traffic burst or an intensive agent session, you can pre-warm the prompt cache ahead of time. Coupled with our new efficient model variants, cache read costs drop by up to 95%, making large-context deployments economically viable.

Host: Even with million-token context windows, long-horizon agents running over days will eventually exhaust context limits. What is the current state-of-the-art methodology for context compression?

Nikunj Handa: We support two primary paradigms. The first is server-managed automatic compaction: when the trajectory nears context thresholds, Responses API automatically executes background semantic pruning and summarization transparently.

Nikunj Handa: The second approach is manual compaction via explicit endpoints like /compact, mirroring the design in our open-source Codex-Harness. Advanced engineering organizations often prefer programmatic control over compaction policies, preserving key architectural state while pruning redundant trajectory history.

Host: Looking at the broader picture, this is not just about isolated model releases. It aligns with the long-term vision: OpenAI is methodically constructing an AI-native cloud computing platform.

Nikunj Handa: Exactly. Just as AWS pioneered EC2 and S3 for internet-scale compute and storage, we are defining the foundational primitives for the agentic era—Decisions API for instantaneous control flow, Responses for duplex streaming, and hosted Computer Use environments. We invite developers worldwide to build this future alongside us.