面向 Agent 时代的推理云重构：从单次请求延迟到端到端任务吞吐

Let's get started. Hi everyone. Thank you for coming. This is the late afternoon in the last day, so I really appreciate it. I'm Gon, founder and CEO of FriendliAI. Today I want to talk about agentic inference. I'll first walk through what changed, why it matters, and how we rebuilt the inference cloud for agents.

Before we go deeper, let me briefly introduce FriendliAI. FriendliAI is the frontier AI inference cloud for agents. We run inference for agents at scale—faster, cheaper, and more reliably. We are born from a research team at Seoul National University, and those research roots still define us. We are the team that invented continuous batching, the inference optimization that is now standard across the industry, and our Orca work inspired vLLM, a widely used open-source framework. Today we operate globally, headquartered in San Francisco with a team in Seoul to scale frontier inference.

As you know, 2026 is the year agents go into massive production, and it's driven by two trends coming together. First, agents are going exponential: AI agents are driving explosive adoption across software operations and knowledge work. Second, open-weight models have reached the frontier and make agents economic. They now rival closed frontier models in capability, which means you can run frontier quality agents on open models with much lower token cost.

Let me make the open-weight model part concrete. Open-weight models are now strong enough for these types of real agentic workflows. Here we gave the exact same task—building a tower defense game with a coding agent—to two models. On the left is GLM-5.2, an open-weight model running on FriendliAI; on the right is Anthropic's Opus 4.8. The important point is not that the outputs are identical; the point is that both complete the task at a level that is clearly usable for many agentic workflows. Open-weight models have crossed the quality threshold, but the economics are very different for the same task: Opus 4.8 cost about $150, while GLM-5.2 on FriendliAI cost 27 cents—about 5.6 times cheaper. This is the promise I mentioned earlier: open-weight models give you frontier quality agents at a fraction of the cost.

But model cost is only one part of the story. To make agents actually fast and reliable, the inference stack itself has to change. Let's look at what actually happens inside an agentic workload.

First, let's look at changes in the workload. In the past, the dominant usage was chat. The basic unit was a request: a person asks a question, the model answers, and the person reads it. Latency meant how fast did I get one response. Agents are different. The basic unit is a task. A task may involve many model calls, many tool calls, and it may run autonomously for a while. So the user does not really care about the latency of one individual request; the user cares about when the whole task is completed. That means we have to optimize for tasks, not just individual requests.

Let's look at agent workflows more closely. An agent really runs a session made up of tasks. Each task typically runs in a loop: first it plans (which usually means an LLM call), then it acts (maybe by calling a tool), then it observes the result and adds that back into the context, and it repeats this until the task is done. So we are constantly alternating between LLM inference and one or more non-LLM tool executions, creating a gap between LLM calls. An agent can also create sub-agents and run them in parallel.

Agent inputs also look very different from chat. The prompt and completion length distributions of our internal coding agent runs with GLM-5.2 (which we use day-to-day) show they are much longer. They grow as the task progresses since every observation gets appended back into the context. There's an important pattern here: consecutive agent steps usually share a huge prefix. If we recompute the same prefix every time, we are burning a lot of compute on work we already did. This is one of the biggest opportunities in agentic inference.

How token-hungry are agents? Consider a long-horizon task example like Deep Research: running a task like "explain the speculative decoding framework in vLLM using code" with GLM-5.2 on FriendliAI involves multiple stages, each composed of sub-agents running multiple inferences and tool calls. It might run tens or hundreds of inference steps over minutes or hours, while the shared context keeps growing the whole time. For the user, what matters is not the latency of a single token or one call, but when the task is completed. Agent inference is not just chat with more requests; it's a different problem: context grows over time, tool work is interleaved between model calls, the number of calls depends dynamically on input (preventing fixed-rate capacity planning), and the real metric is end-to-end task latency.

This is where FriendliAI comes in. We rebuilt the frontier inference cloud specifically for agentic workflows around the challenges I just walked through, and we set one goal: optimize end-to-end task latency—the task, not just the request.

Here is the engineering map for how we think about it. We built the stack layer by layer around agent workflows. There are four big pillars I'm going to cover today: prefix caching, KV cache management, cache-aware routing, and agent-aware optimization (alongside underlying model-layer optimizations like sparse attention for long context, fast kernels, and resilient serving).

Let's start with prefix caching. Since agent steps share a large prefix, we compute key-value for that prefix once and cache it. Then on later steps, we reuse the cached key-value and only process the new suffix. Reading from cache is much cheaper than recomputing prefill, which improves time to first token and reduces compute on every step. The longer the task runs in agents, the more valuable this becomes.

But caching only works if the KV cache actually fits and can move around efficiently. We need strong KV cache management: we use frugal memory management to pack more active context onto each GPU, KV compaction/quantization to reduce footprint, hierarchical caching across GPU memory, host memory, and disks to go beyond GPU limits, and distributed caching so one prefix can be served across replicas rather than just inside one instance.

At global cluster scale, routing becomes critical. A naive load balancer may spread requests evenly across GPU clusters, but it destroys cache locality. A cache-aware router at global scale does something smarter: it routes a request to a pod that already has the right prefix cached, turning a cold prefill into a cache hit, while continuously balancing load to prevent hot spots.

The next piece is agent-aware optimization—the next frontier of agentic inference. Today, most systems schedule each LLM call as if it were independent, without understanding that it is part of a longer agent program. When the optimizer knows agent-level context, we can make better decisions: preempting the right work, speculatively prefilling context for likely next steps, or making better cache eviction decisions based on agent context to reduce end-to-end latency.

When we put all of this together, this is the payoff: using the same model GLM-5.2 with Kilo Code to create a simple mobile game across FriendliAI and another well-known inference provider, FriendliAI completes the same task end-to-end much faster thanks to our agent-centric cloud design.

What does this unlock in practice? A stronger production agent stack. Take an agent you already like, plug in open-weight frontier models like GLM-5.2, MiniMax, and Kimi served on FriendliAI: the model gives you frontier capability and better economics, while FriendliAI gives you the speed, reliability, and end-to-end task performance needed in production. That combination of quality, speed, reliability, and cost makes agents truly viable.

FriendliAI powers teams from AI-native startups to global enterprises. For example, Kilo is a hugely popular agentic AI coding tool serving millions of users, and LG is a global enterprise spanning electronics, healthcare, and energy. Over the past year, Kilo Code tested several providers; in a split test of GLM-5 usage against third-party providers and direct usage from the model lab, FriendliAI was consistently 7x faster with a significantly lower error rate, making it a core component of Kilo's stack.

You can consume this stack in three ways:

1. Model API: The fastest way to start via serverless frontier open-weight endpoints.
2. Dedicated Endpoints: Isolated deployments with guaranteed SLAs for production workloads.
3. BYOG (Bring Your Own GPU): Run Friendli inference on your own infrastructure.

To wrap up, remember three things: first, frontier open-weight models make production agents economically scalable; second, agentic inference requires optimizing end-to-end task latency rather than single requests; third, FriendliAI is built specifically as the inference cloud to deliver fast, reliable, and cost-effective agentic inference. You can get started at friendli.ai in minutes. Thank you.