--- author: AI Engineer date: '2026-10-14' guest: '' layout: post.njk source: https://www.youtube.com/watch?v=tZycGA_Fbfo speaker: AI Engineer tags: - ai-agent - model-context-protocol - enterprise-architecture - observability - context-optimization title: 从实验到生产：企业级 AI Agent 的全链路治理与工程化落地 summary: Intuit Credit Karma 资深架构师 Harish Gaggar 深入拆解如何将 AI Agent 从实验玩具推向严苛的企业生产环境。演讲系统化阐述了由 MCP 网关、定制化 MCP 服务、全链路监控可观测性、状态化内存管理、分布式追踪评测、PII 隐私防护前置以及上下文优化器构成的七大核心治理支柱，彻底解决 Agent 生产落地的安全、成本与确定性瓶颈。 insight: '' draft: true series: '' category: ai-application area: tech-engineering project: [] people: [] companies_orgs: - Intuit - Credit Karma products_models: - LangGraph media_books: [] status: evergreen --- ### 生产陷阱与基础治理支柱 AI Agent 在企业生产环境中面临严峻的工程化考验，许多原型实验在脱离沙盒进入真实业务场景时纷纷崩溃，其核心根源在于底层基础设施控制机制的缺失。企业级环境绝非简单的模型推理接口调用，它要求严格的权限验证、可控的工具调用边界、透明的链路审计以及端到端的质量评估。若缺乏统一的**网关治理层**（Gateway Layer），Agent 往往直接无序调用下游服务，导致身份认证断裂、权限失控以及工具接口滥用；同时，缺乏系统级的调用追踪与质量评测体系，使得团队无法洞察每次推理的准确性与异常表现。为了破解这些生产瓶颈，必须构建一套系统化的工程化框架，覆盖统一的访问入口、定制化协议服务、全生命周期评测、长短期记忆管理、前置隐私安全拦截以及计算成本压降机制。
Original English Source Hello everyone. Um, I would like to walk you through that how we can transition in AI agents from an experiments to the real enterprise environment. So, before I begin, a quick introduction about me. My name is Sharish Gokar. I'm a senior staff software engineer at Intuit Credit Karma where I built and own the AI analytics infrastructure and services. Um, and beside that, um, I'm advisor, uh, author, researchers, um, and also a program committee members for various organizations where I mentor the young researchers as well. All right. So, let's, uh, walk through the, um, agenda here. So, first we would like to talk about what are the challenges we have when we build an AI agents and deploy into the production systems and how we, uh, can build a different framework to address that, uh, different challenges here. Um, then how the architecture looks like for it. Um, then, um, how you can build your own MCP servers, um, and MCP gateway. And then some monitoring on top of that gateway as well. Um, then further we're going to talk about what are the different type of agent framework you can have, um, and then, um, some additional framework around you know like managing the memories, tracking and eval, um, how you can block the PII data, uh, going before the LLMs, and then context optimizer to save your token, right? So, without delay, let's, um, let me walk you through the challenge. Um, of course, like there are many agents, uh, uh, generally get failed in productions because their foundation controls are missing. Um, and the key reason behind is that, um, you know, like, oh, most of the people are missing the gateway to make sure that agent can, you know, communicate to the gateway to get authenticated first with the right auth providers so that they can redirect to the right MCP servers, right? Second, that, um, MCP servers are missing the right tool to execute your query. Third, do we have like enough tracking and evaluation system to make sure that every query when you are sending to the agents, whether they've been tracked or not, and are you evaluating the quality response or not, right? Do we have any in memory management system or not in terms of let's say you are having a conversation with an agent, whether you've been, are you passing the right context back to the LLM or not? And then there are other framework, of course, like do you have a context of the optimizer framework in place? So, all those challenges generally, you know, like leads to an agent gets failed in a production systems. So, in order to resolve this, we are not going to, you know, like build a magical framework, but we are going to build a seven different, um, you know, like a framework which address that particular problems.
### 企业级 Agent 全生命周期架构 在确立上述治理支柱后，具体的落地需要一套端到端的企业级生命周期架构来串联从开发到生产的各个环节。首先，团队通过内部开发者门户（Developer Portal）发起脚手架构建，标准化配置所需的插件、预置提示词模板（Prompt Templates）以及专有的执行节点；随后平台自动生成包含所有依赖框架的代码仓库，并在本地容器中完成构建与基础验证。通过与持续集成与持续交付流水线（**CI/CD Pipeline**）对接，容器化制品被无缝推进至生产集群，在此阶段接入生产级的关系型数据库与存储服务，分别承载长短期对话记忆、执行追踪日志与安全凭证管理。当 Agent 实例在生产环境被终端用户或客户端触发时，请求并非直连大模型，而是依次经过**隐私信息过滤层**（PII Filtering Layer）剥离敏感凭证、**上下文优化器**（Context Optimizer）精简 Prompt 负载，再由底层的编排调度系统统一转发至路由治理中枢，由此奠定了高度解耦、可扩展的企业生产拓扑。
Original English Source The first part of that is the agentic framework. So, you can choose either one of them, but in our case, we are using LangGraph. This is help us in building the task-specific node based on your questions, and which is easy to plug in any like marketplace plugins or skills and prompts to make sure that agents respond efficiently based on your query. Further, the second part is we are also building the MCP gateway to make sure that whatever the request has been submitted to the agents goes to the MCP gateway to get properly authenticated. Third is the customized MCP, which means that you are building your own MCP servers to make sure that right tool can be executed for the right namespace. Fourth is the MCP gateway monitoring, which means that continuous evaluations, logging, and metrics can be extracted, and we can take a right action on the real-time systems. And then we talk about some additional framework around you know like how you can manage memory, evaluation, PII data blockers, and context optimizer. Now, let's look at the enterprise life cycle here. So, you are here as a developer, you come in, you go to your developer portal, and you define what plugins, what skills, what prompt you want, and what node you want to define for your task-specific agents. Once you submit this, the dev portal is going to create a repository, which is going to add all those required frameworks here, which is like your core engine, your tool libraries, and then your context optimizer framework, and your PII blocking framework as well. Now, as a developer, you can start building, you know, your application, test your application, and you can spin up your local Docker containers. Once you feel everything is working fine, developers can also do local testing using, let's say, MCP servers locally. And they can also traces on their local memory, maybe cloud span evaluator here. And you can also store this tracking and eval in some of your DB, maybe SQLite or something, and then check all the log locally. Once everything looks fine, you submit your PR, raise your PR, get it reviewed, and then push to the main branch. And then the CICD pipeline is basically, you know, whatever the local Docker container you built here is going to build that container into production system. And at the same time, the infra code, which is going to provision all the production-ready frameworks, which is like your RDS databases for your long-term memory, which is Postgres in this case. And then memory DB or tracking and eval tables and then agent secret management, KMS keys, everything has been provisioned through the CICD pipelines. Once everything is done, the Docker container gets deployed on top of EKS cluster. Once everything in place, um agent is further going to communicate to the rest of the systems, which is client will communicate to agent through Slack or web interface. And the agent is basically using that context optimizer framework, PII blocker framework, LangSmith, all this framework will be plugged in, and which is going to interact with your LLMs and then MCP gateway to make sure that everything has been secure, efficient, and cost-effective.
### MCP 服务协议与网关安全路由 针对工具执行的碎片化问题，**模型上下文协议**（Model Context Protocol: Anthropic 提出的标准化 AI 与外部系统通信规范）为异构工具集成提供了标准底座。在企业实现中，开发者既可以利用开源社区生态，更需要自主构建符合内部合规规范的定制化 MCP 服务端。在定制服务端内部，每一项工具声明都必须明确界定其能力边界、入参模式（Schema）以及环境隔离属性，特别对于写操作（Write/Update/Delete）必须施加额外的权限锁控。为了防止客户端与模型直连后端工具造成的越权风险，必须建立统一的 **MCP 网关**（MCP Gateway）。客户端发起调用时，首先向网关申请一次性 **JWT 令牌**（JSON Web Token: 包含用户身份与授权作用域的加密安全凭证）；随后，携带命名空间（Namespace）与工具路径的调用请求发送至网关，网关完成解密鉴权后，精准路由至对应集群内部的 MCP 服务端执行，从而在物理与逻辑层面将企业核心资产与外部交互彻底隔离。
Original English Source So, now firstly, we talk about the MCP server as well. So, we talk about architecture. Now, the next is the MCP server. So, MCP server is something that you can definitely use something which is outside available in the open source, but at the same time, you can build your own customized MCP servers, which can be useful based on your organizational policy. So, the way the framework has been built here is that uh you you can leverage the Anthropic MCP framework. And then you make sure that, you know, like every tool when you define in MCP server, whether it is a read-only or read-write, that should have a proper schema definitions and clear purpose of what tools do. Um and third part is like a of course, you need to make sure that you build your own custom plugins, let's say JIRA plugin or Slack plugin, or maybe GitHub plugin, which you can be useful based on your enterprise setup. Now, the gateway part. So, the how the gateway works. So, this is again, you can build it your own internal MCP gateway if you are again like looking for more secure and safe way. So, the moment you submit your first query to your agent, it will send that one-time auth request to the MCP gateway. MCP gateway recognized and take your request. It will redirect to the respective auth providers, whatever the auth providers you configure, and get you that one-time JWT token. Once client will receive that JWT token, it will know that what MCP server needs to be execute. So, it will submit that tool {slash} call request on a given name space {slash} on a given tool names with that bearer auth token. So, once MCP gateway receive that, it will understand that what name space, meaning that what server it needs to redirect. So, it will take your payload, redirect to the respective MCP servers. Now, once MCP server receive it, it will make that respective API calls to that external systems and execute your request, and then send the response back to user. So, once the user receive the token, you know, MCP server basically, you know, use this bearer token to validate whether user has been authenticated or not. So, what it shows is that, you know, like we build this MCP gateway to make sure that none of the client can communicate directly to your MCP servers. They has to go through the MCP gateway through proper authentications uh before they even communicate to the MCP servers.
### 网关级可观测性与审计监控 在建立网关鉴权屏障后，保障系统长期可信赖的关键在于构建端到端的**网关级可观测性**（Gateway Observability）。缺乏透明监控的 Agent 绝不可直接交付企业业务使用。网关监控体系必须并行捕获四维核心指标：记录每次调用的时间戳、工具名称与调用者身份的详细**审计日志**（Audit Logging）；实时采集延迟分位数（P50/P90/P99）、吞吐量及错误率的**运行时指标**（Runtime Metrics）；用于在分布式链路中排查故障、识别慢工具瓶颈的**分布式追踪**（Distributed Tracing）；以及监控工具可用性与服务存活状态的**健康检查机制**（Health Checks）。所有请求数据在进入网关时均被同步推送到中央日志存储（如 Log Store）与时序数据库（如 Prometheus/Grafana），使运维与安全团队能够随时审计 Agent 的每一次外部交互，确保一旦发生行为越界或服务降级，系统能立即告警并介入阻断。
Original English Source So, now the part is the gateway monitoring, right? We talk about again, we talk about architecture, we talk about MCP server, MCP gateway. Now, the MCP gateway monitoring. Again, the AI agent cannot be trusted without continuous monitoring, evaluation, and observability. This is why we are building the MCP gateway uh monitoring. So, the way it works is that any request comes to the MCP gateway, they've been captured across the four different segments. The first segment is the audit logging, which means that every tool execution timestamp, which tool was called, what was the parameter has been passed, and who was the caller. So, those kind of requests can goes to the log store. The second part is the metrics, which means that what is the latency of that particular tool execution, error rate, throughput, everything has been extracted and pushed into your metrics storage. Third is the tracing, which means that end-to-end distributed tracing across multiple tools execution to make sure that we can identify what tool was taking more latency, where was the bottleneck, and how we can debug. Fourth is the health monitoring, which means that periodic checking that whether that particular tool is alive or not, is it healthy or not. So, once you have all this four layers of monitoring on top of your gateway, you can export these logs and metrics into your monitoring dashboard like Prometheus, Grafana, Datadog, or whatever tool you use, and build the right alerts and dashboards for your on-call teams to make sure that they can take action on any abnormal behavior, right?
### 基于图状态机的 Agent 编排范式 在基础设施准备就绪后，Agent 内部的核心决策与执行逻辑依赖于健壮的编排引擎。相比线性的 Prompt 链式调用，采用基于有向图状态机的 **LangGraph** 架构提供了更为严谨的控制能力。它支持环路迭代（Loops）、分支路由（Branching）与全局状态持久化，能够原生模拟复杂的“规划-执行-反思”（Plan-and-Execute）认知闭环。系统在接收到任务输入后，首先由规划节点将目标分解为具体的执行子步骤；执行节点随后通过结构化状态流转，调用注册好的 MCP 工具并检查其读写属性；若属于只读查询则放行执行并记录审计日志，若属于写操作则触发强制校验；最后由反思节点验证工具返回结果是否满足预期，并在结果不满足时自动回溯重试。将 LangGraph 的严密状态流转与 MCP 网关的安全调用机制相绑定，彻底杜绝了模型陷入死循环或产生无序副作用的风险。
Original English Source So, the MCP server gateway and monitoring is covered. Now, the next is the agentic framework. So, we use LangGraph, as I mentioned. It's provide a graph based architecture that support loops, branching, state management, and multi-agent coordination. So, the reason we chose LangGraph is because it allow us to build a deterministic and predictable workflows on top of non-deterministic LLMs. In our pattern, we build a planner, executor, and reflector node. So, the way it works is that user query comes into the planner node. Planner will understand the intent and break down that complex query into the small subtasks. Then it will pass that state to the executor node. The executor node is going to execute each of these subtasks one by one. Meaning that in the plan like once you know what operation you're performing, it will check whether it is a read-only, then go ahead and execute, and I will save all the record points into our audit logs. If it is a write operation, then we can also put a human-in-the-loop validation or additional approval before it actually execute into the real production databases or systems. Once executor done, it will pass the result to the reflector node. The reflector node will evaluate whether the answer produced by the executor is accurate and meets the user query criteria or not. If not, it will loop back to the planner to revise the plan and try again until the maximum iteration reached. So, if we combine all this together, like MCP gateway, MCP server, um and the agentic framework, it provide a complete end-to-end enterprise grade foundation for building any agents. So, ultimately, whatever the agent you're building, whatever the call uh you're making, it has to go through this foundation. So, agent is communicating to the MCP gateway, goes to the MCP server, and to the downstream systems.
### 双层会话记忆与多维追踪评测系统 在完成认知决策编排后，面对长时间维度的多轮交互与海量并发场景，必须部署专门的状态化**记忆管理机制**（Memory Management）与追踪评测体系。该方案将记忆解耦为两层：基于高速内存缓存的**短期会话记忆**（Short-term Session Memory），用于维护当前上下文与工具执行轨迹（Run History）；以及基于分布式关系型数据库（如 PostgreSQL/RDS）的**长期持久记忆**（Long-term Persistent Memory），用于跨会话沉淀用户画像与核心事实。与此同时，针对数以千计的企业级 Agent 并行运转，必须建立集中的**分布式追踪与评测引擎**（Tracking and Evaluation Engine）。该引擎在运行时全量记录每次调用的唯一追踪标识（Trace ID）、底层大模型的 Token 消耗分布与响应耗时，并实时计算评估指标——包括针对上下文匹配度的真实性评分（Faithfulness Score）以及针对意图达成率的答案相关性评分（Answer Relevance Score）。通过统一仪表盘展示每次调用的工具耗时分布与量化质检分，实现了企业 Agent 资产的大规模白盒化治理。
Original English Source So, now the uh we talked a lot about MCP gateway, MCP servers, um the monitoring, and the agent framework. Now, let's talk about the remaining framework. One is them is agent memory management. So, we build a two-layer memory architecture. The first layer is the short-term memory, which capture the immediate conversation context, tool executions, and step-by-step progress within a single session. And this can be stored in some in-memory DB or Redis. The second layer is the long-term memory, which store the user preferences, historical patterns, and durable knowledge across multiple sessions, and that goes into the persistent DB like Postgres or DynamoDB. And in respective databases, you can use in Azure Azure and AWS as well, right? So, just walk you through the quick demo here on the memory. So, I will refresh this again. Now, if I want to also see that, hey, what tool have been executed when I run this query, you can see all the step-by-step tool invocation history, what was the input, what was the output, and how long it took. Now, further I will also want to see that how the progression was happened on my agentic graph, you can see every state transition in the memory UI. Now, the tracking and eval. So, just to make sure that um there is no confusion here like the we talked about tracing on the gateway level, now this is the agent level tracking and eval. So, we do this because we really want to track that what query you have been submitted, what was the prompt used, what was the completion tokens, what was the prompt tokens, and what was the latency for each agent execution. And imagine that if you have a thousand plus agent, every agents will have their own tracking and eval tables. So, if you combine, let's say if you have 100 agents, you will have 100 different tables or a centralized tracking store. So, let me walk you through the the tracking and eval um demo here. So, on this UI, you can see all the trace IDs, the user query, the model used, whether it is GPT-4 or Claude, and then the total tokens. Now, the eval scores. We calculate faithfulness, answer relevance, and context precision scores in real-time or asynchronously for every single execution. Now, the tool invocations, so every tool call, which MCP tool ran, how long it took, did it succeed or fail, all these metrics are populated. So, this gives engineering teams full confidence that whether the agent is improving or degrading over time.
### 前置隐私拦截与上下文成本压降 在生产落地的最后一环，合规安全与推理开销构成了最现实的商业屏障。为了彻底防范合规风险，架构在请求流向大语言模型之前强制插入了**个人身份信息前置拦截器**（PII Blocker Framework）。该拦截器利用高精度的本地命名实体识别模型（NER）扫描 Prompt，自动将信用卡号、社保账号、电话号码等敏感数据进行就地遮蔽（Masking）或哈希脱敏处理；被遮蔽的原始敏感数据绝不上传云端大模型，也绝不写入记忆库或评测日志，从物理通道上消除泄露隐患。紧随其后的是**上下文优化器**（Context Optimizer），旨在解决冗余历史导致的 Token 爆炸与算力浪费。它在请求发往模型前动态修剪膨胀的上下文：对于重复日志、无序系统输出与多轮废话进行结构化语义提炼，并将非核心检索文档压缩为其关键摘要。在实际企业级 Slack Bot 等复杂交互场景中，经过上下文优化器重构后的调用将 Token 消耗从原先的数万直接压降至约 1,500 Token，单次交互节约高达 90% 以上的 API 成本，最终在保障数据绝对合规的前提下达成了极高的商业投产比。
Original English Source Now, let's talk about the last two critical frameworks: PII blocking and context optimizer. When we talk about enterprise, security and privacy is the number one priority. You cannot send customer's raw data or sensitive data to external LLMs. So, we built a PII blocking framework which sits right before the prompt sent to the LLM. It scans the input for any personally identifiable information like SSN, credit card numbers, email, phone numbers, using local NER models or rule-based patterns. Once detected, it masks or tokenizes that sensitive data. Neither it will goes to your memory DB, or nor it will go to tracking and eval tables, or the LLM. It stays completely within your secure boundary. Um the last framework is all about now is the context optimizer. As your conversation grows, or your tools return huge JSON payloads, your context window explodes. That means two things: higher latency and huge cost. So, we build in context optimizer framework where you actually shape your query and compress the historical context and tool outputs before passing to the LLMs. So, let's see this uh demo here. In this demo, without the optimizer, a multi-turn query with extensive database results took around 15,000 tokens. Now, if I ask uh now I enable the context optimizer behind the scene and I'm rerunning the exact same query, the optimizer strips out the redundant whitespace, deduplicates schemas, and summarizes older turns. I can rerun the same demo on the Slack bot as well where I'm asking Slack bot, "Give me the summary of recent transactions and analysis." Without optimizer, it consumed massive context. Now, what I will do is I will turn on my optimizer behind the scene and I rerun it. Um and then if I run this query again, um it will show like it exactly used 1,500 almost token, which is again like 90% saving. So, to wrap up: first, make sure you choose the right agent framework like LangGraph for determinism. Second is like your Make sure that you have a proper gateway system. Make sure that every tool call request either you are doing this from your local or from production, uh make sure it authenticated before it reaching to the MCP servers. And then make sure that your MCP servers take a control on what tool um you know they should expose and enforce read-only checks. Third, continuous monitoring and tracing on your gateway and agent. Fourth, proper short-term and long-term memory separation. Fifth, PII blocking has to be in place. And last but not least, make sure that context optimizer should always be there to justify your cost on your agents, right? So, just want to share is that I released my MCP book recently on Amazon. So, feel free to download. It's free in Kindle. And it's hard copy is also available. All right. Thank you so much for watching this video. And if you do have any questions, feel free to, you know, like reach out to me over LinkedIn or Twitter. I'm happy to answer. Thank you again.