工作流而非自动驾驶：Flywire 的企业级智能体架构实践

Eduardo: Hello, everyone, and welcome. My name is Eduardo, and Javier here will represent the AI Platform Experience Team at Flywire. Our core mission is to build practical agentic infrastructure that empowers our sovereign engineers, while following our established engineering standards and practices. Today, we'll share how structured workflow automations deliver real enterprise value, ensuring control, safety, and reliability across our development process. Before we dive into the deep technical details of our AI platform, let us give you some essential context about Flywire.

Eduardo: Flywire is built to process the world's most complex payments across global industries like education, healthcare, commerce, and business services. Today, we serve over 5,000 clients across more than 204 countries around the world. Because we process high-volume international transactions, regulatory compliance, privacy, and security are not optional items. They are core requirements of our business.

Eduardo: We operate under strict compliance frameworks like PCI DSS and SOC 2. This means that customer data, personal identification, and payment details must remain completely safe. We cannot simply take sensitive context and pass it to third-party services without strict governance and architectural boundaries.

Eduardo: To give you an idea of our engineering organization: we have around hundreds of engineers distributed globally across our teams. We maintain diverse technology stacks, including Java, Ruby, Go, Python, .NET, and PHP. Almost everything we build and run is deployed on Victoria, our internal Platform-as-a-Service that operates on top of GitLab and AWS infrastructure. Victoria provides standardized build pipelines and consistent deployment practices.

Eduardo: So how did our AI journey begin? About six months ago, back in April, we formally established a dedicated AI Platform Experience Team. Even before that, passionate engineers across Flywire were already experimenting with AI tools, creating skills, building MCP servers, and assembling an early code review bot. That organic activity proved that our engineers wanted AI automation. It also showed that we needed a dedicated team to provide infrastructure for safe and reliable agentic solutions.

Eduardo: Since then, we have moved fast with close to a thousand commits to our internal tool. But the industry does not stop. In the same six months, Claude Code alone published more than 160 releases, almost one per day, and the major AI labs released more than twenty new models. Every week brings a new model, a new tool, or a new way of working. Our challenge is not only to build fast, but to keep up.

Javier: This brings us to Orbit. Orbit is an internal platform that eliminates daily engineering repetition loops by including LLMs strictly into a structured workflow. Instead of letting AI agents freely open-end, our approach wraps procedural logic into deterministic code, and we call the LLM only where reasoning is actually needed. When we evaluated whether to build or buy, building in-house made complete sense for our environment. Frontier models made feature development really fast, which drastically lowers the cost of a custom platform.

Javier: More importantly, as a financial enterprise handling PCI, PII, and PHI data, keeping execution inside our existing Victoria infrastructure ensures that sensitive data never leaves our controlled environment. We also skip long vendor security reviews and extra compliance work.

Javier: So what does a workflow actually look like? Every agent in Orbit is defined in a single YAML file that lives in a repository that goes through the same Merge Request review as any other code. Take our code review agent as an example. It starts with a trigger, like a webhook that fires when an engineer mentions that review bot on a Merge Request.

Javier: From there, a sequence of deterministic activities fetches the Merge Request metadata, classifies its risk tier, loads prior findings, and clones the repository. Finally, we call the model in an LLM step that can read code, but has no permission to modify, and that must answer under a strict output schema. Finally, another deterministic step posts the findings back to the Merge Request.

Javier: This is what we mean by workflows, not autopilot: code handles the predictable parts, and the model is called only where reasoning is actually needed.

Javier: We chose to start with YAML files and an MR request review process on purpose. YAML is a format our engineers already know, and the review works the way they already work every day. It also keeps the codebase AI-friendly: coding agents can read, write, and review workflow definitions as easily as people can. However, not every use case comes from an engineering team. For corporate use cases, we are considering an authoring tool in the Orbit UI, where people can build a workflow by dragging and dropping from a set of predefined activities without writing any YAML.

Javier: A workflow can start in three ways. Engineers or another system can call Orbit's internal API directly, or use our internal CLI. Webhooks from GitLab, Jira, Sentry, and AWS start that workflow from the moment something happens, such as a Merge Request or a new production error. Scheduled jobs run workflows on a fixed cadence, like our daily translation syncs, feature flag cleanups, and knowledge base refreshes.

Javier: Beyond these entrypoints, we are currently building an MCP server for Orbit so that other AI services and tools we use at Flywire can start workflows dynamically. For example, an engineer will be able to ask Claude in Slack through Claude Code to run an Orbit workflow without ever leaving their conversation.

Javier: We started small with clear, high-impact use cases. First, daily engineering work: automated reviews of Merge Requests on our Victoria upgrades agent, which verifies Victoria compliance in the findings. Second, translations: one workflow now handles the full cycle—it fetches new strings, translates them, updates the files, and notifies the teams with updated docs. Third, production errors: a Sentry webhook starts the Sentry investigator; it assesses the issue, posts the assessment in Slack, and shares a Merge Request with a fix in the same place. On the support side, ticket triaging classifies new tickets and routes them to the right engineering area. AI readiness scorecards measure how ready a repository is for AI-assisted development, and a fixer acts on the findings. Finally, the Jira Implementer takes a ticket and turns it into a Merge Request. It works like a small software factory.

Javier: One of the biggest benefits that we found is that we can make changes in bulk across many repositories, which lets us do software engineering at scale. It is very convenient for migrations, library updates, and deprecations, and we also use it to run Code Doctor and to launch our own AI readiness coworkers. Even with all this automation, the Merge Request is still our unit of work, and it is where a human stays in the loop. We operate in a regulated environment, so people still review and approve those Merge Requests.

Eduardo: Let's take a deeper look at the technical architecture of Orbit. Originally, we began with a simple container setup on AWS ECS Fargate, where the orchestration layer, harness, and the runtime environment lived inside the same container.

Eduardo: As our needs grew, we evolved toward separating the orchestration layer, the agent runtime environment, and the session state into distinct components. This improves isolation, maintainability, and resource allocation. It also makes the system more resilient because an error in the agent does not kill the sandbox, and it lets us scale each piece independently and adapt to changing requirements.

Eduardo: Fargate lets us scale tasks elastically, so we can run a very large number of agents in parallel. It was also really easy to deploy Orbit because we already have Victoria, our PaaS that scales applications without any effort. But that scale needs guardrails so the platform is not abused. First, timeouts: every agent stops after a configurable amount of time. Second, rate limits, which are aggressive by default: a broken API integration could start spawning thousands of agents and make our bill much bigger; rate limits keep that under control. Third, concurrency control: the control plane uses locks to prevent duplicated executions of the same task. For example, our control plane uses locks to prevent duplicate requests from running agents twice.

Eduardo: Victoria comes with container limits. Victoria uses standard Fargate task definitions, so the containers are lightweight and entirely restricted. Our baseline agent container includes common language runtimes, but it intentionally leaves out package managers like Bundler, Maven, and Composer. So heavy agents like our Jira Implementer cannot easily run native builds or unit tests inside the container.

Eduardo: When we started Orbit in April, AWS managed microVMs did not exist yet; AWS released them in June. So our first version ran only on Fargate. These managed microVMs work like Lambda functions with resumable processes, so they can pause and continue where they stopped. That fits the nature of agentic execution really well, so we now use them with a runtime chosen by project type. Each tech stack gets its own container quickly in an isolated environment.

Eduardo: These are not full sandboxing yet, and we are actively moving there. The goal is to support setups like Docker-in-Docker or multiple microVMs, so agents can run tests or more complex verifications on any repository inside.

Eduardo: For each LLM step, we did not build our own harness. Instead, we reused the Claude Agent SDK, the same harness that powers Claude Code. This gives us a much more well-tested harness with tool usage, sub-agents, and skills out of the box, so we can focus on the workflow rather than the plumbing.

Eduardo: What we add on top is control. We restrict different things for every step: the network the agent can access, the tools it can call, the skills it can load, and the MCP servers it can use, down to the specific tools allowed from each server. All of this is declared in the YAML file itself. A search step, for example, can read and search the code, but cannot edit files or push changes. And our Sentry investigator can query traces through our MCP server, but only with a short list of read-only tools. In practice, each LLM step runs with only the access it needs, fully visible and reviewable like any other code.

Eduardo: Skills themselves come from a single place at Flywire. We maintain a central repository of skills that is published internally as a Claude plugin marketplace, and Orbit connects to that marketplace at build time. This means the same skill works in two places: an engineer can run it in Claude Code on their laptop, and Orbit can run it inside an automated workflow.

Eduardo: That shared foundation also shapes our recommendation to teams: start with a skill. A skill is cheap to write, easy to iterate on, and useful from day one. Once a team is sure that a task is worth automating, we move it into Orbit and turn it into a workflow. The predictable parts become deterministic code, the model is called only where it adds value, and each run becomes more reliable and less expensive than an open-ended session.

Eduardo: Workflows also run in the background, so engineers do not have to sit with an open session and guide it step by step. That means less context switching and more time for the work that needs them.

Eduardo: Beyond runtime isolation, we also rely heavily on structured output to guarantee systemic reliability. Every LLM step in an Orbit workflow enforces strict JSON schemas for model responses. By constraining output formats, we ensure that subsequent execution branches, error handling, and state transitions are driven by deterministic control flow rather than unpredictable string parsing. If an LLM response fails validation against the expected JSON schema, Orbit automatically triggers a targeted retry or fallback path before proceeding.

Eduardo: Another crucial pattern we rely on is isolated component evaluation. Evaluating complex end-to-end agentic workflows in synthetic benchmark environments is often noisy, expensive, and difficult to reproduce consistently. Instead of trying to evaluate entire workflows from start to finish, our evaluation harness isolates individual LLM steps. By injecting exact historical state and context directly into specific workflow nodes, we can evaluate model output quality, measure prompt adjustments, and catch performance regressions quickly and cost-effectively.

Eduardo: So how do we know when an agent is good enough? Today, a new agent usually reaches production after a vibe check: the team tries it on a few real cases and confirms that it behaves as expected. Once it runs in production, we trace every execution end-to-end with OpenTelemetry, including each activity, model call, and tool call.

Eduardo: We then extract traces from real executions to add to a golden dataset. For now, each golden dataset is versioned in the repository next to the code, and we run evaluations on demand. Based on those results, we improve the prompt or change the model, and we run the dataset again to compare. We do not have fully automated continuous evaluations yet, but we are working on them internally. Our goal is to run the whole evaluation loop inside Orbit from the UI or through the MCP server: first, you deploy your agent, then you monitor and evaluate its execution in production, and finally, you improve the agent based on those evaluations without the need to deploy it again. That is our roadmap for evaluation: to trace, measure, and improve every workflow.

Eduardo: The operational impact of this approach has been significant across our engineering organization. In the past 30 days alone, our automated code review agent completed over 9,700 runs, averaging under 3 minutes per Merge Request. Beyond code review, our Audit Fixer opened 159 Merge Requests to fix vulnerable dependencies, and the AI readiness fixer improved around 80 repositories. The Victoria linter fixer fixed 430 projects, and the merge conflict bot was run over 921 times. Across these workflows, we ran over 11,000 executions with a 99% success rate.

Javier: Reflecting on our journey, adopting a self-service model was one of the best decisions we've made. Standardized scaffolding allows engineers across SRE, Security, IT, and Support to build custom workflow steps easily and safely.

Javier: We also learned that cost control starts with model selection: not every step needs the most capable model. So each LLM step in a workflow declares its own model. For example, a planning step and an implementation step can use different models. Matching the model to the task keeps cost under control, and that is why a cost-aware model router is on our roadmap.

Javier: On the other hand, managing data separation remains an ongoing responsibility. We maintain a strict isolation between engineering SDLC data, corporate operational data, and sensitive payment production data. Additionally, because demand for automation is high, maintaining focus and prioritizing high-impact work is essential.

Javier: Looking ahead, our roadmap focuses on expanding our platform features and increasing overall architecture flexibility. We are actively moving toward complete sandboxing, so agents can handle complex builds, local environments, and test suites seamlessly across all project stacks. Moving forward, we are also introducing resumable sessions to handle complex multi-step tasks. We are also building model evaluation tools directly into Orbit, so teams can measure the quality of their workflows on the platform itself. In parallel, we have refactored our orchestration layer to be model-agnostic, enabling flexible routing across different model vendors, including an automated model router that selects the most cost-effective model for each task to help us control cost as we keep scaling the platform. We also want to make the platform harness-agnostic, so that we are not tied to a single harness.

Javier: To wrap up, three takeaways: First, start small. Pick a repetitive, well-defined task, and prove it works before you scale. Second, use the model only where it adds value. Everything predictable is solid as code, and that is what keeps the workflows reliable. Third, keep guardrails and humans in the loop: timeouts, rate limits, and Merge Request reviews. That is how we move from fragile AI experiments to dependable workflows our engineers use every single day.

Eduardo: Thank you so much for your time and attention today. Feel free to reach out to us if you have further questions or comments.