银行部署 AI 的本质困境：为什么这不是一个工程问题

hello, i'm welcome today. we will be talking about deploying AI ahead of a bank and how it is not an engineering problem. before we get into that, i want to introduce myself, my name is Paden Gayle. i worked across many different fields. so i started off at consulting at Accenture, moving over to Bloomberg, getting into a software engineering team before i worked into the infrastructure team and the data platform team. then afterwards, i was at Google working on security and deployment. and right now, i am a founding Forward Deployed Engineer. i started up and writing the book on Forward Deployed Engineering with O'Reilly.

in this example, the system reviews client documents and flags exceptions for analysts. the demo shows the intended workflow, and everyone can see why it would be useful. then security asked, okay, which environment will host it? the data owners asked who agreed to this use of documents. and someone from risk asked, who will review a wrong exception before it reaches the next system. those are reasonable questions, right? each question can change what we are allowed to build. the team has already invested in a design, while several people responsible for that design had never seen it. so in this case, the demo works, the model was fine. however, it still did not ship. i want to now show you how to uncover those decisions in week one, so you can design a system with a credible road into production.

labels represent decisions your deployments depend on. for each one, i want a person who can explain the decision, the evidence they need, and the next step. so for every dataset, who owns the source, who can get access, can you reproduce this in the workflow? those responsibilities may belong to different people, but being able to query data does not settle every question about how the application may use it. for the model risk, find a person who can tell you which review applies to the intended use, ask what risk submission looks like, who will own the system after launch, and what evidence the owner will need to maintain it.

for third-party risk, draw the actual dependency chain including the model provider, the hosting arrangements, vendor review, and operating service. find out which relationships have already been reviewed and which proposed changes require a new decision. for audit, ask what someone would need to reconstruct a historical action: which input was used, which configuration produced the output, what did the reviewer see, and who approved the action? those are the questions that affect the records you need to capture from the beginning.

now imagine getting those answers after you built the whole entire system, but a deployment boundary, a data path, or action that may have to change. the useful move is to bring these people into the design early. their responsibilities help us identify the system we can actually operate. and here, discovery is the engineering. so now you want to talk about what discovery is and who we need to really find out to understand what exactly we need to build.

so you want to think every dataset has an owner. the work i did at Bloomberg on the data platform team made this concrete for me. building infrastructure for an LLM meant understanding which datasets could be used, and who could approve that use. carry that into each bank deployment, honestly any deployment. so before designing retrieval, find the person accountable for each source, confirm the allowed use and the environment where the processing can actually start to happen. if those answers change, your design changes with them.

imagine the key on the slide stands for a real permission with a real owner. we should be able to name that owner to know where exactly this key needs to come from when starting to scope out an architecture. all the questions you may have can be answered by looking at the org chart. read the top first: documents go into the model, and analysts use the output. you can probably sketch that system in a few minutes. now read the right one underneath it: the documents have a data owner, the model service needs a platform owner, the analyst workflow needs someone who can confirm their own risk requirements, and the last line has a question mark. that question mark is a dependency. we have designed a workflow without finding the person who can tell us whether the intended use is acceptable and what evidence will be required.

so how to walk through this diagram with a sponsor, or someone you trust, and figure out: okay, who owns the documents? have they heard about this use? who can help us reach the appropriate reviewer? then how to confirm the answers directly with those people? the real map will have more relationships than this. a technical component may depend on several approvals. some decisions span the entire system. the point is to make those dependencies visible while the design is still easy to change, and a named owner becomes a specific task with a person responsible for resolving it. that is how discovery turns uncertainty into an implementation plan.

and given that, you may start to ask yourself: okay, what sort of questions should i be asking? so here are some of the questions i would bring to a discovery call: where does the data live and who can approve reading it? who has stopped a similar project before and why? which existing controls could cover this workflow and who owns them? where will this run in production, and who are the administrators for the environment? who signs off the relevant risk review, and what have they signed off before? what obligation is the sponsor personally worried about, and who can explain it?

notice what i want to leave with: it's how to name decisions for next conversations. if someone says "security will be fine", ask who will actually review the design and when you can meet them. then close the loop: send each owner a short description of the proposed use and confirm what they need from you. by Friday, the people whose decisions that deployment depends on should know the project exists. that gives scoping a much firmer starting point.

and once all that is done and we know who owns the decisions, i can scope the work even when the customer cannot write or give out the spec. i rely on these three things: shadow, draft, and shrink. for example, at Bloomberg, i worked on a system that handled cash flows that had to reconcile, and that gave me a concrete place to start: follow the flow and the people responsible for it. use that approach when a sponsor wants AI for onboarding: sit with an analyst, watch which systems they open, where they make a judgment, and when they ask for permission. ask what happens when the document is missing. write down the actual sequence; that is the beginning of every specification.

now imagine we are building the document exception system. this is a hypothetical example, but i would still put these three lines in front of the sponsor: 1. review documents; 2. draft the exceptions; 3. analyst submits. then i would make each line specific: which documents can analysts access? is that access appropriate for this processing? what should the model produce: a highlighted passage, a draft form, or both? and who checks the draft? also, who submits the final version? suppose the output is a highlighted exception and a draft form: the analyst checks the source, edits the form, and submits. that is a boundary someone can correct.

i would also propose success criteria: either less review time without an increase in missed exceptions. we still need to agree on how to measure that against a current benchmark, because that is the whole point. it is a proposal until we have an agreement. take the draft to the sponsor and the control owners. they can cross out a document type, they can change a reviewer, or they could reject the output. each correction gives a decision we can build around.

once the workflow is complete, scope becomes a series of approval questions. so now let's grow our hypothetical system one step at a time: first, it summarizes a document that the analyst already has (we will still confirm whether that existing access covers its use); next, it reads from a second system (now there's another data path, and somebody needs to approve it); next, it drafts a response that a customer will see (who checks the content and which review applies?); finally, it sends a response automatically (who owns that action? what happens if it sends something wrong, and how do we stop it and recover?). those are four different engineering scopes. the applicable reviews depend on the institution and the use case. so i'd confirm them with people who own those decisions.

then i ask which increment creates useful value today. could i draft inside the analyst's current workflow to solve enough of the problem? if so, start there. confirm that the existing control really fits and build the smallest useful version (MVP) with a credible path to approval.

that brings us to autonomy: start with the authority that the problem solves, then ask what task can be proven as evidence to give the system more. so when working on cash flows at Bloomberg, i thought of automation a lot of the time: every automation step needs a control, and somebody owns the number. i carry that discipline into AI: find the consequential action, name its owner, and decide what happens when the system is wrong.

for example, in a hypothetical exception workflow, the submission is that action. an analyst reviewing the draft needs the source evidence and visible uncertainty to make a real judgment. a review button only helps if the person can actually check the work. so when thinking about autonomy, i want you to read the ladder from bottom up:

• "Suggest" means the analyst sees the recommendation and chooses what to do with it.
• "Draft" means the system prepares something, the analyst checks and submits; that is a high-leverage step.
• "Acts with approval" means a person authorizes each action before it happens.
• "Acts" means the system operates within limits that have already been approved.

each step changes the consequences of an error. however, it also changes which controls must work and which reviewer needs to be convinced. i would start at the lowest step that creates value, then collect evidence from there: are the reviewers overriding the drafts? which errors keep recurring? can people recover when something breaks? we use those observations to make the next authority decision. a model does not become more capable by itself; answer how much authority this particular workflow should give it.

now let's make that concrete with the same hypothetical exception assistant. suppose the initial request is an agent that reads client documents, decides which ones fall outside of policy, and files exceptions automatically. in that design, the model's judgment can become an operational action without an analyst checking. now keep the model and change the action boundary: it highlights a likely exception, points to the open source, and prepares the draft form; the analyst checks the evidence, corrects the form, and chooses whether to submit.

take that design to the people we found during discovery: can this use the analyst's existing workflow? does the approved data access cover the processing? which controls apply and what evidence do the reviewers need? human review does not guarantee approval, and it does not determine a regulatory category on its own. we still need to confirm the intended use and the required reviews. the useful question is whether this drafting version solves enough of the problem to ship first. if it does, you can collect evidence about errors, overrides, and recovery while operating within that boundary. the next decision about authority can draw on what happens in real-world use. that makes evaluation part of the operating plan.

and that brings us to our fourth pillar: evaluation. the risk team can use evidence, which really just means the controls they are responsible for managing. at Google, the question that needed answered was: what happens when a system breaks? who notices? what stops? who takes responsibility? how long will it take to bring it back up? for an end deployment, i want the concrete plan: the intended use, a threat model, the failure modes, and a response. show me the controls working when the model gets it wrong; name the person who receives the alert and can stop the workflow. that gives a practical way to evaluate.

going back to the hypothetical exception assistant: if it invents an exception, the analyst checks the source passage and rejects the draft; if extraction produces unreadable text, stop and reclassify the file; if it misses exceptions, targeted offline cases help expose that weakness (they do not detect every live miss, so keep high-impact work under manual review). for each failure, name the detection path, the fallback, and its owner. put those decisions in an evidence pack.

these five files make the review complete:

1. "Intended Use" records users, boundaries, and prohibited actions.
2. "Data Lineage" records each example's source and usage approval.
3. "Failure Modes" records detection and fallback results.
4. "Results by Segment" records coverage and errors.
5. "Controls" records review thresholds, stopping rules, and owners.

to generate the evidence with each evaluation run, preserve the input prompt, model version, configuration, output, and reviewer action. that lets us reconstruct the historical decision. calling the model again later may produce a different answer, so we preserve what actually happened. a reviewer can then inspect that supporting record.

now we can look at what is used to organize that result in the fourth file: for a hypothetical workflow, i would separate clean documents, scans, rare exceptions, and new document types. measure their errors and control behaviors in each group, then lead with the weakest result; an average can hide a serious gap in one segment. if a group has too little evidence, say so and restrict that use until you have enough. this tells reviewers what the results support, and it brings us back to our week one decisions.

rounding everything up by Friday, i like five answers:

1. Is the data owner engaged and aware of this project?
2. Are the production environment and its administrators named?
3. Can the sponsor describe the workflow without saying "AI"?
4. Can the smallest useful scope fit existing controls confirmed with their owners?
5. Do we know the risk signer, and what happened to the person who signed last time?

use those answers as a heuristic: 4 or 5 "yes" answers suggest a credible path to ship; 3 means unresolved decisions need owners and target dates; 0 to 2 means resolve the decisions before committing to delivery, including possibly shrinking or redefining scope. this score does not predict approval, but it does tell you what to do next week.

what i want people to walk away with is that discovery is the engineering: draft the spec, then shrink it; give the system only the authority it needs; build the evidence pack. and remember: the demo worked, the model was fine, but it still did not ship. so what do you do? you go find the people.