Revolut 如何通过自建开源模型降低 90% AI 编码成本

My name is David.

I lead AI operations at a company called

Revolut.

We have 50 minutes today together.

We'll talk about high-performance

engineering and

self-hosting open-source model on

dedicated GPU. How's that sound?

Cool. So, it's going to be a technical

one, so bear with me.

this is the US and you may not know

Revolut.

Uh quick words about our company.

We're a digital bank in Europe. We have

80 million customers.

We're live in 45 markets.

With the most downloaded financial app

in Europe for the last year.

And as of July, we're Europe's largest

private company.

Um

So, we're doing okay.

The

the emphasis of Revolut is, you know,

shipping a lot of product and so there's

a high focus on engineering culture and

uh

excellence on that domain.

And so, we'll talk about that.

We have a chart here. I going to see two

lines. This is a sample of

2,000 engineers performance over the

last year.

So, the green line is going to be lead

time. So, it's going to be

when engineer picks up a feature until a

feature is merged and ready to be

deployed.

The blue line is going to be coding

time. So, the green line is in days. The

blue line is in hours.

people forget because AI moves really

fast.

Cloud code came out

around February 2025, right? And that's

the Cloud Code Sonnet 3.7, the one would

delete your prod database, right? Not

the safe Cloud and Codex that we have

today.

So, bear in mind when we talk about

adoption.

January 25th, first introduced AI in the

company, right? Gemini.

First time people were able to use

private data within AI.

Around February, we start playing around

with Cloud Code.

And then January is when we first

launched Cloud Code at scale.

What you see is

Gemini had 10% improvement in lead time,

20% improvement in coding time, so

that's people

no longer needed to look Stack Overflow

and read documentation.

But then Cloud Code

not so much impact at the time.

skeptical about

the cost of AI. We were seeing engineers

spend a thousand a month, ten thousand a

month with not so much ROI.

And so, when we launched it, we launched

it with

strict budget limits.

And so, we're looking at the next thing,

right? So, how can we provide

unlimited tokens to engineers so they

can use AI guilt-free

for a fraction of the cost. And so, we

looked into open source.

self-hosting open source model. We're a

big fan of GLM for coding.

So, around that time what you see, April

26th, GLM 51

we see 60% improvement in coding time,

70% improvement in lead time. So,

dramatic impact.

What do we learn from this? Well,

there's more impact on the lead time

than on the coding time, so

what we find is the bottleneck for

engineering is not necessarily coding.

You can save a lot of time not having to

dig through your inbox or

messing with Jira and Confluence. And

so, when you have unlimited token,

you're free to do that.

You no longer have to save your tokens

for

end of the quarter or the most difficult

task.

Well, open source is now a priority at

Revolut and yet the economics are

favorable.

The billing is flat, we don't have any

bill shock at the end of the month.

And the French here is the exception.

So, obviously, we keep using CodeX code

code.

The latest models are fantastic. Uh

Jimmy and I are going very well as very

good as well.

But, um we keep them as exception.

Some metrics that matter when you're

self-hosting model.

So, to serve 2,000 engineers, we roughly

need 56 B200.

And that's to uh host GLM-53. GLM is a

big model, right? 800 billion

parameters.

What you want to aim for is sub-3 second

time to first token.

This is empirical evidence. As soon as

you exceed 5 seconds,

my Slack inbox starts to blow up and so

we know that's the limit.

In terms of throughput, it doesn't

really matter as much, right? So, we

serve in terms of interactivity,

200 tokens per second.

People are fine, you don't need to go

higher.

Um the way people work has changed, we

no longer look at the stream of thinking

tokens, people run tasks in parallel and

check in when the task is done.

But, if you want to have high

throughputs, when you're an engineer at

Revolut, you just connect after 5:00

p.m. when no one is on GPU and you can

get 800 TPS and then,

you know, it's a dream.

In terms of cost, in terms of impact, uh

self-hosting GLM, our blended cost of

token is 30 cents per million.

And so, our invoice is 1/10 of what we

were spending on

Anthropic and um

OpenAI.

into running open source models.

If you

just get GPUs and upload

weights on a GPU and

you start doing some inference, you're

going to get terrible performance. It's

because your workload is going to be

very dependent

um on the parameters that you set.

So, if you look all the way to the left

of that chart, the blue line is going to

be demand, so that's request per second.

That's going to be what's going to be

the bottleneck for your latency for your

GPUs.

And the red line is what we're

optimizing for, that's the time to first

token.

So, when we started

back in April, you could see

request per second around three, which

is high but not so high.

And time to first token over 10 second,

which is very slow.

All of the data that you see is the same

number of GPU. We didn't increase GPU

count, we're still at 56.

And by the end, you see in August

request per second, we're about over

seven, and time to first token we're

around 1.6. So, we've had

88% improvement in performance for 76%

increased demand.

So, the learning from this is there's a

lot of parameters that you can

improve on, and you really need to

really scaling if you're looking to

serve for some model.

So, let's talk parameters.

There's no secret recipe, but the things

to consider

are the following.

long context is a scam.

You really need to be mindful that even

though you can serve 1 million token,

256 256 is really what you need.

Quantization, if you're running

Blackwell generation GPU,

you don't want to run anything higher

than FP4.

And VFP4 is best. Roughly going from FP4

to FP8

is a 40% saving in GPU footprint for

zero loss in

uh, precision.

Other parameters to consider, tensor

parallelism, even though GLM fits on uh,

TP4,

you don't have enough KV cache, and so

maybe a TP8 works better depending on

how large your workload is.

And then the more esoteric,

check your cache hit. Coding test needs

to be above 90% uh, cache hit.

And you can look into optimize

spec decoding, uh, sequential pre-fill,

PGD zag, all of these.

production setup.

Gnome obviously is a text-only model,

which is a limitation for some people.

The trade-off I would say we would make

every time is get the smarter model and

figure out how to do image recognition

another way.

You can easily have a local router. Open

Codex is one

where you would route any image

recognition to

Gemini or Luna

and keep Gemini for the main task.

If you really need some vision model,

Gnome 5G flash

works really well. We have our UI

designer

making great use of it.

Turnaround for

UI product has gone a really, really

long way.

And then the last one is going to be

about the harness.

So, all of these endpoints are going to

be OpenAI compatible API with response

API and so you can use them

inside Open Code

Codex

Codex app as well. Works really well.

And so, that's what we we recommend.

last message, this one is for the CIO in

the room.

There's been a lot of stigma in using

Chinese open source model.

So, the thing to consider is you're not

using

the inference dedicated endpoint that

goes to China. You just getting the

weights and putting them on a US-based

service.

So, for data sharing side, this is

totally fine.

Revolut in particular, because we're a

EU regulated bank, we fall under the

most

strict regime in terms of compliance.

And so, what we find that works

is

all of these coding tasks are not

considered autonomous agents.

They're always with a human in the loop.

And so, how do you give uh reassurance

that you have a human in the loop?

You start by creating a sandbox that

guarantees that. And the sandbox means

now, if you've used Codex code, it has

all of these features of approve for me.

You just have to set the rules for the

operations that you don't want to allow.

So, you can allow grab but not Gradle,

for instance.

And that's going to give confidence uh

to your regulator that

this is not an autonomous agent that

needs different uh supervision.

rogue use, right? You may have bad actor

inside your company, and you want to

shield uh for that.

You 'd be surprised what you find in a

large company.

And then, the last one is prompt

injection. Not not so much a thing uh

now given how smart models are, but

always a thing.

In terms of data privacy, we spoke about

it. So, as long as the data sits

in one of the countries that are in the

data equivalency framework, so US, EU,

UK, Israel,

you're fine. You can get your data to be

used there.

All of your inference provider or GPT

provider are going to have

zero data retention by default.

And then, the last thing you need is a very strong uh open telemetry so that if

you ever have an incident, um you need

to be able to go deep uh and look at the

logs.

Also gives you some uh yeah, analytics.

All right. Thanks, everyone. That was

all for my talk.

[applause]