Hey, my name is uh Cormbrick. I work um on Google AI edge, which is a way of kind of bringing models to the edge. This is something we use internally for our own products. It's also something we make available as open source products. As part of this, we also work really, really close with the Gemma team because they publish a lot of models that are targeted to edge devices as well. Um, yeah, quick bit of background on me. Um, yeah, I've been focusing on Edgi for the last like, wow, like maybe 10 years at this point. Um, so started off in 2016 with like running a uh Google Net on a USB key hardware accelerator plugged into a Raspberry Pi at like Nurips's 2016. So yeah, it's been a lot of fun over the years. I then joined Intel, worked on led the architecture for the NPU that goes into all of their laptops these days. Three years ago, I moved to Google to work as tech lead on Edge AI here. Uh yeah, and that's been great fun like been a crazy few years as you can imagine. Um but yeah, happy to share like both where we're up to and also give a kind of overview of like as part of that we can also kind of cover the story of like where is um where is mobile mobile AI up to today right like what's the state of the art what are the different patterns we see for model deployment... yeah like there's a lot of benefits to running on the edge. There's um like latency or UX improvements for some really sensitive in the loop um things like live voice translation for example. Uh that's something our team shipped on Pixel last year where you can do a live voice translation. Very challenging to do that with the required latency using a cloud service. So being able to do that offline and on the edge is key to be able to meet latency or user experience requirements. Um privacy that's um uh that's definitely a thing as well... offline use yeah that's kind of obvious and then savings um yeah like we see this being increasingly relevant for laptop users... some folks are interested in exporting savings as well.

Oh yeah. So this is kind of important uh concept is we see two trends happening today. Uh one is we see kind of system level genai. So for very very large models, the way these tends to turn up on mobile phones isn't that you know when you launch a single app it will download a four billion parameter model uh just to uh you know just to help you find a good restaurant in you know uh in whatever app you're looking up looking in right instead the trend is to build larger models into the OS. So we call this like system level gen AAI. These models tend to be in the kind of two to five billion parameter range, right? Like it depends on the OS and um whatever. But like this is one choice that we see both the Android team who we work closely with and the um Apple intelligence team. we see kind of similar choices being made by both mobile OS vendors where there's a central model built in... so then the other trend we see is like inapp genai which is where the kind of tiny LLMs or um TLMs as we're calling them in this presentation are more uh are more relevant right... inapp genai generally these are custom to tasks they're loaded with the app or with the web page. We also deploy some of these uh on the web and generally they're targeted at kind of like wider reach... we can get really reliable performance from models in the um uh really reliable performance from models in the like 100 to 500 million parameter range depending on the complexity of the task, right? ...for the really really tiny models uh certainly less than 500 in our experience you need to fine-tune to get kind of production level reliability.

The thing we're the thing that's newest from my perspective is kind of agent skills on device... what's actually happening is um the way we've built the skills is they're efficient. Uh so the instructions can be kind of loaded on demand. So this kind of uses a principle... of um kind of uh wow um like kind of progressive disclosure of conditional or conditional depth. So instead of like in an MCP workflow where you need to describe everything about all of the functions that you need, the way we've structured the skills is there's a kind of oneline description, the oneline descriptions of the skills is what the agent sees and then if it thinks that sounds interesting, then it asks for more. It asks to load the skill... and then it goes in and loads the skill and it finds out all of the the details about how to use it, what function calls it can use to to do that skill. So this is like that pattern is particularly important for kind of token efficiency and frankly like reliability on edge models... if you can have a more condensed context window that'll kind of uh up your batting average in terms of kind of quality metrics you're looking at... within our structure of skills we have skill.md and then there's optionally kind of scripts or assets... within scripts, we do also support um uh people kind of writing custom JavaScripts that can then be rendered within the app... [Examples mentioned: Mood tracker, Map skill, Wikipedia skill, Mood music scale, Flashcards, Piano playing].

Okay. So this is like what we do as like our team we do enable people to build apps. uh we do uh media pipe uh lighter TLM which is a LLM runtime that works on mobile and edge and then lighter t which is just a standard inference framework this was a long time ago known as tensorflow light... it's also you can take the same TF light file and ship that to iOS, Mac OS, Linux, Windows, web or IoT devices just from that same file... One caveat there, that same file deploys on CPU and GPU. For NPU, we need to do some special compilation and you end up with a special NPU file... under the hood we actually invest an awful lot of work in um optimization libraries. So we've X and which is a CPU optimization library and ML drift which is an optimization libraries for GPUs... For NPU we need something a little more specialized. For npu, we need to call a vendor compiler plug-in upfront and that uses an ahead of time compile workflow. So you need to produce an artifact that's particular to a particular npu. ...for the two billion parameter model we can get really compelling performance on a like a high-end Android phone can do thousands of tokens per second on a GPU... on a Raspberry Pi we can get um about 133 tokens per second.

What I'm going to show next is an example of a app that we've built using uh tiny LLMs... called AI edge eloquent. Um what this is is a kind of transcription um model but instead... it can kind of remove all of the kind of interjection and and words... one of the other things it has as well is a uh a biasing uh list... any standard transcription service would when you say Laura, it would translate that to the name Lau or A... Whereas you can give it a list of kind of keywords or technical terms as well. And then the model will kind of um uh bias to those words... the microphone input goes into a speech recognition engine that delivers a unfiltered transcription... both of these go into a text polishing engine which is then a um dedicated L mini LLM just for text polishing... generally what that workflow looks like to give you a bit more insight is we'll use a workflow of using a much stronger LLM in the cloud to generate lots and lots of synthetic data that corresponds to the type of thing we want. And then once you have a few, you know, lowdigit millions or tens of millions... you kind of put that into a fine-tuning workflow and you fine-tune the base tiny model you're working with to get a derived model. ...for tiny models uh certainly less than 500 in our experience you need to fine-tune to get kind of production level reliability.

Questions about safety on edge models. So firstly the Gemma team spend an awful lot of time on this... the system vendor will typically have some sort of input and output safety checker on the model, right? Uh, as a kind of aftermarket addition... the mediumsiz models have a context window of 128K and the smaller models [E2B/E4B]... 32K. ...the amount of kind of KV cache that's required for each input token that was something that was optimized. So the models behave pretty well on that front. ...if somebody asked me about deploying the 2B on an uh on a robotics platform I would be like absolutely you should fine-tune Lauras for each of your things for each of your tasks and then like our runtime supports loading the model and like hot swapping Loras so you don't even need to kind of load and unload the model to load and unload Laura adapters uh that it's built for that particular use case for like robotics or IoT platforms.