[Host]: I'm really good at sort of summarizing themes and so I prepared six themes that I wanted to set your pallet with I guess for the rest of this conference.

[Host]: The first thing I wanted to talk about is how AI is increasingly not just models. Um here's two sideby-side shots of open AI saying into um a model is increasingly the harness right AI21 is pivoting to is as well and also uh AI is also about services and data and brand and product right so I think this all these trends are really good for AI engineers because it is everything outside the model.

[Host]: But the second trend also shows that the models are and I think the realist world benchmark is probably the amount of code that's written by AI. Uh this is a screenshot of cloud code commits in GitHub over time. Uh as of February, it was about 4 to 5% of all code. Right now, it's probably sitting at around 10% and towards the end of the year, it's going to be around 40 to 50%. Everyone is sufficiently appreciating that is currently changing the entire world.

[Host]: Cloud code agents are I mean agents are also just self-hosting and enabling slot forks which are really interesting. And also of course obviously there's the Oplaw moment which I don't need to tell you guys about you've heard enough about it already but I'm also very which starts to mark the emergence of agents in non-verifiable domains and I think lastly the high potential is exceeding normal knowledge work knowledge work towards the frontier uh science and technology and math software development life cycle first starting with the idees last year.

[Host]: This is called the jevans paradox as well. And really this is my argument around why AI engineering will be the last job, why you guys all have job security and why you guys will take over the world. So I hope over the next two days you have really fruitful discussions and I'm looking forward to catch up online. See you.

[Host]: All right. And now Sean really owes me one. So we'll be getting here before too long. Don't you worry. All right. So to kick off the talks proper, we have George Cameron. So George is an Aussie who now is in San Francisco. There seems to be a bit of a a patent there and started something kind of to solve their own problems a couple of years ago. There's a great interview with Shan Wang on Leighton Space with uh the founders there at artificial analysis which is the firm that he founded.

[Host]: So if you want to hear the backstory and everything they do, I highly recommend that. Um but what they do is essentially they burn lots of tokens to uh objectively verify across all kinds of benches and benchmarks their own and uh and open ones that um uh uh you know what how the models rank, how they compare. So this is very valuable as part of the conversations we're going to have this morning where we're going to look about tokconomics and and how uh kind of the cost of using inference is starting to build into our product strategies. But to set the scene with all the insights that he has into the modern uh model landscape, would you please welcome George Cameron from artificial analysis.

[George Cameron]: All righty. Are we uh there we go. Great. Thanks for having me here. Um so I'm George, one of the co-founders of Artificial Analysis. Very quickly about us. We're an independent AI benchmarking company. We help millions of users with our website artificial analysis.ai understand what's happening in AI and choose between all the different technologies with our independent uh benchmarks.

[George Cameron]: Um and we're also quite widely referred to in the industry. At the start of this week, Jensen referred to us for the Neotron 3 Ultra release. Last week, Anthropic with Opus 4.8 and our GDP vala benchmark the week prior to that. Sunda with a GPT, sorry, Gemini uh 3.5 flash release. Um and so very happy to be here and share a bit about um what we do and give an overview of how we see the state of AI in June 2026. We benchmark across the AI stacks. We benchmark agents, models, uh in inference providers and and hardware. Uh and we also benchmark um kind of text focused language models but also image, video, speech, uh and music models as well.

[George Cameron]: I think to start off with um I'll provide essentially this chart. It's a bit hectic. Um, which shows the leading releases from the leading AI labs over the last few years. And it's a bit hectic, but I think that's probably no surprise to us in this room because it's been a hectic few years. There's been a lot of releases uh and a lot of progress. And I think why I like to start with this chat is it shows that AI progress has not slowed down. I think claims of that have been greatly uh greatly exaggerated. I heard a lot of that in late 25, but then um Opus 4.6 came out. Um agents started working for much longer horizons and uh people quieted down a bit. And I think this chart shows that really well is that there's more dots on this chart than ever. Um if you look at the last 3 months here, which shows there's been more leading releases um and it's going up and to the right pretty quickly, which is showing progress in intelligence.

[George Cameron]: I'll first introduce our artificial analysis intelligence index metric. Now this is a synthesis metric of 10 benchmarks that we run to test language model intelligence which provides an highle synthesis overview uh and relative comparison of the intelligence of these models. We have the current set of leading models on this chart here. And I think to note, Claude Opus 4.8 last week took the mantle from GBT 5.5 um as the leading language model in terms of intelligence.

[George Cameron]: But I think rather than kind of stopping there, if if we're able to stop there, it'd make our jobs a lot easier. But I think there's a the story is is that there's a reason still uh to use other models considering the cost, the speed, and other trade-offs at play amongst language models today. I'll first take a look at different perspectives on thinking around the current state of models. You can see here that it's very much a US and China story when looking at where the labs making these leading language models are based. Also present on this chart is France with Mistral, South Korea with a number of labs in a very successful sovereign AI initiative. Um and then also the United Arab Emirates. Um I think to us in this room notedly missing is Australia. Australia doesn't have kind of language models that are competitive with the frontier in terms of intelligence. Um and from my perspective, I I don't think that's going to happen uh h happen soon either.

[George Cameron]: Another perspective is looking at open weights or colloally open source models um compared to that of proprietary models and proprietary intelligence achieved. On the y-axis we have our intelligence index uh and then on the x-axis release state and what these lines plot is the leading model that is kind of proprietary compared to open weights and open weights has always trailed proprietary intelligence. This chart is really going back to um kind of mid mid 23 with the Llama 270B release and you can see that yes open weights has trailed open uh sorry proprietary intelligence but it's in a sense kind of kept up roughly 3 to n months behind that of proprietary intelligence. This was not a given. It was an open question uh years ago um or only a few years ago people asking around the commercial models um that could support open weights um but I think what we've seen is kept up for a number of reasons and I think our perspective is this is going to continue with there being sufficient incentive there um particularly for labs wanting to serve those looking for open weights models for their flexibility the ability to fine-tune uh and other reasons.

[George Cameron]: And then secondly, I think another perspective on this is if open weights is kind of 3 to 9 months behind, we're looking at opus maybe 4.5 territory GBT 5.2 for the intelligence that um you can essentially get with open weights models with a Kimk 2 point Kim K 2.6 or the the latest kind of Deep Seek uh V4 Pro. Um and so it remains an option. Um you could do a lot with with with Opus 4.5 or GBD 5.2 And so it remains an option. Um and it looks like that's going to continue based on based on the last few years and our house perspective.

[George Cameron]: We benchmark using quantitative benchmarks but we try and essentially ensure that those benchmarks are aligned from real world use. ... and I think what we tell here with our GDP vala agentic benchmark that uses an open AI data set we turned it into an aentic benchmark is a perspective of okay a leading model a year ago with claude 4 ... and then now with GBD 5 GBD 5.5 and you can see progress is being made in terms of real world knowledge work output... I think one kind of two things that are hard to reconcile in AI today is one that we have cheaper intelligence. So it's cheaper than ever to access GPT4 level of intelligence. Kind of going back to what I was saying earlier, you can get kind of Kim K 2.6 now cheaper than you can get Opus 4.5 uh you know six months ago.

[George Cameron]: And so I think this speaks to you can get intelligence cheaper than ever, but we're also spending more than ever uh within our companies. We're kind of upgraded to the $200 a month uh claude code or codeex plans and kind of why is that is and I kind of break it down into six drivers here that we see is the most critical... 1. smaller models able to achieve more intelligence... 2. software efficiency gains... 3. hardware efficiency. A NVL72 node is able to offer a lower cost when serving at scale... 4. insatiable demand for frontier intelligence... 5. reasoning models. Labs are increasing intelligence through more reasoning tokens... 6. agents. ... Dealing with multiple turns for the same task. ... commonly seeing 60 turns uh for a GDP valet task. Um and so kind of 20 to 100 turns is quite sensible for many knowledge work tasks and it acts as a multiplier on the cost.

[George Cameron]: And what we see here is really in kind of six to uh 18 month periods you have the cost of that intelligence falling often 10 to 100 times. And this is something that we can all think about and take advantage of. If a task was able to you could do it with Opus 4.5. Odds are now that you can there's orders of magnitude at play. It's not 2x cheaper. It's you can go 10x cheaper in many cases by choosing a cheaper model that has more recently been uh been been released.

[Host]: So Sarah is now head of AI at notion. ... she's the head of AI at Notion, but she's formerly been at Robin Hood and Google and she she's the engineering lead on AI modeling uh reasoning, agentic orchestration and model infrastructure. ... she's going to talk there's a bit of a theme emerging as I mentioned around tokens the cost of tokens how that feeds back a kind of back pressure of itself of its own back into our product strategy. So to hear about what they're doing there uh at notion would you please welcome Sarah Saxs.

[Sarah Saxs]: Hi. Thank you for having me. ... I completely agree with everything that was just presented. And I think what I want to talk about is the challenge of actually serving that and making those decisions. ... What does it mean to work at when you don't have the scale and the negotiating power, but you want to be delivering what's best to your customers?

[Sarah Saxs]: Exhibit A, a reasoning model gets upgraded, but don't worry, it's the same price. Okay? Um it might be the same price per token, but you run it on the same exact task. And what happens? It's three times as many output tokens. ... Exhibit B. There's a successor model. ... It's 40% more expensive than its predecessor. But breaking news, the predecessor is being deprecated in four months. And you've built your whole product on top of it. Are you increasing your prices by 40%. No, hopefully not.

[Sarah Saxs]: So the supplier is your competitor, right? ... My job at Notion is to represent that Fortune 5 million because they're our customers. Especially with usage based pricing, the accessibility of AI is determined by the price by the price of those workflows. ... we're handling um tens and tens of trillion tokens a month. Um but we also have over a hundred million customers. What does that mean? It means that I get to play that job of negotiating at scale, but bringing it to the rest of you.

[Sarah Saxs]: One, I see that people are locked in. They have no exit. ... they've committed $20 million to one of that uses the best model available whatever it is. So for instance at notion um we just launched this. This is our manage agents product. We believe that there's value in optionality. You'll see in this example, you can use decagon agents, move them so Claude code can write a fix and then perhaps have Codex review it and then put it in a task list for humans to review, right?

[Sarah Saxs]: Not all traffic is equal. ... In notion, it looks like this. Changing a database field, triaging an email... let's get him on Miniax as soon as possible. ... Summarizing meeting notes. These are all things that we actually don't want to pay 40% more when GPT6 comes out... But for the frontier task, we need to be giving that frontier to our customers. Data analysis, deep research. The point is that our customers need us to choose for them. Otherwise... your product is inaccessible.

[Sarah Saxs]: We're changing our default model for our customers probably every three to four weeks. ... if you are, the second you go from January to February, you're giving your users a worse experience. Okay? It's not worth the discount.

[Sarah Saxs]: But what if you don't need an LLM at all? I know we said that this is welcome to Token Town, but we're leaving. ... Why would he ever need an LLM to do that? Why would you want that repeated task constantly using reasoning tokens to navigate MCPS? Okay, well because Frontier Labs want you to token max. ... Your users don't. Your users want you to outcome max.

[Sarah Saxs]: At notion, we believe that many tasks do better work on CPUs. Determinism is a valuable thing. State machines existed, right? ... So we've partnered with Verscell on the capabilities to launch what we call workers where you can actually call on internal APIs computer sandboxes host small small code as an action that your LLM calls. We've seen this decrease token cost by up to 80% for some of our customers on repeated tasks.

[Host]: So, please welcome or or maybe curse the man who brought us Ralph the Ralph Loop, um, Jeff Huntley.

[Jeff Huntley]: I'm going to say some pretty provocative things like software development now costs less than minimum wage. I want you to think deeply about this. Software has been commoditized. ... Product managers can now be a software developer. Doesn't mean they're a software engineer.

[Jeff Huntley]: ...talking about the unit economics of business have forever changed. ... software is now easy to create. Doesn't mean it is like you're creating the right things. ... Roslin is not a classical software developer by any means. At this meetup, there was product managers, designers. They're all having the time of their lives, folks. not really software engineers up there because our skill set has been commoditized.

[Jeff Huntley]: Previously, software was gated. ... But that's just completely changed now. That's just smashed. The paradigms have smashed. Everyone can now control the computer. They can now write code. Previously, this was gatekeeped.

[Jeff Huntley]: I saw something yesterday. PewDiePie, believe it or not, PewDiePie is writing better tests than most software engineers here right here today. Go look at his GitHub. He's using Antivus with Bomadil with property based testing. Meanwhile, you're using Playright. What does it mean when the these skills that used to be scarce are now abundance.

[Jeff Huntley]: I think there's now two classes of companies. We now have the companies that are already lean ... Meanwhile, we got everyone down the down on the bottom, which is every single other company out here, and they got to go through a J curve transformation program. ... You might have seen this block lays off nearly half its start because of AI. ... My honest take is Jack is right. AI would would allow to reimagine the organizational charts within most companies.

[Jeff Huntley]: ...I call this a curiosity test ... A senior engineer should be able to explain how AI works under the hood. ... why is it when I ask a software engineer what is an agent? ... An agent is really simple folks. ... It's a wild true loop. It takes it takes the prompt, adds it to an array. You send it off for inferencing, and you look whether it needs to execute a tool ... and it sends it off for another turn. It's really simple.

[Jeff Huntley]: ...removing waste in from your systems and processes is a bigger accelerator than AI itself. ... asking them what is AI broken in the systems and processes like do you use agile anymore? ... ideas are more important execution because execution is now commoditized.

[Host]: So, next up we're going to try the highwire act again and we have Eigor Costa. ... Eager was a former leader of GitHub's AI efforts behind Copilot and over the last year or so he has founded AutoHand. ... he's going to talk to us about why our coding agents forget everything and the architecture of memory.

[Igor]: Um in today talk I'm talking about like why agents forget things? ... We've added context window. ... and we treating this as normal. ... we treating the same thing as context and memory. They are very different things and probably they're the same thing. I don't know.

[Igor]: ...let's talk about like the nine types. ... I probably focus in four. So let's talk about semantic memory. ... then we went to procedure memory ... and then we talk about episodic memory. ... then comes the reflective memory.

[Igor]: ...we open sourced the CLI ... we came with this concept at Autoan that we open so it's called commander. ... the memory is the model. ... the agents instead instead of like you try to load everything in the context, you load what is task at hand, you reflect upon and then you spin a new version of that with a different opinion.

[Igor]: ...let's use the taste that Linux 12 has. So I want I I run I've built this application. We run this to for more than 10 months. ... We are still trying to migrate the Linux kernel to Rust. ... um they came up with this new architecture called hierarchical reasoning model. So the mo the memory is the model.

[Vampi]: This talk was to share a little bit from the trenches what we experienced when we started building full duplex voice agents ... scaling for countries like India ... where even the slightest glitch or pause would really ruin the voice user experience.

[Vampi]: ...state machines and reg x go really well together. You don't need an LLM to constantly operate on the transcript. uh you could essentially have a reg x of a million odd patterns to essentially do what a small capable LLM can do. ... it gives ultra low latency. You don't need to waste an LLM call to do that, right?

[Vampi]: If you want to use production voice agents at scale with a millisecond budget, you know, you've got to move to Rust. And the idea is that the latency budget is the architecture here for voice agents. ... you don't have any UX forgiveness it's completely unforgiving so therefore we chose Rust.