I'm the CEO and co-founder of Estuary and we're a right time data platform. And what we mean by a right time data platform is that we are a unique platform that can work with both batch and streaming data. Um so, you know, data at your at your speed whatever you need it. The idea for Estuary came from a previous company that myself and my co-founder made. It was called Arbor and Arbor was in the MarTech space. Um the MarTech space is kind of interesting because you don't think of it as um very new and fresh, but uh it actually has a very strong use case for real-time pipelines. And so we wanted to be able to deliver real-time data to our customers alongside being able to also deliver analytics, reporting, things like that. Um so we ended up building a platform that works for both streaming and batch. And that's a pretty novel thing, right? It It's not something that it really exists today. You know, it turns out that that actually works really well for AI pipelines as well and I'll talk a little bit about that and and how it actually works and what it does. So, let me talk a little bit more about Estuary. I have to remember to stay here. Um Estuary is a as I mentioned a right time data platform. We try to make it possible to get data from any system that you want to any system you want at the time you want and you know, wherever you want. And so we do that in a way that the alternative would be using something like Kafka for streaming um and maybe a homegrown pipeline or a set of you know, batch pipelines that were managed by a vendor for for batch. And you know, I think most companies in the world actually do that today. You'll have a team that's on a a software engineering team that's building with Kafka and most likely a data engineering team or maybe another, you know, marketing team that's building with batch. So, that's pretty common. It has some drawbacks in that the pipelines don't necessarily stay in sync, but overall it tends to to work okay.

Um about I think it was 3 years ago, my co-founder was on this stage and he talked he did our debut. Um, he actually showed our right time data pipelines and it went really well. What we didn't realize at the time was we had really just scratched the surface of what we needed to build to make something valuable. Um, there's so much more that goes into it and Matt can attest to this. Um, you know, it things like schema evolution are absolutely critical for any sort of data pipeline out there. You can't break a pipeline if someone adds a column or misses or, you know, break it if someone changes a data type. So, handle every possible edge case. Do it in a way that you can strip PII if you need to. So, may omission and hashing, things like that. Do it in a way that um, you can also transform data um, and prevent present things like SCDT type two slowly changing dimensions. All these little bits and pieces that go into creating real production grade pipelines. You know, when it comes down to it we also we didn't realize it but um, obviously you'll have to build a lot of compliance related things such as SOC 2 compliance and HIPAA but we ended up having to also build like BYOC where we can deploy in a customer's environment to totally isolate it from everything else. So, that's great and we got to the point where about 80% of our users could go in and build a pipeline with one try but there's still 20% that that they can't, right? And so that's for us that was something that's pretty important. We wanted to make sure that we're optimizing towards that 20%. It turns out that most of the time with those users are doing is they're running into an error message that they can't diagnose from a third party system. You know, it's it's impossible when you have 200 connectors to catch every single error message from every single third party system. So, we wanted to make sure that we could we could do that. And the only way that we found that's possible to do that is really to use AI. So, you know, that's a place that where we've been integrating AI into the platform for real-time feedback and diagnosis.

Another one is real-time transformations. They're complex and they're something that, you know, I think is um, most operators who have used something like Snowflake or Postgres SQL, they think that, you know, real-time transformations can't be that much harder. They're a little bit more difficult though. But AI makes them really easy too. So, it's another place that I think is really valuable. So, finally, I think there's a huge opportunity just when creating data pipelines to be able to use them differently than we have in the past, right? So, what I want to be doing when I'm creating a data pipeline is to prompt, "Hey, create this data pipeline to me for me." And then get back a draft which I can view in maybe a UI or a CLI using a YAML specification. And then finally, test it and make sure that the output looks like exactly what you'd expect. And and then be able to publish it when you're ready, check it into GitOps. And I think that's the future, right? Like that's where we're all going to be going to. And so, that's what we've been basically building. We're at the point right now where it's built through some Cloud Skills and an MCP server that we can we can show you here. And then I'm going to get into something that's I'll show you how it's actually built. But what you're seeing right here is this is application that we built using Estuary's MCP server and a few Cloud Skills. I see people already figured out what it is. So, basically, this is a real-time data pipeline. And it's powered by Estuary. It was built in about 30 minutes just using using basically Cloud. And so, the the primary amount of time was just getting that QR code to be the right size so you could actually take pictures of it from the background. So, if you all take out your phones and submit, you know, a an app, you'll see submit a submission. What you'll see is a few things here. You're submitting something into a Postgres database. And that Postgres database is managed by Neon. We're then doing using CDC to get it out of the Postgres database and we're putting it into a few different places. A BigQuery, Snowflake, and then a post another Postgres database. And that's all happening in real-time. You can track the latency here. It shows you, you know, P95 latency and all that stuff. So, that's um you know, you can see it all happening. The The point of this is that it's showing you three different use cases. One is analytics um with BigQuery and Snowflake. Another there is a live dashboard for operational data. And then a third is a vector store. So, it's an AI use case that's also um happening in real-time.

I love some of these. There's also two transformations that are being applied in real-time. And again, this is all just written by Claude. So, we um we ended up creating these transformations that are doing a very poor job of of uh cataloging what what you're asking for, but they weren't expecting pizza. Um and then another thing, the vector store can take um this and look for similar ideas. So, we can see other submissions that were similar. And the idea here is that that's more of like an AI use case, right? If we're vectorizing documents. Okay, so I want to show you what it actually looks like um in in the app. Um and so, I'm going to show you what was built by Claude. We have 51 submissions that have come into our Neon Postgres database here right now. Um and that's built in a way that can fan out. So, we can on the other side um in Estuary, just refresh it real quick. Uh you can see how many submissions have made it. I guess post Snowflake and BigQuery are a little slower than the Postgres side, which is actually part of the demo. That's what you're supposed to see. Um but what we're doing here is we have basically three transformation two transformations, uh Postgres database, and then these two Snowflake destinations. This one's vectorizing documents. It's using Python under the hood. We offer a Python transformation that you can use for uh you know, really nice um integration with AI type use cases. And then this enriched one is just using a SQL transformation under the hood. And so what I'm going to do is I'm going to get into um a quick video demo that shows how you used it. Cloud was down today, so I didn't want to take risks on trying to live code this on stage. I'll talk over it um and you can you can basically see what what this is doing. So, I put in a quick prompt um that says I want to create a Google Cloud Cloud SQL Postgres capture and capture all tables with CDC. And Cloud loads a skill. Um we have a bunch of publicly available Cloud skills that are on our GitHub right now. Um and an MSCP server that you can also find there as well. And so the cool thing about what this is doing is it's actually proposing um a YAML specification for you that you can review and choose um whether it makes sense, like decide if it makes sense to to be able to publish. Uh and then once you do, you just publish it and it it goes. Uh a quick transformation by a secondary Cloud skill here. Um and you know, while this is loading, I think it's important to say that like this is not the the end state we wanted it to be in. We don't want it to be through a CLI. So, in about 2 months you're going to be able to load a quick chatbot here in the app and be able to see it actually live and and um loading. So, I think that's going to be really nice. Um that's going to allow us to go, you know, back and forth in between environments um in real time. So, you know, Cloud here, what it's going to do is it's um it's loading the skill. It's going to be able to look at the actual tran- the actual schema of the source data, design um a derivation that's using um SQL around this, and this is the the schema of the source data, and then propose that to us. So, we end up getting the SQL transformation that's just using some case statements to uh to end up building. That's really it. From there, there's one more thing. I'm going to ask it to create a a materialization. That's what we call when we push to a destination system. And so Cloud's able to do that pretty easily, too. I don't need to go through the the whole video here, though. What what we've learned over the course of the process. We've been building Estuary for a few years now, and we were we were around and kind of wondering what we should do when Rag came out. I think we were all wondering what made the most sense. So, for us, what we what we learned is that really the building blocks are the most important things to focus on. There's There's going to be AI is going to move faster than anyone here can imagine, right? Any human can really imagine. We need to be able to provide things like Python transformations that can work seamlessly with the future formats that we're working with data in. And that's the goal for us. That's what we want to be able to offer.