从停滞的 POC 到生产环境：利用非结构化数据治理构建高质量企业级 AI 智能体

Jeff Carr: OK. Hey guys, let's get started. Can everyone hear me? Is everything okay? Perfectly. My name is Jeff Carr. I am a full-time customer service engineer at DZee Labs. As you can see on the slide, it says that this is provided to you by Collibra. Collibra acquired DZee Labs last summer. Collibra is more of a data management tool for structured agents and AI-based models. So, the gap with Collibra was that we didn't have unstructured data, and that's where DZee shines. So, today I have Leo Platzer with me.

Leo Platzer: Yes. Hello everyone. I am Leo. I was previously the CTO at DZee Labs, which, as Jeff mentioned, was acquired by Collibra last year. And as you can see, we are both twins. We also have our own hats that we are giving away at our booth, so come on in and get them. Whoever came first was served. So today we’re talking about how we can move from stagnant POC to production with DZee data curation. So we really focus on unstructured data.

Jeff Carr: Our demo scenario, which I usually talk about, is the manufacturing company North River Manufacturing. You can see that there are 40,000 employees there. They operate in 20 different countries. In fact, they are trying to have millions of documents in SharePoint. They've been accumulating them for years, and they need to figure out which ones are relevant for their chatbot. They are investing heavily in artificial intelligence to improve operational efficiency. And their main goal this year is to create an artificial intelligence chatbot for legal transactions. Now they want to help legal and procurement teams answer questions about supplier contracts, partnership agreements, and it needs to happen in real time. So, some of the questions they're trying to answer are: what are the termination provisions for a particular vendor, what deals are ending this quarter, what deals are ending next quarter, and what partnerships have, um, exclusivity terms. So, among the people on the teams we worked with is, um, Sarah. She is the head of the data and artificial intelligence department. So she is responsible for accuracy. There is also Jim. He is an artificial intelligence engineer. He just needs to know what files I need. I have thousands and thousands of files in SharePoint. I need to find only the relevant ones, without sensitive information. Priya is the head of the knowledge management department. She wants to make sure that no confidential information is leaked, and to make sure that if there is any data that is, um, classified, that is also redacted. Then there's Michael. He is the director of legal operations. He needs a chatbot to provide accurate answers to his team. So he's from business. And finally, there's Jackson. So Jackson, the head of enterprise transformation, and this is his flagship use case, but he also has 15 other use cases that he needs to deal with, right? So he wants to have a structure, a repeatable, successful process. He doesn't want to do it just once. He wants to do this for several different use cases.

Jeff Carr: So what they've been able to do so far is take 40, about 40 different files, pre-selected files, and create a little mini-pilot project where they can do parsing, fragmentation, optical character recognition (OCR), embedding, and then put that into a vector database and a knowledge graph, and they've gotten pretty good answers, right? Seems pretty simple. You can do this with 40 files. But when they try to do this with many, many different files, with over 80,000 files, right? Now suddenly it gets a little more complicated. And so some of the difficulties they faced was that I couldn't find the files I needed. I don't know where my sensitive data is located in SharePoint. I can't trust the quality of the data. So, my experience...I've worked at both Informatica and Databricks. So for me, data quality usually consists of six different dimensions: data quality, completeness, relevance, accuracy. But when you're dealing with documents, it's a little different. I want to see if there are duplicates of information, conflicting facts, right? Leo will talk about this, he has a very good slide that shows why this is so important. And also about the freshness of the data. And also, everyone talks about context, right? So when we want to tag metadata, we want to use it in our chatbot, directly in the database, to make it more relevant, to have better accuracy in the chatbot. So you can see some of the problems they faced, right? They get the wrong answers. Chatbots are leaking confidential information. It takes a long time to go to production because they can't find the files. They spend too much time searching for files instead of writing code and doing what they really enjoy, right? This is difficult to maintain. I work with another telecommunications company and they have six or seven different SMEs and they're trying to process 20,000 HTML files but they can't keep up, right? So it's very fragile, very difficult. And we provide the opportunity to automate this entire process.

Jeff Carr: So here you can see what they want, they want to try to access and use over 80,000 different files. So they went from 40 files to over 80,000. And they really want to find which files are relevant to their chatbot, which are high quality, which are relevant in terms of knowledge. They don't want to use a version where the contract is old and expired, right? And they want to enrich their chatbot with metadata that they have tagged. And of course, they don't want any sensitive data. So, we're providing an opportunity to do that here, right? We can access the data, um, here. We can understand what this data is. We can determine which files are relevant, perform optical character recognition, fragmentation, etc. So let me show you a quick demonstration. So here you see SharePoint, very similar to what North River has. They have a whole SharePoint site full of different, you know, there are call transcripts, there are PDFs, there are Excel, there are Word documents. And I don't know which ones are relevant, which ones focus on legal topics. Now, when we look at Daisy, we've already opened the program right here, and I see that I've already uploaded 339 files. And now I see both the page level or the fragment level and the file level. I see all the information right here. And then there's this thing we call taxonomy. A taxonomy is a grouping of tags or a tag tree. So I got 339 files and now I can either manually create a custom tag. I can use reusable tags from my tag library, or I can use AI to automatically suggest which tags I should have in the first place, right? I can use artificial intelligence to tell you that these tags are relevant. So here I just enter my tag name, data, document categorization, give a query, and this query is what we're going to send to LLM to determine if this file, how should we tag this file? Now, if I know what I'm looking for, I can enter the classification value. This is optional. And again, if I can use artificial intelligence to automatically suggest all of these tags for me. Now, it should also be noted that if I have my taxonomy here, my valid values ​​already, I can upload them via CSV. And so, once we do that, I see that I've started to build this tag tree.

Jeff Carr: Now, another thing I can do is have conditions for my tag tree. So, remember I said I only needed legal and contractual documents? This is what they are looking for. Now I can have subtags. So, just legal and contract files, I want to pull more data by contract type. Is this a supplier contract? Is this part of a non-disclosure agreement? Right? Is this a materials and time contract? And now that we're adding this, you can see that we have different branches in our tag tree. And I can also have unconditional tags. I can have any number of different tags. Now we are going to generate metadata. So, I choose which files I want. I can choose which tags I want. I can say "generate metadata," and we'll run it, and you'll see the results here. So now I see the document category. I can hover the mouse cursor. There are engineering documents. I have my contracts and legal documents, right? I have 68 different files. And if I click on any of them, it automatically filters me. Now I can see the evidence too. Why did the AI tag this? Where in the document did he find evidence that this was a legal contract, right? And I can also use "thumbs up" and "thumbs down", so there's little step-by-step learning here. So I can give an example if I want. Now, as you can notice, I also have filters, meaning I have two different filters. I quickly reduced the number of files from 339 to 20. So we also have the ability to scan for sensitivity. So, whether it's PII, PHI, um, it could be anything. I was in London at the Gartner conference, um, a few months ago, and some people from Finland came in and wanted to know if we could find a Finnish national ID card? I don't speak Finnish, so I put it there. I did this demo, and we can work not only on the basis of artificial intelligence, we can work on the basis of patterns with context, and as a result I was able to figure out the syntax of the regular expression um, but also added some Finnish words that I think are probably related to the Finnish national ID card. So, there will be fewer false positives.

Jeff Carr: Okay, let me continue. So, another thing we can do is address quality. So, again, there's a quality dashboard for us where we can look and see different groups for um for duplication, for conflicting information, uh, and then we'll use artificial intelligence to automatically suggest which ones to keep. Now, another thing we can also do is again for freshness, we can say, "I only need files within a certain number of days or within a certain date range." And the last thing we can do for data quality is, you know, the metadata snippets that I absolutely need, and then we figure out, you know, out of 339 files, maybe you really need 200 files. And then from there you can create a slice of the data. And then we can also give it context. Here it is. So, I'll watch another video. And we also have a live demo for you that Leo will do in a second. So, in this case, I can now use a data slice. So my data slice can be filtered based on certain criteria that I want. So, in this case I have 14 files. These are only files that relate to legal information. And now I'm copying my use case description. What I can do is say, here's what I'm trying to build, and I can either do a flat taxonomy or multiple levels in my tag tree. And then when I launch it, it will automatically suggest those tags to me. A person participates in the cycle, so I can choose which ones I like, which ones I want, which ones are relevant. But again, it's all done for me based on what I'm trying to do. We also gave it context based on the content of the files themselves. And then here he even creates a classification meaning for me. And then, if we want, when we run, when we generate metadata, we're in the process of working, now we can see this confidential information. Okay, here's all the different information we have. And again, I can see evidence at the file level and at the fragment level. I can see the reliability indicator. So, coming back from artificial intelligence, what did the model return? I don't need to guess, you know, if it's 100% certain, 95% certain, that's why. So, you can be very confident in the AI-ready data product that you guys are going to create. Good. And finally, what we can do for delivery and service, so you know, another thing that's really important is that with our data slice technology, we have workflows that we can put on a set schedule so that they automatically refresh. So you want to make sure your chatbot never gets outdated. So if there are new records or, excuse me, new files that are added to SharePoint or deleted, we automatically pick them up and add them on a set schedule to my data slice or to my AI-ready data product. And then our SDK will pick that up in the chatbot, and then when we load the vector database, you'll always have the most up-to-date metadata. So what does this mean for North River? So, essentially, they were able to reduce data preparation time from 4 months to a few days. They were able to increase the use of data for AI. They improved the accuracy of their chatbots because they were able to not only identify which files to use, but also the metadata from the files, they were able to insert it and use it through our SDK in their chatbot, right? And they also reduced the risk of non-compliance. They knew where all the sensitive data was and made sure it wasn't part of their data slice, it wasn't filtered out in their data slice. Okay, so I'll pass this on to my brother from another mother, Leo, and he'll give you the context.

Leo Platzer: Yes. Hello everyone. I just wanted to say, so what, right? Technically, you deduplicate your data, you reduce the search space, right? You clean them, you have a fresh look. What does this really mean? Yes? So if we look at a vector space, right? When you ask a question, it comes in and you have duplicate data with the same information or maybe even a different version of the document with conflicting information, right? Remember your messages in Slack. Today a person says this, tomorrow a person says that, right? The truth changes over time. If it changes, if it stays in your knowledge base, in your search base, your top-K will always be full of practically irrelevant information, right? So, for example, in Deezy, something that is considered a duplicate is either another piece of context, like your database has changed and I can use it, or it's just redundant and we'll remove it from the knowledge base. The second fragment is the same as before. If your information is old, right? If you're suddenly working with data from before the 2000s, if you're working with information that was relevant only last week and is irrelevant this week, then in the future, some use cases, when agents actually go into production, let's call it, for the most difficult use cases, it has negative consequences.

Leo Platzer: And we've done a lot of evaluations on what the impact is of ensuring that the quality of your unstructured data is maintained, is consistent, and is always processed deterministically the same way, right? And we actually see that if your raw corpus is okay, imagine you have a thousand files, and 30% of your files are outdated or duplicated, it could be that up to 80% of your actual context that your agent uses to answer the question is filled with outdated information, meaning that it was actually using 80% of completely redundant, unnecessary information and knowledge. And this needs to be managed. We saw that by running the same tasks, i.e. this is a chatbot for a rack, it wasn't running a real, say, cloud code task or codebase for a coding task, it was just a multi-step rack assessment. And we saw that if we look at the recall of one, I mean, we practically double the recall and overall improve our accuracy on these kinds of tasks by 10 to 15%. And why is this so important? This is so important because it has nothing to do with the use case you are working with. This is true regardless of what you actually use the data for, right? Because if you really know what you want to use your data for, you can apply relevance and all the other data quality metrics, that's right. But duplication and freshness are true regardless of the circumstances.

Leo Platzer: And yesterday, right? There are many contextual topics here, many contextual flags. Hmm, I would like to demonstrate very, very quickly how you would use DCS DK to prepare a context repository. And I'm going to run this in real time. So this essentially gets all the metadata that we have about a particular SharePoint site. We then contextualize each folder on that SharePoint site, and also create an index for all the folders, so that your cloud code, for example, or your codebase doesn't even need to look at individual files to figure out what to look for, but just looks at the contextual MD file. So what you see here is essentially, with DC you can curate contextual files where you know directly what exactly is in each folder, right? What is its purpose? Right? When do you want to use it? For example, this is an unambiguous reflection of what I actually have in SharePoint. But now, instead of looking at those 23 files to figure out if I should even look at that folder, I know exactly what types of documents I have, what they're about, what the key topics are, when to use that folder, and what sample questions. So, that's what all these use cases look like today, right? How can you leverage simple use cases like your in-app chatbot, or leverage simple options to actually improve performance and reduce, for example, the token cost for your agents. Hmm, but yeah, basically, I'll turn it over to Jeff again to summarize this whole presentation.

Jeff Carr: Thank you, Leo. Good. So guys, we're almost out of time. Our stand is far behind in the second stage. Please come in. Bring any different uses you have. Hmm, we're happy to have discussions, do live demonstrations, talk about your problems with unstructured documents. Before we go, I want to let you guys know that we have weekly POCs. So we are happy to demonstrate our technology. Again, you know, you guys bring in your business problems that you're trying to solve with AI, bring in some sample data, a bunch of different files, and then we can generate metadata for each individual file. We can get a full report on your data privacy, a data quality report, as well. And then we'll show you how we can create a complete taxonomy based on your specific use cases, your specific files. So, I appreciate you all spending time with us. And finally, we only have one box of these hats left, so come in and get them. We will be at the back. Thanks, guys. Thank you.