[Instructor]: Hi everyone, welcome to another lecture for CS230 Deep Learning. Today we're going to talk about enhancing large language model applications, and I call this lecture Beyond LLM. It has a lot of newer content, and the idea behind this lecture is we started to learn about neurons, and then we learned about layers, and then we learned about deep neural networks, and then we learned a little bit about how to structure projects in C3, and now we're going one level beyond into what would it look like if you were building agentic AI systems at work, in a startup, in a company, and it's probably one of the more practical lectures.

[Instructor]: Again, the goal is not to build a product end-to-end in the next hour or so, but rather to tell you all the techniques that AI engineers have cracked, figured out, or exploring, so that after the class you have sort of the breadth of view of different prompting techniques, different agentic workflows, multi-agent systems, EVAs, and then when you want to dive deeper, you have the baggage to dive deeper and learn faster about it.

[Instructor]: Okay, let's try to make it as interactive as possible, as usual. You are all familiar with pre-trained models like GPT 3.5 Turbo or GPT 4.0. What's the limitation of using just a base model? What are the typical issues that might arise as you're using a vanilla pre-trained model?

[Student]: Lacks some domain knowledge.

[Instructor]: You're perfectly right. You know, you we we had a group of students a few years ago was not LLM related, but you know they were building a an autonomous farming device or vehicle that had a camera underneath taking pictures of crops to determine if the crop is sick or not, if it should be thrown away, like if it should be if it should be used or not, and and that data set is not a data set you find out there, and the base model or a pre-trained computer vision model would lack that knowledge, of course. What else?

[Student]: Trained on high quality pictures, but in reality, the pictures are very dark on the camera.

[Instructor]: In fact, yes, the distribution of the real world might differ as we've seen with GANs from the training set, and that might create an issue with pre-trained models. Lacks current information. The LLM is not up to date. One story that I found funny, it's from probably three years ago or maybe more, five years ago, where during his first presidency, President Trump one day tweeted "covefeve." Just "covefeve," and it was probably a typo. That word did not exist. The LLMs, in fact, that Twitter was running at the time could not recognize that word. This is an example of nowadays, especially on social media, there's so many new trends, and it's very hard to retrain an LLM to match the new trend.

[Instructor]: The first line of optimization, which is prompting methods. I'm giving you a very simple prompt here. Summarize this document. You can actually improve these prompts by doing something like summarize this 10 page scientific paper on renewable energy in five bullet points focusing on key findings and implications for policymakers.

[Instructor]: Then there's chain of thoughts. This is actually a popular method that's been shown in in research that it improves. Approach the task step by step and do not skip any step. Few shot prompts are very popular. Zero shot refers to the fact that it's not being given any example, into a few shot prompts where the model is given in the prompt a set of examples.

[Instructor]: Chaining is the most popular technique out of everything we've seen so far in in prompt engineering. It's not chain of thought. This is different. This is chaining complex prompt to improve performance. One advantage of chaining is you would separate the prompts so that you can debug them separately. A first prompt is extract the key issues. Second prompt, draft an outline. Prompt number three, write the full response.

[Instructor]: Let's talk about RAGs. RAGs is important. It's a very common interview question. Standalone LLMs have challenges including hallucinations, cutoff dates, and lack of sources. RAG integrates with external knowledge sources.

[Instructor]: The way it works is you have your knowledge base of a bunch of documents. You use an embedding to embed those documents into lower dimensional representations. You will store those representation into a database called a vector database. You run a retrieval process, find the relevant documents. Then you add them to the user query with a system prompt. The prompt template can be answer user query based on list of documents.

[Instructor]: Andrew actually coined the term agentic AI workflows. Calling everything an agent doesn't do it justice. So Andrew says let's call these agentic workflows because in practice it's a bunch of prompts with tools with additional resources, API calls that ultimately are put in a workflow.

[Instructor]: I want to talk about the pirating shift. Traditional software deals with structured data. Now more and more companies are handling freeform text. The software itself used to be deterministic. Now you have a lot of software that is fuzzy, and fuzzy software creates so many issues. You sort of want to think about your software as like your manager. If I was to delegate my product to be done by a group of humans, what would be those roles?

[Instructor]: What are the core core components of an agent. The prompts. That travel agent also has a context management system, which is essentially the memory. That context management system might include a core memory or working memory and an archival memory. And then you have the tools. The tools can include APIs. Anthropic also talks about resources. Resources is data that is sitting somewhere.

[Instructor]: You probably have heard the term MCP or model context protocol that was coined by Anthropic. MCP, it's really about putting a system in the middle that would make it simpler for your LLM to communicate with that endpoint. It's agent to agent communication, which allows more scalability.

[Instructor]: Let's do a case study with evals. Let's say your product manager's manager asks you to build an AI agent for customer support. How would you know if your system works? One is an end-to-end metric. You look at the user satisfaction at the end. You can also do a component-based approach. LLM traces are very important. If they don't have LLM traces, it is pretty hard to debug an LLM system.

[Instructor]: Another way to look at it is what is objective versus what is subjective. An objective example would be the LLM extracted the wrong order ID. You also have subjective stuff. You probably want to do either human rating or LLM as judges.

[Instructor]: Why do we need a multi-agent workflow? The main advantage of a multi-agent system is going to be parallelism. The other advantage is an agent can be reused. You might have a hierarchical setup here.

[Instructor]: What's next in AI. I wanted to call out a couple of trends. Ilias Sutskever raises that question about are we plateauing or not? It's probably architecture search. Whoever discovered transformers had a tremendous impact. The other set of gains is from multimodality. Being good at images makes you better at text. Finally, the velocity at which things are moving. The half life of skill is so low. You want to come out of the class with a good breath and then have the ability to go deep whenever you need.