Host: Hi everyone, we're going to get started. Welcome to AB talk. This event is sponsored by our company, Stasic. Stasic is a modern data platform for product development and we offer essential tools such as feature gates, experimentation, and analytics. Stasic is hosting this event because we provide services to a lot of AI companies such as OpenAI, Anthropic, and Character.AI, so we want to learn from the AI community about the best practices in product development in the GenAI era.

Let me do a quick intro about our speakers tonight. First, Hou is a technology veteran in Silicon Valley with a career that began by founding WiW's Network division. His diverse experience spans founding startups, holding executive positions, serving as an entrepreneur in residence at Greylock Partners, and mentoring at Stanford University. He recently started a YouTube channel called "Fight into Future."

Next, I'm pleased to introduce Yangqin (Jia Yangqing), the founder and CEO of Lepton AI. Prior to Lepton, Yangqin was a vice president at Alibaba and the director of AI infrastructure at Facebook. There he created Caffe and Caffe2, and he co-led the development of PyTorch, which has become the most popular machine learning framework.

Next, we have Jason, the founding engineer at Martian, a fast-growing LLM router. Martian router is about selecting the best AI solution for our tasks. Before Martian, Jason did research at Chicago Human+AI Lab and ByteDance. He leads one of the largest AI communities in the Bay Area.

And the last one, Jay, the co-founder and CTO of Opus Clip. His startup pivoted to the current AI video clipping product in 2023. In less than a year, Opus Clip is reaching 15 million annual recurring revenue (ARR) and is growing fast.

Jay: Some things change, some stay the same. You still need to understand deeply what the user needs, refer to theorems such as Jobs to Be Done. Also design and marketing is still very important. People might think we just need a line of input like ChatGPT and you don't need design—that's pretty much wrong.

Of course, there are things that are different. User tolerance is typically higher compared to incumbent spaces. But cost can be a lot of concern. We started by having 5x or 10x more cost per processing power in the beginning and we tried really hard to bring it down, otherwise we would just be working for OpenAI basically. Something that is very different is Prompt Engineering becomes a thing. Before you might have research engineers, now you need people who can write prompts—could be PMs or Ops people working with your AI engineers.

Hou: I think it's very different from the last generation. For enterprise software, precision and correctness are very important. The predictability and reproducibility is very important, and there's no reproducibility with GenAI. You have to rethink how you do it. You limit the input space and output space to make the result a lot more predictable. QA is another aspect. In traditional software, you sort of do a happy path and then figure out some corner cases. But in generative, it's unclear to me if I add one character, maybe the output is totally different.

Yangqin: In the old days computer science is all about going from the sloppy input (our abstract needs) into a logical world (code, JSON, APIs). Now suddenly this whole thing changed with ChatGPT. You find that this AI has become a black box and gives some sort of sloppy random output. Basically two things started emerging: one is building applications on top of the sloppy input/sloppy output situation. The second thing is we are starting to solidify the old way but using the technology—we still want to go from a sloppy world to a fixed logical world, like code generation or language-to-SQL. That's why we see a lot of work putting "handcuffs" on LLMs like JSON output and function calls.

Jason: The first is the difference in marginal cost. In traditional software, the cost is a sublinear function of usage. But for LLM, it's a linear function—call GPT-4 one time is 10 cents, two times is 20 cents. Another thing is to what extent AI applications can have network effects. For someone using OpenAI API, it's very hard for you to use your own data to improve the performance of the AI API you're calling.

Jay: The way to look at optimizing the cost is basically the same as Tesla. They started out by building an expensive car and then optimized the supply chain to drive the cost down. OpenAI has a product called Provisioned Throughput where they give you a discount (5% up to 50%). In my case, I hired a superstar in audio/video engineering and he wrote code to replace my gigantic FFMPEG code and saved 60 to 80% during the rendering pipeline.

Yangqin: I'm not particularly worried about cost. There's one simple rule in supply chain: predictability is rewarded with a lower price. If you tell Starbucks for the next whole year every morning at 9 a.m. I'm going to be asking for 100 cups of coffee, you will be getting a very good discount. Conventional wisdom of supply chain just kicks in and price is going to be significantly dropping down.

Jason: Aggregating retail demand is one big problem, but we have another smart routing logic. Let's say 90% of the math problems are simple, it can be answered by GPT-3.5, and 10% are hard. GPT-4 is 30 times more expensive. For each new prompt, we automatically recognize whether it's a hard or easy problem. We can save 85% of the cost.

Hou: In computation history, we went from IBM mainframe to PC and then to the cloud—a pendulum swing. Sam Altman's argument is: just use OpenAI API, you don't need to have your own model. But for some customers, there's no way they give you the data. Innovation will happen somewhere else. I feel it's going to be a pendulum back and forth.

Yangqin: Centralization and decentralization could be discussed on a different dimension. In the TOC world, it's very centralized—it's Google. In the TOB world, it's massively decentralized. We actually don't have the "Google of Enterprise Search." I believe TOB world things are going to be massively decentralized.

Jason: What determines the end state of industry is the economy of scale. Llama-3 8B is a good example—it's way better than many previous models and it's not being produced by OpenAI. There's a declining marginal return in putting more compute and resource. Also, AGI requires context and specific knowledge to be useful in specific scenarios. The best results are not delivered by the latest cutting-edge foundation model but rather the Compound AI Systems (agents, chain of thought) built on top. That process will be decentralized.

Jason: Evaluation is probably the first and the foremost important thing. Evaluation for GenAI is very difficult—it's similar to how we evaluate humans. We need a consistent, robust metric. At Martian, we have a combination of the human annotation, machine-assisted annotation, and automated evaluation.

Jay: We outpace our competitors because we use AI to evaluate AI. We use GPT-3.5 (stupid but cheap) to evaluate the quality of clips with multiple dimensions (is it interesting, does it have defects like sudden ending). We rank the candidates and only pick the 20%.

Yangqin: Numerical stability is a very important thing. This is not a new problem; the scientific computation or HPC world already knows that when you are doing numerical computation, things are just not possible to reproduce. AI is naturally just statistical—it's no longer the world of fixed input and fixed output. We just need to accept it and build security fences around those things.