Today, I'm talking about what, in my opinion, is the biggest lie that the AI industry has told and continues to tell, why I believe that GPT-5 exposes it as a lie, and what I think the implications of that lie are.

Now, if I'm going to do that, of course, I'm going to need to start by telling you what the lie is: "These models are as bad as they are ever going to be, which I totally accept. Like, that is true." Wait! Was that Hank Green? What the Fu---

Welcome to the Internet of Bugs. My name is Carl. I've been a software professional for more than 35 years now. And I'm trying to do my part to make the Internet a less buggy place.

I used the Hank Green clip just now to show how pervasive the lie is and how often people hear it and repeat it, and how reasonable it seems.

I don't expect Hank Green to recognize that as a lie or call it out.

In part of that video, he discusses the idea of "staying in your lane" on the Internet and "debunking AI lies" is not Hank Green's lane.

It is, however, mine. And again, let me say for legal reasons that this video contains my opinion and I'm going to explain to you why I hold that opinion.

So let me reframe this argument with a quote from Sam Altman, who is the CEO of OpenAI.

"GPT-4 is the dumbest model any of you will ever have to use again by a lot."

So that quote was from a Q&A session at Stanford University in April of 2024.

There are a ton of other quotes I could have picked.

The idea constantly comes up during AI commentary and it's virtually always goes unchallenged.

Now, why do I consider that to be the biggest lie in AI?

Because the whole house of cards that props up the AI investment bubble is based on this premise.

The industry doesn't want you to think too much about however good or bad AI might seem at the moment.

They want you to take on faith that it will only ever get better.

And if they could actually pile improvement on top of improvement, out top of improvement, it seems almost plausible that eventually the AI might displace enough workers to generate enough revenue to justify its current valuations.

There are trillions of dollars of investment that are hanging on this idea.

And it's the basis for so many other lies like "AI will replace all programmers" or "AI will displace 80% of all workers" or "artificial super intelligence is going to happen soon and kill us all" or whatever.

If that foundation is faulty, if AI progress is not inevitable and next year's model isn't guaranteed to be better than this year's, then there's no reason to believe in any of the other stuff they're trying to sell you.

I think it's clear that this was a lie because now GPT-5 is out.

It's true that there are some people that have claimed that GPT-5 is actually smarter than GPT-4.

Altman has certainly made that claim.

But this report from Artificial Analysis shows that GPT-5 scored lower than o4-mini on the LiveBench and SciCode coding tests.

Now that's proof.

There are ways that GPT-5 is better, but there are also ways that it's worse.

But the most damning proof is that OpenAI was forced to re-release GPT-4.

Now remember, it's not like the claim was that GPT-5 would be a little bit better or somewhat better or ambiguously better or better in some ways but worse than others.

And the industry only needs for each generation of AI to be not better than the previous generation.

It needs to be exponentially better because they're promising investors the money they make will grow exponentially through 2030 at least.

The industry loves to pull up a graph of some arbitrary statistic related to AI over a cherry-picked range of the last few years to show how the progress looks exponential.

I've seen industries try to do that over and over.

If you go back decades, you'll see some graphs like these with several of those exponential looking spikes right before dips.

Statistically, for the vast majority of you watching this, this is your first AI boom.

Those of us that have lived through past AI booms, and the AI winners in between them, have seen this before.

Of course, most of the AI people are now saying, "Well, this time is different."

But then AI people always say that.

Maybe you remember how big a deal it was when IBM's Watson AI beat the world champion at Jeopardy?

Maybe you remember how IBM said that it was going to revolutionize healthcare?

AI promises have been repeated over and over for decades.

So given that the progress in the computer industry over the last 30-some odd years of the worldwide web, the idea of new technology that grows exponentially for decades doesn't seem out of the question.

After all, computers have pretty much only gotten faster and cheaper year after year for decades.

But what has gotten faster is the hardware, and AIs, despite the way the industry wants to talk about them, aren't brains, and they aren't hardware, they're software.

Let me say that again, they're just software.

AI folks talk about how LLMs are simulating human brain or the like, and they act like that means that the rules of software don't apply, but they do.

These AI's are just software running an algorithm or set of algorithms against a huge data store.

And the fact that the algorithm in question is derived from something found in nature is nothing special.

Every 3D game engine ever runs lots of algorithms derived from the natural world, from gravity to collision detection to ray tracing and first-person games certainly aren't immune to the performance problems that can plague any large software project.

Generative AIs are cool and useful, but there have been lots of cooler and more useful advances over the years.

I had a section in the video here going into where I think LLMs fit into the history of computing and how I disagree with the way the AI industry likes to position itself, but what I wanted to say grew to the point that it needed its own video, so that script is in progress and that'll happen, so subscribe if you're interested in seeing a video like that.

All software has limitations, and LLMs are no exception, and software doesn't automatically get faster and better every generation the way the hardware has for the last 60 or so years.

Has your phone always gotten faster every time you've updated the software?

Because mine sure hasn't.

Can you honestly say that every version of Microsoft Windows has been unambiguously better than the version before it?

F***ing Windows 98! Get Bill Gates in here!

You told us Windows 98 would be faster and more efficient with better access to the internet!

It is faster, over 5 million!

How about Microsoft Word? Excel? Adobe Photoshop? How about the Acrobat PDF reader?

Has that only ever gotten better over the decades?

Why do you think Adobe pushed everyone to subscription licenses?

Because people weren't buying the new version every year anymore, because the new versions just weren't better.

Software doesn't have a clear path to get better year over year.

The hardware it runs on can get faster, and sometimes that causes software to get faster too, but sometimes this year's software release is even slower than it was last year, even when running on new, faster computer.

This can happen because of software bloat, new features that are added that cause the old features to have to do work that you don't care about, the constant addition of stuff that is intended to give the company a reason to try to justify making you upgrade, 'til the company just gives up and declares "We're not supporting this version anymore. Switch to the new version, whether you like it or not, and pay us for the privilege, of course, otherwise you're going to get pwned by every bug that anyone finds from now on."

Windows 10 Anyone?

Making the hardware better is clearly understandable, might not be easy to do, and it often isn't, but it's pretty easy to measure whether you did it or not.

The goal is, "just like the last version, but faster, or able to do more things in parallel, or both."

Software isn't like that.

The idea of "better" isn't automatically clear.

It takes a lot of effort to define what you want software to do, and it takes time, and usually the faster you go, the sloppier the resulting software turns out to be.

AI has this issue.

Part of the problem with this "every version will be better" AI lie, is that it's not at all clear how to define "better", or "smarter", or "dumber", in an actionable way.

And there's a lot of discussion and even arguments about what "intelligence" even is.

OpenAI is said to define artificial general intelligence as, quote, "highly autonomous systems that outperform humans at most economically valuable work."

That's not helpful as a specification.

You can't make it to-do list out of that, and they are in a huge hurry, so this is not a setup that's great for success for them.

Commercial-sized software projects are always about trade-offs.

You can make this part go faster, but it affects the other parts.

A cache or an index can make reads faster, but it makes the write operation slower and more complicated.

You can add other new features, but that's one more thing the customer has to remember and keep track of, and the software becomes that much harder to use.

There's a horrible failure condition of software projects that I've seen happen, and I talk about a lot.

It's commonly referred to as "whack-a-mole."

Every time you fix a bug or add a feature over here in this part of the software, some new bug or failure pops up over there in some other part of the software.

There are a lot of things that professional software engineers do to try to prevent this from happening, but a lot of those techniques aren't available to machine learning and neural net-based projects for reasons that a lot of you probably don't care about.

I put up a video on my other channel about that and how it relates to Star Trek and the history of programming languages.

I'll put a link to that below if you're interested in a deeper dive.

We do know that tuning a model to be better at one thing often makes that model worse at other things.

The literature calls this effect 'negative transfer', and as models get larger, more and more training gets added, and each new training has more things that it might negatively affect, it can even lead to a problem called 'catastrophic forgetting'.

Guess what? It's not a good thing.

There are a number of lines of inquiry going on in the industry trying to mitigate this, but the thing you need to understand is this is not a trivial problem with an obvious answer.

There just aren't automatic wins on this path.

There's no reason to believe that it's always going to get better.

The AI companies have particular benchmarks that they love to point out with each new model, and I have no doubt that there will be some benchmark that each company can point to for each new model released to make the argument that it's an improvement, but higher scores on a cherry-picked collection of benchmarks is not the definition of smarter or better.

Like with GPT-5 scoring lower on LiveBench and SciCode, as I mentioned earlier, it's quite possible, if not likely, that there will be tasks that each new models do worse on than previous versions, not that you would expect the AI companies to advertise that.

OpenAI is starting to try to mitigate this somewhat, with routing under the covers.

Now, when you send a request to GPT, a router process will look at the request and choose which of several models to forward the request on to.

This will help them hide any places where particular models have regressed to poor behavior on certain tasks, but it's just a band aid.

So to recap, these AIs are software, not hardware, and there's no reason to accept the lie from the AI companies that every model from now on will be better than the one you have now, or the one you have then, and there are several reasons, including some academic research that indicate that, at least for some tasks, the opposite may well be true.

Don't believe the lie, don't repeat it, and call it out when you hear it, even if that means that sometimes you have to call out somebody like Hank Green for uncritically repeating industry talking points.

Because pretending that software always automagically gets better is how you end up with an even buggier Internet, and the Internet already has far too many bugs, and anyone who says differently is selling something, or they're repeating talking points from someone who's selling something.

Thanks for watching. Let's be careful out there.