生成式AI在科技史上的位置：一个反主流观点

I did a video a year or so ago about how AI hype was out of control.

I regret to inform you, it hasn't gotten any better.

So that video I did on hype focused on how the AI industry use demos and press releases to try to sell exaggerated promises about their products that they can actually deliver.

This video focuses on how the AI industry tries to sell a trivialized version of the past to make the current generation of AI technology seem to have a more important place in history than it actually does.

So here's Sam Altman talking to the New York Times:

Everybody has their analogy for what AI is like, you know, I saw Sundar is going to be here.

He talks about it like electricity.

A whole bunch of other people talk about like the industrial revolution.

Some people talk about like the Renaissance.

The one I like is the transistor.

To me, comparing generative AI to electricity, the industrial revolution, the renaissance or the transistor is just obviously ridiculous.

But there are so few details in that kind of statement and the events are so long ago that it's difficult to even have an informed discussion about it, which means these claims go largely unchallenged and of course, repeated endlessly and uncritically by reporters and influencers.

The primary problem with statements like these is that the definition of AI, e.g. artificial intelligence, is very ambiguous and intentionally so, which makes it really difficult to argue with.

Whatever AI might become in 100 years or 1000 years or 10,000 years may well be as impactful as some fundamental technology.

On the other hand, what OpenAI can get done before their money runs out is an entirely different thing.

That's one of the reasons why it's so hard to argue with these points because what they mean by the term "AI" can change from sentence to sentence deliberately.

So instead today, I'm going to discuss AI in the context of a much more concrete statement made by someone else in the AI industry, a statement about how AI compares to other advances in software over the decades.

Because if I can put AI in the context of the last few decades of the tech industry, and if that helps convince you that AI is less impactful than several other software innovations that were enabled by transistors, that hopefully will go a long way to explain why I think that comparing AI to something like transistors or even less possibly electricity or the industrial evolution is just a load of Bullsh-----

Welcome to the Internet of Bugs.

My name is Carl.

I've been a software professional for more than 35 years now, and I've been talking about software here on YouTube for a year and a half or so, and I'm trying to do my part to make the Internet a less Buggy place, which includes trying to explain to people by believing the hype about generative AI is a really, really bad idea.

Now, instead of trying to argue with a vague statement like "AI is like a transistor," I'm going to discuss a much less vague, much more modest statement from a presentation given at a recent conference for the startup accelerator Y Combinator.

So here's that quote from the presentation by the former head of AI at Tesla:

"I think, roughly speaking, software has not changed much on such a fundamental level for 70 years, and then it's changed, I think, about twice, quite rapidly in the last few years."

Now, I want to be clear that I'm not trying to attack this speaker specifically.

The presentation he gave that I'm going to talk about today was pretty tame compared to Altman's quote I gave earlier, but it is an example of the kind of statements that are regularly made by AI luminaries that make the general public think that AI is a much bigger deal than it actually is.

And that in turn, that tricks the hype believing public into making poor decisions that they and society may well regret later.

That some of us already regret.

I'm picking this presentation as a jumping off point because this statement has a much more definite timeframe, 70 years.

It doesn't use the vague AI marketing term, which can mean whatever the speaker wants to this sentence.

Instead, it's about changes that have happened in the last few years, which means that the AI relevant to this quote and the AI I'll be discussing in this video is generative AI, which is also called large language models.

Also, since this quote is about software, it's much better fit from my channel and my audience and my experience.

And although I'm only in my fifties and I haven't lived through the last 70 years, I have lived through most of it and I've been writing software for more than half that time.

So it's something I'm feel qualified to talk about.

So today I'm going to make a case that not only has software changed drastically at a fundamental level several times over the last 70 years, but that the changes we are living through now and what we've seen over the last few years are by comparison, not actually even all that fundamental.

I'm even going to try to put AI in the context of where I think it ought to be in the history of technology and what it really means in my opinion, but we'll get to more of that later.

So 70 years ago was about mid 1955, but I don't want to have to worry about accidentally talking about something that happened 70 and a half years ago.

So I'm going to start my date at 1956.

For my cutoff date for the last few years, I'm going to pick 2015.

Again, I'm going to try to avoid partial years.

So I'm going to say that from January 1st, 1956 to December 31st, 2015 is well within the 70 years that supposedly software has not changed much.

And the period from January 1st, 2016 to now should definitely cover the last few years, supposedly during which it's changed twice quite rapidly.

January of 2016 is roughly a year and a half before the "attention is all you need" paper that kicked off the latest general of AI craze.

So I think it's a generous cutoff date.

And I'm going to make the case that the changes since 2016 have honestly been pretty boring in comparison to the changes from the 60ish years before that.

I'm going to make the case that, in context, large language models aren't actually all that fundamental compared to many of the changes from the previous era.

So step one, let's talk about the state of software in 1956.

In 1956, experiments were being done trying to figure out how to get a computer to take input from a keyboard instead of the punch cards that were used at the time.

1956 was the year the first reference manuals published for the first third generation language or compiled language Fortran, although it would be a while before the language actually stabilized.

Algol was two years in the future.

Cobol was three years in the future.

The first freshman computer programming course in the US would begin at Carnegie Mellon in two years.

And the first general purpose time sharing system that supported software development, something that students could actually do their homework on, would be released at MIT five years from 1956.

Now, I don't know about you, but I would consider the ability to type into a computer to be pretty fundamental to developing software.

I don't have anywhere near enough patience to deal with punch cards.

So that means I'd say there's already one fundamental thing that's happened in the last 70 years.

Additionally, I think the introduction of the first ever college class and how to program is pretty fundamental developing software.

And I think actually having a computer that exists that you can practice programming on as a student is pretty fundamental to developing software.

But let me take you through more of it, because I think it will help illustrate where our current generative AI really fits into history.

I'm going to skip a ton of stuff.

I'm not going to go strictly chronologically.

I'm going to go through these pretty quickly, and I'm going to oversimplify a lot for the sake of time, but I'm hoping it will be interesting to you to see the connections.

Our next stop is the 1970s.

So I can introduce you to an old concept. We don't talk about much anymore, which is called data processing back when it was all about punch cards.

It wasn't programming in the way we think of it now, really.

It was mostly just mathematical operations, except for the error handling.

As you can imagine, when you are reading data of a bunch of physical cards that are punched by hand by someone who didn't have a screen, typos were inevitable, and so were folds and tears and cards, dust in the sensors, et cetera, et cetera.

So the most complicated part of the instructions for this kind of thing was making sure that you didn't get bad results or crashes from bad data.

We're going to keep revisiting the idea of data processing as we go through this list, almost like a different track happening in parallel with software development for reasons that will become clear at the end.

The next development in the software development side we're going to talk about was the rise of third generation languages like C.

The compiler turned fewer, much simpler and easier to read instructions and many more lower level, highly detailed instructions that the computer could understand.

This was expected to make programmers more productive, and it did.

And some people thought that wouldn't mean fewer programmers would be needed.

But the additional capabilities that the new language is enabled meant software could do more.

And so we actually ended up with more programmers.

Let's just say that's a theme we'll come back to you later.

Back in the data processing land, relational databases were invented.

Early on, there were multiple query languages.

I'm actually the first one I learned wasn't SQL.

But now everyone is pretty much standardized on SQL.

I can't overstate how much these databases change modern life.

The vast majority of the queryable data in the entire world has been kept in this kind of database for the majority of the lives of everyone watching this.

If you drill down on so much of the modern world, especially anything that has anything to do with money, you'll likely find a SQL database at the core of it.

One of the really useful concepts in all of these databases is called constraints.

It's a way of validating data before it's written to the database, which drastically speeds up a lot of the querying and error checking than in the punch card days from a programming perspective.

Relational databases allowed the intersection of the kind of data processing tasks that have been done by hand or by dedicated tabulating machines and the new third gen programming languages opening up a myriad of possibilities for uncountable numbers of real world tasks that were now possible that wouldn't have been before programming language wise.

Our next advances object oriented programming, which, well, seemed to be like a great idea at the time.

And then some of it even was.

Again, the goal here was to make programmers more productive so we could do more faster with less.

There were a lot of things at OO enabled that would have been much harder to do without it, more on that in a bit.

And again, it just made more demand for programmers.

Back in data land, I want to highlight indexed full text search.

These indexing techniques allowed for the first time for so much of the sum of human knowledge to be accessed quickly.

Most of you probably take this for granted now, but it's one of those people that grew up with card catalogs and libraries. I can tell you this was a fundamental change in the very process of how people learn.

But it's also significant in that it helps me illustrate an interesting programming problem that comes up here, which is scale.

Up until now, it's theoretically possible to verify the correct answer for most programming tasks that had been written to this point.

But full tech search puts you firmly in the realm of processing huge stacks of data on punch cards.

You can test your code on small data sets and verify that for those, it gets the right answers.

But once you turn it loose on the huge data sets, then you can't verify it's correct anymore.

So for example, imagine you're doing full tech search across all the books in a library and you're looking for things that contain a certain phrase.

If there's one book that does contain that phrase, but it's missing from the search output, for whatever reason, unless you happen to stumble across that phrase in that book, you don't know that the answer you got from the computer program was incomplete.

So object oriented programming led the way to modern graphical user interfaces.

We had some before, but the ability to make a Window object that has size properties, et cetera, et cetera made it a lot more usable.

With this, we also got integrated development environments and combined with some of the search tech, we got autocompletes, syntax highlighting, which was incredibly helpful.

Most of you have no idea how much slower you program when you have to remember and correctly spell every single thing in your code to get it to compile.

Next up on the data track is data warehousing.

The idea of putting lots and lots of stuff at a giant database and then figuring out after the fact what you want to get out of it.

Again, error handling and filtering out garbage data is a huge problem.

A heavy use of indexing is the only way to make search times tolerable.

And once you get to a certain amount of data, you don't have a good way of verifying if your code is still working correctly.

That said, they're incredibly useful.

We'll come back to data warehouses in a while.

Now, it's time to talk about the internet.

I can make a whole list of the underlying fundamental pieces of tech here, TCP/IP, spanning tree, CDMA, BGP.

I'm going to put encryption and key exchange here too.

But for software development purposes, one of the big software engineering advances that enabled was dependencies and package managers.

The first one I ran into was called CPAN.

It was for the Perl language, then Ant/Maven for Java, but you're likely more familiar with Python's pip or NodeJS's NPM or the like.

By helping developers download code instead of having to write it themselves, it was hoped that we could be more productive again.

I know that some of you are freaked out now by watching AI generate a bunch of code from a simple prompt.

Let me assure you the productivity boost that the industry got by going from each of us having to write all of our own code from scratch, to including thousands of lines of purpose-built code into your project just by typing one package name into a config file, was at least as dramatic, if not more.

Now, this thing is going to be weird, but next to the data space, I want to talk about lossy compression.

Classically, this is like JPEGs and MP3s and MP4s, audio and video type stuff.

Basically, it's way for us to store a lot less data than we started with.

And then when we pull it back out of storage, we make up the parts of it we didn't actually store.

So it hopefully looks close enough.

If we throw away too much information or if the original we're trying to compress has certain characteristics, we'll get a lot of compression artifacts, which sometimes is good enough depending on what you're trying to do and sometimes destroys the experience, depending on what we're trying to accomplish.

But it's not limited to media.

Think about something like trying to store a lot of historical stock price data.

You can have fewer samples per time span and you can round the values before you shore them.

If you're just trying to analyze trends, that's probably good enough.

But it's important that you understand its limitations.

Without this, though, we just would not be able to store or transmit the amount of data that crosses the internet all the time.

And it would mean no podcasts, no YouTube and our internet would be a much less interesting place.

So it's a big deal.

There are a bunch of other fundamental technologies I could have mentioned, like no-code websites, like WordPress and Squarespace, Map-Reduce for distributed web scale computing, virtual machines, cloud tech, that lets software run without being constrained by the structure of the physical hardware it's running on.

Not to mention smartphones and mobile computing, but we've gotten through enough that I can give you an alternative placement for the large language models.

Now, I would argue that LLMs are just the next generation of data processing tech using lossy compression, searching against a cloud scale index.

What some people call "thinking" is really just search with a very rich unstructured query language and the so-called "hallucinations" are effectively just compression artifacts, where it's making up words to fill in the gaps, the way that compressed JPEGs make up the pixels or the way that compressed video makes up chunks of image frames.

It's useful tech, but it's not revolutionary, nor is it the most impactful thing since the 1950s.

It's just another iteration of what we've been doing for years and not nearly as important as a lot of the changes I've talked about today.

Now, I should say I'm not the first to come up with this idea.

I first encountered the idea in an essay from Ted Ching called "ChatGPT ChatGPT is a blurry JPEG of the web," which was in the New Yorker.

Link below.

Ted is, in my opinion, by far the best speculative short story writer of my generation, you might have seen or heard of the movie Arrival from 2016, which is based on his short story "story of your life."

If you haven't read his stuff, I highly recommend you give him a read.

He's just amazing.

So the implications of LLMs being lossy compression are interesting.

First off, the idea that you have a searchable image of the whole internet that will fit on a hard drive or in the RAM of a large GPU is absolutely mind-blowing.

Now, it's not a faithful image by any stretch.

And in fact, it's a pretty crappy copy.

But depending on what you're trying to do, that's often fine.

If you want to write fiction, that's great.

If you want to have an imaginary pen pal, it's a good fit for that.

If you want boilerplate text for a business proposal or boilerplate code that has 35 duplicate entries on Stack Overflow, it's a fast and easy way to get it.

If you want an excerpt from a study guide for solving a tech interview programming riddle or an international mathematics competition problem or college or law entrance exams or professional certifications exams, it's got you covered.

Also, because of the way the information is encoded, it does an amazing job of handling vague and vigorous or unstructured queries, which lowers the bar for what skills a person needs to have to put this tech to you.

On the other hand, if the consequences for mistakes are high, be it for medical diagnoses, mental health therapy, case law in legal court documents, evidence and criminal cases, code that needs to be secure against hackers or commands for killer robots, it's not a good choice at all.

We've been down this road before.

When x-rays and CAT scans went digital, there was a lot of concern about potential misdiagnosis if the images were stored using lossy compression.

And there's been a lot of literature and studies examining that problem since.

The shame is the industry keeps pretending, and apparently sometimes even believing, that these things are living beings or some such crap.

It keeps us from being able to have honest conversations about what uses this technology are good fits for or poor fits for.

There's a whole lot more I have to say about LLMs and how they are just lossy database searchers with unstructured query language.

But it requires me to get in the weeds about data warehouses, data lakes, and talk about some of the large data projects I've worked on.

So I'm putting that in a video on my secondary channel.

I'm not done with that video at this time that this publishes, but it will be on the next one I upload on that channel.

And once I get it uploaded, I'll put a link to it in the description below.

Feel free to subscribe to my other channel if you want to know when that video or other new videos go up there.

There's a link to the other channel in this video's description.

And while you're at it, feel free to subscribe to this channel too.

I realized for a lot of people that the current crop of AI is seen unique and incredibly disruptive from an employment standpoint of nothing else.

But from the point of view of history, this isn't unprecedented at all.

Can you imagine how much paperwork and time was spent with filing cabinets and manual bookkeeping that was all replaced by SQL databases?

I've seen estimates as high as 10% of the entire US labor force.

But unfortunately, I can't find a good source for that.

There are just too many job titles and classifications and it's too vague to know for sure.

But here's a number I do have a good source for: just so you have something to think about.

Guess how many people used to be employed just as telephone operators?

About 1.4 million people, about 400,000 of those working for the phone coming itself and about a million working for other employees out of a total US labor force of roughly 59 million people.

That's one of every 42 working people in the US or telephone operators at the peak.

And that's just one single occupation that's now been completely automated out of existence out of the many occupations that were radically reduced by the march of technology.

The biggest difference between the generative AI wave now and the several previous technology adoption cycles isn't the level of disruption.

It's the levels of marketing, propaganda and gaslighting.

Oh, and the hype.

Don't forget the hype.

Well, as if you could forget the hype.

So from now on, when you're talking about or thinking about good or bad applications of generative AI, try thinking about it as a crappy, blurry copy of the internet instead of some mythical sci-fi soon to be superhuman and see if that doesn't help you put it in a better context.

Thanks for watching.

Let's be careful out there.