On November 10th, Li Feifei personally authored an article, positing that the next frontier for generative AI is spatial intelligence.

In the article, she systematically explained for the first time what spatial intelligence is, why it is so important, and how to build world models capable of unlocking spatial intelligence.

Li Feifei also pointed out a critical flaw in current AI: although AI possesses vast amounts of abstract knowledge, it is almost entirely ignorant of common sense and spatial rules in the physical world.

This deficiency directly obstructs the main artery for AI's advancement.

She sounded the alarm, proposing that the true breakthrough for AI in the next decade will no longer be about accumulating text, but about unlocking spatial intelligence, as this is the ultimate capability that connects perception, imagination, and action.

After its publication, the article immediately sparked widespread discussion on social platforms.

In 1950, when computers were only used for automatic calculations and simple logical judgments, Alan Turing, in his article "Computing Machinery and Intelligence," posed a profound question: Can machines think?

At that time, this question seemed somewhat fantastical, but Turing, with his extraordinary imagination, insightfully recognized that intelligence might not be innate but could be constructed through technology.

It was this idea that gave rise to the scientific field later known as artificial intelligence.

Seventy-five years later, AI has indeed made revolutionary progress, especially in recent years, with generative AI, represented by large language models, moving from laboratories into our daily lives.

These AIs have demonstrated capabilities once thought impossible, but if we calmly examine them, we will find that current AI still has significant limitations.

Li Feifei mentioned several key issues in the article: firstly, the vision of autonomous robots is still far from being realized, and the robot scenarios we see in science fiction films are still confined to laboratories or highly restricted environments.

Secondly, AI's potential in accelerating scientific research has not yet been fully unleashed, with many critical steps in research still requiring manual completion by humans.

Finally, AI is still unable to truly understand and empower human creators; whether in molecular chemistry, architectural design, or film modeling, AI has not become a true helper because it lacks an understanding of space.

Why do these problems arise? Li Feifei believes the core reason is that current AI has not mastered spatial intelligence, a capability that is precisely the foundation of human cognition.

To understand this, we must first return to the fundamental question of how intelligence evolves.

When intelligence is mentioned, many people first think of abilities like language, logic, and mathematics, but in reality, spatial intelligence is a more fundamental and ancient form of intelligence.

In the article, Li Feifei refers to spatial intelligence as the scaffolding of human cognition; it supports our perception, understanding, reasoning, and creation of the world, much like scaffolding during building construction.

From an evolutionary perspective, the simple act of animals perceiving the world through their senses has already initiated the evolutionary path of spatial intelligence.

As evolution progressed, animal nervous systems became increasingly complex, gradually forming a core loop from perception to action.

Many scientists speculate that this loop drove the evolution of intelligence, ultimately making humans a species capable of perceiving, learning, thinking, and creating.

And spatial intelligence is at the core of this loop.

For each of us, spatial intelligence is actually a capability we use every day without conscious awareness.

For example, our perception of spatial layout, real-time judgment of spatial relationships, spatial intuition formed from long-term experience, and the most basic spatial reasoning all rely on complex spatial perception, inference, and prediction, which is precisely what current AI most lacks.

More importantly, spatial intelligence is also a key driving force behind the progress of human civilization.

Li Feifei cited three highly representative examples in the article.

The first example is the ancient Greek scholar Eratosthenes measuring the Earth's circumference.

In the 3rd century BCE, Eratosthenes observed an interesting phenomenon: on the summer solstice, in the region of Syene, Egypt, the sun would shine directly to the bottom of deep wells, and objects would cast no shadows.

At the same time, in Alexandria to the north, a vertical pole would cast a shadow forming a 7-degree angle.

Utilizing spatial intelligence, Eratosthenes transformed these two seemingly unrelated phenomena into a geometric problem, reasoning that the 7-degree angle was due to the Earth being spherical and the latitudinal difference between the two locations on its surface.

Thus, by measuring the distance from Alexandria to Syene and multiplying it by 50, he ultimately calculated the Earth's circumference to be approximately 40,000 kilometers.

This result is astonishingly close to the actual value measured by modern technology.

The second example is James Hargreaves inventing the Spinning Jenny.

In the mid-18th century, the British textile industry was still manual; a worker using a traditional spinning wheel could only spin one thread at a time, which was very inefficient.

While observing his wife spinning, Hargreaves suddenly had a spatial insight: what if multiple spindles were arranged side by side in the same frame, allowing one worker to spin multiple threads simultaneously?

Based on this idea, he invented the Spinning Jenny, increasing spinning efficiency eightfold.

This invention also directly spurred the Industrial Revolution, transitioning humanity from manual production to the era of machine production.

The third example is James Watson and Francis Crick discovering the DNA double helix structure.

In the 1950s, scientists already knew that DNA was genetic material, but its structure remained unknown.

Watson and Crick did not rely on complex instruments but sought answers by building 3D molecular models.

They used metal plates to represent DNA bases and metal wires to represent the chemical bonds connecting them, continuously adjusting the spatial arrangement of these components until they found a structure that matched all experimental data.

Ultimately, they discovered that DNA is a double helix structure composed of two antiparallel strands.

This discovery unveiled the mysteries of genetic inheritance and laid the foundation for modern molecular biology.

These three examples span thousands of years of history, but all point to the same conclusion: spatial intelligence is humanity's core ability to understand and transform the world.

Whether exploring natural laws, inventing new tools, or driving scientific revolutions, all depend on the perception, reasoning, and creation of spatial relationships.

The reason current AI cannot function in these areas is precisely because it lacks this capability.

So, what do we need to do to enable AI to possess spatial intelligence?

Li Feifei and her team proposed a key solution: building world models.

The core capability of such models is to understand, reason, generate, and interact with a world that is extremely complex semantically, physically, geometrically, and dynamically, far exceeding the capabilities of current large language models.

So, what core capabilities must a true world model possess? Li Feifei clearly outlined three points in the article.

The first core capability is generative: it must be able to generate endless, diverse simulated worlds based on semantic or perceptual instructions, and these worlds must satisfy perceptual, geometric, and physical consistency.

In this regard, a key question currently being explored in academia is: how should world models represent this spatial information?

Should it use implicit representation, indirectly storing spatial information through neural network parameters, or explicit representation, directly storing objects' 3D coordinates, shapes, and physical properties?

Li Feifei's view on this is that both are needed.

The second core capability is multimodal: when humans understand the world, we simultaneously use multiple senses such as sight, smell, touch, and hearing.

This ability to integrate multiple modalities is crucial for our understanding of the world.

Similarly, world models also need to possess this capability.

Although current multimodal models can process various inputs, their core remains a language model, with modalities like images and videos serving only as auxiliary inputs, not truly deeply integrated with the language modality.

The design philosophy of world models, however, is natively multimodal, meaning that various modalities are equal within the model, and they collectively serve the core goal of understanding and building the world.

The third core capability is interactivity: if we input an action or goal into the model, it should be able to output the next state of the world, and this state must conform to the world's historical state, physical laws, and semantic logic.

This is one of the most critical capabilities of world models, as it directly corresponds to human agency.

More advanced interactive capabilities also include goal-driven action planning, meaning that a world model must not only predict the state of a goal but also plan what actions are needed to achieve that goal.

This is also the core foundation for future autonomous robotic actions.

If robots can plan actions through world models, they will be able to autonomously complete tasks in complex environments without requiring human reprogramming.

Li Feifei believes that building world models with these three capabilities is the decisive challenge for AI in the next decade, because the difficulty of this challenge far exceeds that of building large language models.

This is because language is a one-dimensional, sequential signal, whereas the world is a three-dimensional, dynamic, and multimodal system.

Simulating such a system requires solving a series of technical challenges.

World Labs, founded by Li Feifei, is an institution dedicated to world model research.

In the article, she shared the three core challenges currently facing world model research, along with some of her team's exploration results.

Challenge One: Designing a Universal Training Task Function

We know that a key reason for the success of large language models is their simple, elegant, and universal training task: predicting the next token.

However, for training world models, such a universal task function does not yet exist.

Why is designing this task function so difficult? Because world models need to process complex systems involving space + time + multiple modalities, rather than one-dimensional text.

This involves more continuous dimensions, and they are more complex.

Li Feifei believes that the training task function for world models must reflect geometric and physical laws, as these are the inherent properties of the world.

A possible task function might be predicting the next state of the world, but the specific design of this task function requires further exploration.

How should the world state be defined? How should prediction accuracy be measured? These questions require continuous experimentation and optimization by researchers.

Challenge Two: How to Obtain Large-Scale, High-Quality Training Data

The training data for large language models primarily consists of text data from the internet, which is vast in quantity and low in acquisition cost.

However, training data for world models needs to include spatial, physical, and multimodal information, making it much more difficult to acquire.

Current sources of training data primarily include three categories: the first is image and video data from the internet, such as the vast number of videos on YouTube and Instagram, which contain rich spatial and dynamic information.

However, the problem is that most of this data is two-dimensional and lacks critical information such as depth, tactile feedback, and physical properties.

Therefore, the key to research is how to extract three-dimensional spatial information from two-dimensional images/videos.

The second source is synthetic data, which refers to spatially annotated data generated through computer simulations.

For example, generating a large number of virtual scenes using a 3D engine and labeling each object's 3D coordinates, physical properties, and dynamic rules.

The advantage of this type of data is its precise annotation and strong controllability, which can compensate for the shortcomings of real-world data.

However, synthetic data also has drawbacks; it may have discrepancies with the real world, leading to suboptimal model performance in real-world scenarios.

The third source is complementary multimodal data, such as depth maps obtained from depth cameras, tactile data from tactile sensors, and motion data from motion capture devices.

This data can provide richer spatial information for world models, but its acquisition cost is high.

Li Feifei believes that future training data for world models will inevitably be a fusion of real data, synthetic data, and multimodal data.

The key to research is how to build model architectures that can efficiently utilize this data, how to enable models to learn the world's diversity from real data, precise spatial and physical laws from synthetic data, and cross-modal relationships from multimodal data.

Challenge Three: The Need for New Model Architectures and Representation Learning Methods

Current multimodal and video generation models are mostly based on the Transformer architecture and process images and videos by converting them into one-dimensional or two-dimensional token sequences.

This method has a fatal flaw: it destroys the three-dimensional structure of space and the dynamic continuity of time, preventing the model from truly understanding spatial relationships.

Therefore, world models require the development of new architectures that go beyond current paradigms.

Li Feifei mentioned two possible directions: the first is 3D/4D-aware tokenization methods, meaning that images and videos would no longer be converted into 2D tokens but into 3D spatial tokens or 4D spatiotemporal tokens.

This way, the model could directly process the three-dimensional structure of space and dynamic changes over time, rather than inferring them indirectly through 2D pixels.

The second is a spatial memory mechanism, meaning that the model needs to have dedicated memory modules to store spatial information about the world, such as what objects are in a room, and what their positions, shapes, and physical properties are.

This way, when the model processes dynamic scenes, it can maintain scene consistency by recalling spatial memories.

The World Labs team has already made some explorations in this area, such as their recently developed Real-Time Generative Frame-based Model (RTFM).

The core innovation of this model is the use of spatially-based frames as spatial memory; that is, the model stores generated world scenes as a series of frames with spatial coordinates.

When the next state needs to be generated, the model calls upon these frames to ensure spatial consistency.

Currently, RTFM can achieve efficient real-time generation while maintaining spatial consistency in the generated world, which is a significant advancement in world model research.

In addition to technical challenges, World Labs also showcased the first application product of world models: the Marble platform.

This is a world model tool for creators that can generate and maintain consistent 3D environments through multimodal input.

Currently, Marble has been opened for testing to a select group of users.

Dr. Li Feifei stated that the team is working diligently to optimize it and aims to open it to the public as soon as possible.

After discussing so many technical details, some friends might ask, what changes can spatial intelligence and world models ultimately bring to our lives?

In the article, Li Feifei outlined the application prospects of spatial intelligence across three time dimensions: short-term, medium-term, and long-term.

The first important application area for spatial intelligence is creativity.

Li Feifei believes that spatial intelligence will fundamentally change how we create and experience narratives, moving from traditional two-dimensional storytelling to effortlessly generating explorable 3D narrative worlds.

Design processes will also undergo an efficiency revolution; with spatial intelligence tools, designers can quickly generate 3D models using text and sketches, allowing for real-time adjustments.

Immersive experiences will also become more widespread, breaking spatial limitations and fostering closer connections between people.

The second important application area for spatial intelligence is robotics.

Li Feifei believes that for robots to become collaborative partners with humans, they must possess spatial intelligence, as robots need to move, manipulate objects, and interact with humans in the physical world, all of which depend on an understanding of space.

The biggest bottleneck in current robotics technology is the lack of spatial intelligence, and the emergence of world models will drive breakthroughs in robotics technology in three aspects.

The first breakthrough is solving the problem of scarce robot training data; world models can simulate countless virtual training scenarios, allowing robots to learn quickly in virtual environments.

The second breakthrough is enabling robots to become collaborative partners who understand humans; future robots will need to possess both spatial intelligence and semantic understanding capabilities to comprehend human needs and intentions.

The third breakthrough is expanding the forms and application scenarios of robots; current robots are mostly humanoid robots or industrial robotic arms, with relatively singular forms.

In the future, they will take on diverse forms according to the needs of different application scenarios.

Beyond creativity and robotics, spatial intelligence will also have a profound impact on long-term fields such as scientific research, healthcare, and education.

In the field of scientific research, spatial intelligence will become a tool for accelerating discovery; for example, scientists can use world models to simulate the three-dimensional dynamic changes of global climate systems, or the relationship between atomic spatial arrangements and material properties, or even the three-dimensional structure and evolution of galaxies.

The core of these simulations lies in spatial relationships and dynamic laws, which is precisely where world models excel.

Through spatial intelligence, scientists can overcome experimental limitations and explore phenomena that cannot be replicated in laboratories.

In the healthcare sector, spatial intelligence will serve as an assistant to enhance diagnostic and treatment levels.

Li Feifei's team at Stanford University has been conducting research in this area for years, such as accelerating drug development, improving diagnostic accuracy, and optimizing patient care.

In the field of education, spatial intelligence will become a tool for revolutionizing learning methods.

Spatial intelligence can make learning more immersive and interactive, transforming it from rote memorization into hands-on experience and active exploration, thereby greatly enhancing learning efficiency and interest.

In the conclusion of the article, Li Feifei mentioned that she has been dedicated to the field of AI for 25 years, consistently inspired by Turing's vision.

The research into spatial intelligence is based on this vision; it is not about AI replacing human creativity, but about AI becoming a partner to humanity, helping us overcome our limitations and achieve goals previously unattainable.

Finally, I would like to conclude with a quote from Li Feifei: she said that nearly 500 million years have passed since nature bestowed the first glimmer of spatial intelligence upon ancient animals.

We are fortunate to be a generation of technologists who may soon be able to endow machines with the same capabilities, and have the privilege of using these capabilities to bring well-being to people around the world.