The “Mirror” of Replication and the Possibility of “Intelligence”

This post began as a technical study note, but new ideas emerged while I was writing and took it in a different direction. Because it records the evolution of my thoughts rather than presenting a single polished argument, some of my views change along the way, and the technical details have been simplified. I therefore moved it from llm-algorithm to Reflections.

The main thread of that evolution appears in the Digression section.

Original title: dLLM: AR or Diffusion?

Introduction

Diffusion models originated as image-generation algorithms. During successive rounds of the forward process, noise is added to an image until it approaches Gaussian noise. The reverse process is then trained to remove that noise step by step, ultimately learning to generate images.

Applying this idea to LLMs gives us a generation paradigm different from today’s autoregressive models: text is produced through iterative diffusion.

Here is a survey for reference: Discrete Diffusion in Large Language and Multimodal Models: A Survey.

An Analogy?

As an analogy, today’s autoregressive model resembles a continue-the-story exercise. It writes one token at a time, serially, and every token depends logically and temporally on those generated before it.

A diffusion LLM is closer to a cloze test: take a passage, mask some of its words, and fill the gaps using the full context.

My blunt first impression was that current dLLMs still feel rather toy-like and offer limited results compared with autoregressive models. With the way my mind works, it is difficult to imagine a generation process without a logical or chain-of-thought sequence—one that produces the result as a whole—achieving strong “intelligence.” I cannot think of anything quite like it in nature. During each round of sentence diffusion, words before and after a position can see one another and coordinate through masking and remasking, but methodologically I remain something of an empiricist/Bayesian: I believe the future must be inferred from observations of the present.

Then again, perhaps this is like quantum mechanics. Superposition and entanglement do not seem to make intuitive sense; they feel thoroughly a priori/probabilistic. A dLLM has a similar a priori quality. It is not constrained by a causal mask, but observes and completes the whole text, which suggests considerable potential.

What follows was prompted by that thought and departs completely from the original topic.

Digression: Imagining the Future of LLMs

Kant, Critique of Pure Reason, page xvi:

Bisher nahm man an, alle unsere Erkenntnis müsse sich nach den Gegenständen richten. … Man versuche es daher einmal, ob wir nicht in den Aufgaben der Metaphysik damit besser fortkommen, daß wir annehmen, die Gegenstände müssen sich nach unserem Erkenntnis richten.

It has hitherto been assumed that all our knowledge must conform to objects. … Let us therefore try whether we may not make better progress in the tasks of metaphysics if we assume that objects must conform to our cognition.

Looking again at the LLM itself, it is built from scratch and plainly born from experience. Its operating mode is empirical as well. In mainstream autoregression, regression is itself a statistical concept: a summary of experience. When we train an LLM, are we trying to make it a simulator of the world, reproducing how the experienced world operates, or are we asking it to extract the laws of the universe—a priori and necessary principles?

On sober reflection—or from a skeptical point of view—it seems unlikely that something can surpass its source at a high level. In other words, artificial intelligence may never reach human intelligence. If it truly could, the development of intelligence would become recursively nested and eventually explode exponentially. I cannot imagine such a scene, so I can only doubt and reject it. Technically, a large language model also compresses the corpus that already exists. It may be highly knowledgeable, but from an information-theoretic perspective it remains inside humanity’s existing closed system—and perhaps only inside the linguistic system. Because the compression is lossy, it may not even reach humanity’s upper bound. Perhaps we should improve ourselves first. We already face the prospect of information collapse: AI-generated material flows onto the internet and may become training data for the next generation of AI. Without new “negative entropy” entering from outside the system, creativity could dry up. The more extravagant visions may owe a great deal to the media.

The greatest ideal that is also realistic and achievable today is to extract compressed good empirical regularities from training. With more parameters and increasingly sophisticated architectures and algorithms, we can try to fit as many situations in the textual world as possible.

Now consider the dream of discovering “universal laws.” The nutrient-poor media narrative undoubtedly bears some blame, but could genuine laws exist in some form? Could there be some other kind of law—something like Platonic Forms? Statistics can fit a probability distribution extremely well, yet in theory one can always identify a mismatch between the fit and the real thing. To obtain a truly a priori object, we would need to begin from first principles and reach the “thing in itself.” But in existing philosophical systems—particularly Western philosophy—the thing in itself is always inaccessible. I agree: we encounter phenomena directly and use reason to extract something from them. How could we guarantee that what we extract is the thing in itself? Is there a first thing behind it, and a zeroth thing behind that? Looking upward from our lower-order perspective, we cannot imagine what we would see. This is the sole and most fundamental, insoluble problem faced by the a priori vision of LLMs. The empirical vision, represented by current autoregression, instead rests on an infinite regress of justification because experience operates by induction. This reveals that I do not believe in empiricism—or at least not in the form of experience available to human epistemology. I lean toward the idea that the world is a priori and inaccessible.

But is there truly no path forward? The Manifold Hypothesis suggests that some high-dimensional features in nature can be mapped into a low-dimensional space. Superstring theory, for example, tells us that the universe may have far more than four dimensions. The computation required to process high-dimensional information grows explosively with dimension; consider the complexity of life on Earth. Humanity’s present 3D + 1D—space plus time—computational capacity has already reached a kind of peak. Although the brain accounts for only about 2% of body weight, it consumes roughly 20% of the body’s energy. Humanity nevertheless distinguished itself because we happened to evolve a system of abstraction that compresses the space. Machine computation, at least for straightforward calculations, is now extremely powerful. AI can readily operate in dimensions that humans struggle to understand and can only describe as a black box. In a large language model, semantic space is an extraordinarily complex manifold. After embeddings lift inputs into a higher-dimensional representation, the intermediate computation takes place in that high-dimensional space, invisible to us. Only when the LM head projects it back down does it enter a form we can understand.

I think the essence of today’s LLM training is using gradient descent to feel its way across this manifold. Perhaps human consciousness itself also emerges from high-dimensional computation, but the bandwidth of our brains constrains it. Our abstraction mechanism compresses information so aggressively that we cannot form the full range of associations available to an LLM. If language is supported by some low-dimensional structure that reflects the world, could an LLM, while approaching that manifold, accidentally develop an internal logic that goes “beyond the corpus”? In modeling the manifold of language that we chose as its feed, could it discover a higher-dimensional manifold than language itself—one that compresses the manifold of the universe with less loss than humans can manage?

Let us return to the more practical question that originally motivated this post: is the dLLM promising? After following the argument this far, I now expect more from it. An autoregressive LLM crawls toward the manifold one step at a time. Diffusion begins from high-dimensional noise and denoises it, a process that seems closer to a collapse from high dimensions into lower ones. The noise consists of data scattered across the wilderness of high-dimensional space; during training, it gathers back toward the surface of the manifold. A dLLM might therefore capture the highest level of human experience more effectively, or even stumble across a better manifold while approximating it. Yet language, the feedstock, limits our ability to tell whether that manifold is better; the model may merely pass through it.

Predictions That Follow

After the preceding reflection, and speaking as an optimistic skeptic, I do not deny that autoregressive LLMs will change our current way of life. I simply cannot imagine a more decisive long-term future for them. In both speed and accuracy, I think dLLMs may well surpass autoregressive models in the near future as training techniques improve. Their current limitations include the explosive activation costs required by reinforcement learning—which was the technical topic I had originally intended to study here.

Nor do I deny that language contains the essence of humanity. Thousands of years of logic, causality, and accumulated wisdom about how the world works have been abstracted into language. Language is a projection of the world as it is in itself. It is merely the best projection available to our cognition. Its discreteness appears to constrain higher-order understanding, and it is also what makes a dLLM difficult to train: during denoising, how can the model move smoothly among these discrete points until it encounters a good manifold?

Language is produced by the human process of perceiving the world and forming abstractions. Unfortunately—and discouragingly—that process receives the universe’s phenomena through sensory systems that act as dimensionality-reducing receivers. As carbon-based organisms, we possess no high-dimensional physical interface. Even so, through an extremely inefficient but remarkably robust iterative process, we evolved into the intelligent form we have today. What if a system could receive signals from the universe and also receive punishment from the universe rather than punishment from humans? Following the route of human evolution but equipped with more advanced hardware, could it evolve “intelligence”?

This is what embodied intelligence and world models are now attempting: providing AI with signals beyond the corpus, shifting machines from compressing human knowledge inward to receiving physical phenomena from the universe outward. Even if they cannot reach the thing in itself, could this create a form of existence one level beyond humanity?

At this point, the argument has effectively refuted my earlier claim that the limits of “bootstrapping” prevent AI from surpassing humans.

Perhaps, just as “God” created the environment in which we evolved, we may be unable to create intelligence directly but still able to prepare the cradle for a higher intelligence.

---

Afterword

This post grew out of a spontaneous train of thought, and many parts are intuitive rather than rigorous. When I have time, I hope to fill in more of the relevant background:

• The Manifold Hypothesis
• Superstring theory
• Neuroscience
• The philosophical arguments
• The dLLM discussion also requires more study. See limitations of dLLMs. Some argue that natural language actually contains a Markov structure and therefore strongly favors left-to-right modeling. Whether the difficulty lies in training or in the method itself remains an open question.