In Hilbert Space, All Things Are Quantumly Possible
At the heart of quantum mechanics lie a few sacred rules for how to use the theory. First and foremost is, roughly, that thou shalt not think about ordinary objects presently whizzing through ordinary space. Rather, quantum mechanics predicts — in exquisite detail — all the possible ways that an object might turn out to be in the future. Exploring those possible futures requires tracking an entirely different mathematical object — an arrow known as a vector, one oriented in an expansive, alien domain.
These arrows aren’t pointing at locations. “It’s a much more abstract space than that,” said Lucien Hardy, a physicist at the Perimeter Institute for Theoretical Physics in Waterloo, Canada. They’re “really pointing in a direction in a possibility space.”
This possibility space is called Hilbert space, and it acts as the primary arena for quantum physics.
The early quantum pioneers didn’t realize — at first — that the arcane math that strikingly captured the conduct of atoms had left the real world behind. It took a visionary mathematical physicist, John von Neumann, to recognize and define the quantum world as a Hilbert space. Once he did, exploring the ins and outs of Hilbert space would lead physicists to a deeper, more unified understanding of quantum physics.
Here’s how von Neumann’s first commandment of quantum physics came to be, and how to understand it.
What Is Hilbert Space?
Von Neumann’s commandments, or axioms, were his way of making sense of the two distinct forms of quantum mechanics developed back-to-back in the 1920s. First came Werner Heisenberg’s “matrix mechanics” in 1925. It used inscrutable tables and, in later formulations, interminable towers of numbers to calculate the odds that an electron circling an atom would jump to a higher or lower orbit. The next year, Erwin Schrödinger introduced his “wave mechanics.” It used waves to track, for instance, the probability of a particle being found at a certain location in space. While the pictures evoked by these two physicists looked completely distinct, they yielded identical predictions. Heisenberg and Schrödinger had come up with two radically different incarnations of one theory. But what was that theory?
The question fascinated David Hilbert, a renowned mathematician who had devoted much of his life to rebuilding physics on a sturdy foundation of crisp axioms. He got von Neumann thinking about the problem in the mid-1920s. In 1927 the 23-year-old prodigy — building on insights from Paul Dirac — solved it in a single-author trilogy of papers.
The mathematician David Hilbert sought a mathematical structure that would unify the different forms of quantum mechanics.
MacTutor
“The ideas specifically were von Neumann’s, but the inspiration — why do you axiomatize and what for — this is something that he took from Hilbert,” said Leo Corry, a historian of mathematics.
These papers laid out the rules for quantum mechanics, carefully defining the theory’s central objects and how they behaved. Von Neumann showed that Heisenberg’s towers and Schrödinger’s waves were reflections of the same entity, just as 0.5 and ½ indicate the same point on the number line. They both represented the main character in quantum mechanics: the quantum state.
Everything has a state. A coin can read heads or tails. A grandfather clock’s bob can take on any number of positions as it swings. Most of physics amounts to capturing an object’s state and predicting how it will change.
Von Neumann’s quantum rules complicate the notion of a state. Before you observe a quantum object, it does not have a fixed set of properties, such as a specific position. Instead it has a combination of possible properties unique to quantum mechanics — a “quantum superposition.” A superposition combines, for instance, all possible places the particle might end up being. Those possibilities can be precise and informative; perhaps there is a 99% chance you’ll find your particle to your left and a 1% chance you’ll find it to your right. You know how to place your bets, but you can’t know for sure if you’ve won until you check.
Von Neumann rendered the quantum state as a mathematical arrow called a vector. This arrow points in some direction through a space capturing all the possible futures of any quantum object — a Hilbert space.
Imagine a quantum traffic light with three possible states — red, yellow, or green. Its arrow exists in a three-dimensional Hilbert space, where the three axes represent the three possible future colors. Until the moment the light is observed, it doesn’t have a color, but rather a mixture of possible colors. So its arrow points into the space’s central region. The more closely the arrow aligns with, say, the red axis, the more likely the light is to shine red.
The state of any object, from an electron to a galaxy, can be captured by such a vector, pointing in some direction through such a Hilbert space. This is von Neumann’s first rule of quantum mechanics.
What Happens in Hilbert Space?
An arrow moves through Hilbert space in one of two ways. Von Neumann’s other commandments specify how.
The first possibility corresponds to what happens before an observation. As the world influences the object, changing its state, the arrow turns smoothly through Hilbert space. It might get closer to the green axis, which would make our traffic light more likely to be measured as green, or to red or yellow. The point is that all this happens smoothly and predictably.
Then, if you actually observe the system, the vector will instantly and randomly snap onto either the red, yellow, or green axis. The more aligned it is with one axis, the more likely it is to snap to that axis instead of the others, but its fate is ultimately unpredictable. Let’s say it goes green. You’ll observe a green light, and there is now a 100% chance that it will still be green in subsequent measurements, because the arrow is fully aligned with the green axis. The quantum superposition is no more.
What Properties Define Hilbert Space?
The more possible futures an object has, the bigger its Hilbert space. A coinlike particle with two possible futures is a “qubit,” the computational building block of quantum computers. It has a two-dimensional Hilbert space. Our three-color traffic light has a three-dimensional Hilbert space. But that’s just the beginning. A freely floating particle could be found in any location in the universe, so its Hilbert space must span an infinite number of dimensions.
This size — whether it’s two dimensions or an infinite number — is the only fundamental feature of a Hilbert space, according to von Neumann’s rules. The axes are arbitrary and imagined by us; they aren’t intrinsic to the space.
Consider an electron. It has one state, one arrow, pointing in a vast Hilbert space. Its Hilbert space spans all possible measurements — energy, position, momentum, etc. If you are curious where the electron might be, you can mark the space with the axes that represent possible positions. If you are wondering where the particle might be going, you apply a different set of axes, those representing possible momenta. No matter which measurement you intend to make, the underlying Hilbert space remains the same.
This freedom to carve up Hilbert space as we see fit is what allowed Heisenberg and Schrödinger to come up with two distinct versions of the same theory. Heisenberg’s picture essentially put in axes and let them rotate around the vector, while Schrödinger’s picture did the opposite: It put in a fixed set of axes and let the vector rotate relative to them. They were two completely different mathematical perspectives on the same arrows, in the same Hilbert spaces.
In the first of his 1927 papers, von Neumann laid out two mathematical criteria that defined such a space. First, it had to be “complete.” It couldn’t be missing any regions or points. And second, you had to be able to calculate the alignment between a state and an axis, which you can visualize by imagining a light shining straight down onto an arrow so it casts a shadow on an axis. (The longer the shadow, the more aligned with that axis the arrow is.) The space had to allow for this operation, known as an inner product. Any space with these two features, no matter its size or origin, was a Hilbert space.
“This was a major step in creating what we call Hilbert space quantum mechanics,” said Miklós Rédei, a philosopher of physics at the London School of Economics. “It’s a beautiful example of how mathematical generalization or abstraction takes place.”
Von Neumann referred to these abstract spaces as Hilbert spaces because his mentor Hilbert had been the first mathematician to explore specific spaces with infinite dimensions in the early 1900s. Hilbert’s work had relied on those spaces being complete and having an inner product, but he didn’t think of them as examples of a more general class of spaces until his protégé grouped them together. The older mathematician may have been surprised to find his name attached to this new mathematical structure. “Dr. von Neumann, I am really curious to know what these Hilbert spaces are, after all,” Hilbert reportedly asked during a 1929 lecture.
Is Hilbert Space Real or Just an Abstraction?
A century after the birth of quantum mechanics, the theory has left physicists in an awkward position. We live in a world where objects change position as they move through three dimensions of physical space. But our most fundamental theory takes place somewhere else, in von Neumann’s vast realm of possibilities. What does that imply about the reality of our world, or that of Hilbert space?
To Sean Carroll, a philosopher and physicist at Johns Hopkins University, the message is clear. If quantum mechanics is the fundamental theory of nature, then Hilbert space should be considered the fundamental theater of reality, he argued in a 2022 paper. One of his lines of research seeks to distill our familiar world from the disorienting Hilbert space that encapsulates all the ways the universe could possibly be.
Other physicists take a more pragmatic stance. Jonathan Sorce, a physicist at Princeton University, says that Hilbert space is a handy mathematical construction that is remarkably useful for describing many quantum systems — but not all of them. He belongs to a community of researchers searching for a mathematical construction that can describe the fabric of space and time as a quantum object. Such a theory is a prerequisite for answering big questions such as what goes on at the heart of a black hole.
Physicists asking these questions have recently focused on an even more abstract space that seems especially well suited for their purposes. This kind of space is made up of the things you could do to a Hilbert space, such as slicing it up in different ways or rotating one slice into another. In this arena, they have found that black holes seem a bit less mysterious.
This sort of über-space is known today as a von Neumann algebra. Von Neumann himself helped develop it as a potential remedy for some logical inconsistencies with Hilbert space that troubled him. “I would like to make a confession which may seem immoral: I do not believe in Hilbert space anymore,” he wrote in a 1935 letter while exploring the virtues of algebras.
Sorce, for his part, doesn’t share von Neumann’s desire for one space to rule them all. He’s content to use whichever mathematical construction best suits the quantum object he’s studying. Often it’s a Hilbert space. Sometimes it’s a von Neumann algebra. And occasionally it might even be one of the many other spaces mathematicians have cooked up over the last century.
“There’s a whole zoo of these things,” he said.
