The Physicist Working to Build Science-Literate AI
Miles Cranmer wants artificial intelligence to help us make scientific breakthroughs.
Philipp Ammon for Quanta Magazine
Introduction
Physics dazzled Miles Cranmer from an early age. His grandfather, a physics professor at the University of Toronto, gave him books on the subject, and his parents took him to open houses at universities near their home in southern Ontario, Canada. The Perimeter Institute for Theoretical Physics was a favorite. “I remember someone talking about infinity when I was super young, and it was so cool to me,” Cranmer said. In high school, he interned at the University of Waterloo’s Institute for Quantum Computing — “the best summer of my life at that point.” Soon he began studying physics as an undergraduate at McGill University.
Then one night during his second year, the 19-year-old Cranmer read an interview with Lee Smolin in Scientific American in which the eminent theoretical physicist claimed it would “take generations” to reconcile quantum theory and relativity. “That just tripped something in my brain,” Cranmer said. “I can’t have that — it needs to go faster.” And for him, the only way to speed up the timeline of scientific progress was with artificial intelligence. “That night was a moment where I decided, ‘We have to do AI for science.’” He began studying machine learning, eventually fusing it with his doctoral research in astrophysics at Princeton University.
Nearly a decade later, Cranmer (now at the University of Cambridge) has seen AI begin to transform science, but not nearly as much as he envisions. Single-purpose systems like AlphaFold can generate scientific predictions with revolutionary accuracy, but researchers still lack “foundation models” designed for general scientific discovery. These models would work more like a scientifically accurate version of ChatGPT, flexibly generating simulations and predictions across multiple research areas. In 2023, Cranmer and more than two dozen other scientists launched the Polymathic AI initiative to begin developing these foundation models.
The first step, Cranmer said, is equipping the model with the scientific skills that still elude most state-of-the-art AI systems. “Some people wanted to do a language model for astrophysics, but I was really skeptical about this,” he recalled. “If you’re simulating massive fluid systems, being bad at general numerical processing” — as large language models arguably are — “is not going to cut it.” Neural networks also struggle to distill their predictions into tidy equations (like E = mc2), and the scientific data necessary for training them isn’t as plentiful on the internet as the raw text and video that ChatGPT and other generative AI models train on.
Machines are good at analyzing certain research data, but Cranmer wants them to generalize from totally new information as well.
Philipp Ammon for Quanta Magazine
Still, Cranmer believes these hurdles are surmountable. “I’m nowhere near as smart as Einstein or other great scientists,” he said. “So if I’m thinking about what I can do to accelerate the entire pace of research, it’s to really push machine learning. That’s what I can contribute.”
Quanta spoke with Cranmer about giving AI scientific memory, extracting insights from neural networks, and what scientists and programmers may soon have in common. The interview has been condensed and edited for clarity.
AI researchers won two Nobel Prizes last year. Don’t we already have “AI for science”? What’s missing?
The biggest challenge, if you abstract away everything, is that machine learning is bad at “out-of-distribution” prediction. That means that if you have a new data point that’s unlike anything you’ve seen before, a machine learning model will tend to do badly. This is the major weakness of machine learning, compared to traditional science.
Cranmer during a coffee break at the Institute of Astronomy Hoyle Building at the University of Cambridge.
Philipp Ammon for Quanta Magazine
Think of Einstein’s general relativity. Physicists had no conception of a black hole in 1915. The math just logically produces that prediction. And we can see evidence that matches it over a hundred years later. This is something that machine learning right now could not do at all — that kind of extrapolation is just out of the question.
I’ve always been really interested in improving that part of machine learning, because I think that’s the missing piece.
But neural networks are just equations, too. How come Einstein’s math gives us models of the universe, but AI’s math can’t?
I would say that the second kind of math, machine learning, doesn’t have memory, while the first kind does. In the physical sciences, if you propose a new theory, all previous observations must still be satisfied by the new framework. We have to obey the same rules we’ve discovered before. Whereas in machine learning, you start from scratch every time you train a model.
So how do we put memory, in this abstract sense of “knowledge accumulation,” into machine learning? One way is using symbolic rules, where we can impose the patterns that appear in physical frameworks. For example, I know that if I walk into another room, the physics doesn’t change. A machine learning model doesn’t know that.
“Machine learning is really good at problems that have massive amounts of data,” said Cranmer, “but for problems that have very few examples it sucks.”
Philipp Ammon for Quanta Magazine
How do you get a neural network to play by these rules?
I’ve spent the last four years working on software called PySR. It’s a symbolic regression library, so it learns equations that match a data set. Rather than a prediction being hidden away in a neural network, this gives you a way of translating the neural network’s behavior into a symbolic equation in a more familiar language to scientists. Forcing the machine learning model to use symbolic mathematics is basically a way of giving it a bias toward the existing ideas that we’ve constructed physics out of.
There’s multiple benefits from this. The equations you get are very interpretable, and they tend to generalize and give you much better out-of-distribution predictions. The downside is that these algorithms are really, really computationally expensive. If you had infinite resources, it would be perfect.
And do the scientific “foundation models” you’re working on get around this problem?
With symbolic regression, you are giving a neural network the symbols that scientists use as a library that it can build things with. Another way is much more data-driven: giving a library of examples. Our approach in Polymathic AI is to take a model and train it on all the science data you can get. You’re still starting from scratch, but you’ve given it so much data that you’re sort of anchoring its predictions.
Cranmer at the Centre of Mathematical Sciences in Cambridge.
Philipp Ammon for Quanta Magazine
I think this is why language models like ChatGPT seem OK at out-of-distribution scenarios: They’ve kind of turned everything into an in-distribution prediction problem, because they’ve been pretrained on so many different things. When ChatGPT came out, we were all really excited to think about how this type of tool could be used in science. And as we kept discussing it, this idea crystallized of pretraining a model not on language, but on scientific numerical data sets.
That was the hardest challenge for us. Getting high-quality scientific data, like spectra of stars, is not as easy as just unleashing robots on the internet to scrape websites for training data, like AI companies do. Luckily, in astronomy, a lot of the data is publicly available — you just need to put it all in a uniform format. We released two data sets: the Well for numerical physics simulations, and Multimodal Universe for astronomical observations. These data sets offer a massive amount of scientific data as a basis to actually build these foundation models.
Will they “hallucinate” — confidently make up false answers — like other AI models do?
The whole reason for doing this pretraining is to bake in a sense of what is physically reasonable. If the model gets into a new situation that it’s never seen before, rather than making some insane prediction, it’s going to do something physically reasonable.
Cranmer uses symbolic rules to instill in machines a greater understanding of past discoveries, helping them process new data and produce more understandable results.
Philipp Ammon for Quanta Magazine
It doesn’t eliminate the problem, but it greatly improves it. I think that’s where symbolic regression could come in, as well: translating parts of the model to analytic mathematical expressions where you can make guarantees.
What do you see scientists doing with a foundation model?
Machine learning is really good at problems that have massive amounts of data, but for problems that have very few examples it sucks. That’s where I’m really excited about using a foundation model, because it gives us a way to attack those types of low-data problems. You can train the model on the simulations, so it gets most of the physics involved. But then you only need to add a few experiments to tune its predictions. It’s not going to be perfect, but it’s going to be better than a machine learning model trained from scratch. So from a few real-world data points, you can extract more science than you could before. That’s the idea.
Could this end up automating the work of scientists?
I definitely think this type of tool will automate many tasks. My goal is to make all scientists capable of doing much, much more. It might change the definition of what a scientist is, but I think that definition has been changing throughout history already.
It’s the same thing with language models. They’re not replacing programmers, they’re just changing the definition of what programming is, in the same way that writing in Python isn’t replacing someone who writes compilers. It’s just different levels of abstraction.
In that sense, I’m not worried about any kind of AI replacing scientists. It just lets us do more with the same amount of time — that’s what I’m really excited about. Understanding the universe doesn’t really have an end to it. It’ll just keep going, and we’ll keep learning more and more and more.
