How ‘Event Scripts’ Structure Our Personal Memories

Baldassano’s research builds upon two key innovations. Over the past decade, a group of cognitive neuroscientists has honed a clever method that uses movies to study the brain’s record of realistic experience, and Baldassano has pioneered advances in decoding the complex data recorded during these cinematic sessions using machine learning.

His approach brought “a lot of sophistication” to these kinds of studies, said Brice Kuhl, a University of Oregon neuroscientist who was not involved in the research. “It’s had a huge impact on the field.”

The analyses have generated a new understanding of how the human brain constructs narrative memories. Nearly the entire brain is involved, contradicting earlier ideas that placed memory in specific brain regions. And memories are built in temporal pieces, each of which ranges from a second to a minute in length.

The brain places those pieces onto the scaffolds of event scripts. “It’s all a construction,” said the neuroscientist Zachariah Reagh of Washington University in St. Louis, who is not involved in the research. “It’s not like you have this video camera of exactly what happened, exactly as it happened. You have to reconstruct, based on pieces of the experience, what you think happened.”

Cinematic Stimulation

Understanding what the brain is actually doing at any given moment as we go about our lives is exceptionally difficult. One of the best tools for studying the workings of the human brain is functional magnetic resonance imaging (fMRI), but the method requires you to lie still inside the bore of a large magnet while you respond to artificial stimuli presented to you in your locked-in position — a setup far from the reality of everyday life.

The brain-scanning method, which debuted in 1990, detects blood flow as a proxy for neural activity (if neurons are firing, they need oxygenated blood). In its early years, fMRI was imprecise: It described the average activity across entire brain regions. Then, in the early 2000s, advances in technology and analysis led to higher-resolution images of brain activity composed of 3-millimeter cubes of brain called voxels. The varying activity in these voxels creates patterns akin to fingerprints that denote different states of mind or types of mental activity.

With voxel analysis, researchers hoped to not only pinpoint where in the brain a specific memory, thought or perception was processed, but also determine its content. Could they tell, from brain activity alone, whether a subject was looking at a tennis racket, thinking about their best friend or remembering their old office? Each of those subjects should activate a unique set of neurons. “If we have a library of these fingerprints — one for each memory — then we could track when a specific memory is being activated by looking for its specific fingerprint,” Baldassano said.

Around the same time voxel analysis was being developed, other researchers wondered: What if we screened movies inside an MRI machine? Prior to that, researchers typically used flashcards or static images — a poor substitute for the vivid, multisensory, evolving experiences that create our actual memories. Movies, however, tell stories. “Narrative is fundamental to the way that people remember their day, their week and their life,” said Janice Chen, a neuroscientist at Johns Hopkins University who pioneered some of the memory studies at Princeton University, where she and Baldassano were postdocs together. However, like the scenery of life, the ever-changing three-dimensional imagery in movies is so complex that many researchers saw no way to make sense of the deluge of fMRI data.

“People didn’t run studies like that before because we didn’t know how we would analyze that data,” Chen said.

In 2004, the neuroscientist Uri Hasson and his colleagues at the Weizmann Institute of Science in Israel started carving a path through the thicket of voxels. In one of their studies, five people, while lying in a brain scanner, watched 30 minutes of The Good, the Bad and the Ugly (1966), a spaghetti western starring Clint Eastwood. Comparing the data from the five participants, the researchers noted when and where brain activity surged or waned in unison. Those points of convergence revealed regions that responded to specific parts of the film, such as meaningful shifts in plot. The experiment showed that it was possible to make sense of the brain’s real-time response to watching a movie.

In 2012, Chen joined Hasson’s lab, then at Princeton, and extended the approach to memory. She had people watch the first episode of the television show Sherlock (2010), featuring Benedict Cumberbatch as a modern take on the legendary detective. Then the study participants talked through their memory of it, while still lying in the scanner.

The experiment worked. Chen and her colleagues were able to match brain activity recorded during participants’ recollections to specific scenes around 60 seconds long — for example, when Sherlock meets Watson. Recalling a scene evoked similar brain activity to watching it, and those patterns were largely shared across subjects, suggesting that different people committed the same experiences to memory in the same way.

The results also meant that movies could be used to unlock a universal human code for recording experiences. But that code has many layers, and it would take a computer scientist with an interest in the brain to start peeling them back.

A Whole-Brain Affair

When Baldassano first learned about Chen’s Sherlock experiments, “I thought that was the coolest project I’d ever seen,” he said. He was at Princeton to interview for a postdoctoral position after nearly finishing his doctorate in computer science. As a graduate student at Stanford University, he had developed machine learning techniques to analyze multi-voxel fMRI patterns recorded from people looking at photographs.

Baldassano had not been interested in memory research; he imagined the field revolved around word lists, which seemed dull. But around that time, he had read the book Moonwalking With Einstein: The Art and Science of Remembering Everything, in which the journalist Joshua Foer recounts his efforts to train for a memory championship by using the method of loci. Baldassano had also observed his then-3-year-old son readily learn information conveyed in the form of a story. He saw then that human memory builds upon the narrative of life experiences.

He decided to join Hasson’s lab. “So much more of the brain cares about movies than the usual things we do to people [in fMRI studies] where we flash images,” he said. “It just seemed like there was a huge opportunity for new discoveries.”

He started by applying his computational expertise to Chen’s Sherlock data. In her study, she had defined the boundaries of the show’s scenes, and people’s recollections, in ways that made sense to her. Her efforts were based on event segmentation theory, the idea that people naturally carve continuous experience into chunks in their memory. But this approach was subjective and could introduce error. Baldassano went searching for the event segments in the brain data. What he found was a surprise.

He tweaked a machine learning technique called a hidden Markov model so it would work with fMRI data. The algorithm can be trained to recognize structure for events that stretch out over time, such as an episode of a TV show. The approach, borrowed from computer science, was new to neuroimaging. “That gave me a wedge,” Baldassano said. With this strategy he was able to sort through a massive data set: measures of neural activity from about 50,000 voxels across the brain, taken every 1.5 seconds during a 50-minute segment of Sherlock.

The model revealed a clear signal in the brain’s default mode network, which is thought to perform a range of cognitive functions involved in constructing internal narratives. A central hub in this network is the prefrontal cortex, which sits behind the forehead and attends to a person’s goals, plans and decisions. The network responds to meaningful shifts in a stimulus, such as a plot twist or a new topic of conversation. In the Sherlock data, dramatic fluctuations in activity occurred every minute or so, corresponding to what viewers perceived as scene changes.

“With Baldassano’s method, you could take that continuous brain data as people are watching a movie and look for where there are sudden changes in spatial activity patterns — and that matched what people would say were the boundaries in a movie,” Chen said. “It was a data-driven way of segmenting experience.”

The model revealed something else. The brain was not just segmenting at the boundaries people recognized as meaningful scene changes. Some parts of the brain subdivided the experience into shorter segments. In the visual system, shifts in activity occurred every second or so in response to changes in lighting and scenery. In mid-level visual regions that represent objects, brain activity shifted every 30 seconds or so to, say, track Cumberbatch as he moved across the screen.

The researchers proposed that the viewers combined these disparate narrative fragments into a coherent experience. Instead of memory being a province of the hippocampus along with a few other regions, as is commonly believed, Baldassano’s research suggested that memory formation involves collaboration among many brain regions.

“Memory is not one thing. It’s not a monolith,” Reagh said. “Your memory has a recipe for different pieces of the experience.”

However, Baldassano did not believe that these pieces are assembled anew with each life (or television) episode. “When an event starts, you already have some kind of scaffold,” he said. “It’s like a coloring book page that’s not colored in yet, but a lot of the lines are already there. When an event is happening, you are filling in the particular details.”

Psychologists had formally proposed that such scaffolds existed in the 1970s. But the line of research had fallen out of favor, largely because no one had good tools for studying them in the brain. Baldassano now had those tools. And he had a clever idea for how to use movies and television to study the scripts on which we build our memories.

Following a Script

Just like life, movies tend to repeat common themes. We have romantic comedies and disaster flicks, superheroes and alien encounters, and across all of these genres there are commonalities, such as a meal in a restaurant or a race through an airport. Baldassano set out to take advantage of this redundancy to locate some pages of the brain’s coloring book.

While each of 33 study participants lay in an fMRI machine, he screened clips from eight movies: four set in restaurants (Brazil [1985], Derek [2008], Mr. Bean [1997] and Pulp Fiction [1994]) and four set in airports (Due Date [2010], Good Luck Chuck [2007], Knight and Day [2010] and Non-Stop [2014]). Every restaurant clip contained more or less the same sequence of events: Characters entered a restaurant, were seated, ordered and ate. Airport clips all showed people arriving at an airport, going through security, walking to and waiting at the gate, and boarding a flight. But the movies differed in their details: genres, actors, plot points.

The study generated reams of data, and Baldassano took a first crack at it himself, checking manually for patterns in brain activity at certain times in the airport scenes. He didn’t expect to see much — fMRI data is blurry and noisy, and there is plenty of activity that the technique can’t access. But he thought he had spotted something interesting. When he shared a slide of his results at a group meeting in 2017, Chen exclaimed: “Oh my gosh, it worked!”

“Once Janice told me that it worked, I was convinced that I hadn’t messed something up,” Baldassano recalled.

Then he let the computers run their analysis. When Baldassano plugged the data into his fMRI-optimized hidden Markov model, it revealed a definitive sequence of brain activation patterns that was shared across people and across movies for a given type of event. Across all restaurant clips, one pattern showed up when actors entered, which shifted to another when they were seated, yet again when they ordered food, and once more when the food arrived. All restaurant stories shared these four event patterns on average, with some story-unique differences added on top. The airport movies were similarly represented in the brain, with each step of the sequence characterized by a predictable cross-brain fingerprint centralized in the prefrontal cortex.

The results revealed that the brain doesn’t simply record what it perceives. Instead, much if not most of the brain’s reaction to an event or story originates in memories of how that type of event usually plays out. In other words, we process the present through the past.

Baldassano’s work, published in The Journal of Neuroscience in 2018, showed that there are two critical steps to constructing memories. As we go about our day, we record the new experiences in pieces of varying size and complexity, from simple perceptions to stunning plot twists. Meanwhile, our brains access templates for these new events based on knowledge of similar ones, and place the pieces of the evolving memory in that context.

Memories, it turns out, are more like paint-by-number than rendered from scratch on a blank canvas.

Frame Shift

Expert chess players need only glance at the board to remember the positions of pieces, though for most people the arrangement would be vexing to reconstruct. Because they’ve played many games, chess masters recognize patterns where others are overwhelmed by detail. Similarly, when you meet friends at a restaurant, you know the server will come to take your food order; because you are not baffled by the basic sequence of events, you’re better able to remember flavors, details of your conversations, the person at the next table.

In other words, having scripts bolsters memory. “It’s so much easier to remember things if they slot into what you already know about the world,” said Chris Bird, who studies memory for everyday experiences at the University of Sussex. By extension, if you don’t have a working narrative template, your memory should, in theory, be worse.

Baldassano could test this phenomenon using his fMRI data. Mirroring Chen’s approach to her Sherlock studies, he had his study participants describe the airport and restaurant scenes to him while they were still in the scanner. Those who better booted up all the key parts of a script while they watched the movies, as indicated by their brain activation patterns, had a better memory for details than those who only weakly revived their scripts.

“If we saw them do a good job bringing this airport sequence of patterns online,” Baldassano said, “that predicted they were going to be able to remember more details of this particular airport story.”

Sometimes, more than one script runs at once in a story, or in life — and which script is activated can impact your memory of an event.

In work published in Current Biology in 2024, Baldassano’s team recorded short audio stories in which a social script (a breakup, proposal, business deal or meet-cute) played out over a location script (a restaurant, airport, grocery store or lecture hall). Then he gave his listeners roles to play as they listened. For a story about a marriage proposal in a restaurant, some listeners were asked to pretend they were restaurant critics, while others were cast as wedding planners.

Which frame a participant took — location or social — influenced the details that person remembered from the story. Echoes of both scripts were active in all listeners, but the dominant script better aligned with significant events in the brain data. For instance, from a wedding planner’s perspective, ordering food is not an interesting development in a proposal, so the activity pattern did not switch at that moment. It switched when one character asked for another’s hand in marriage.

The work underscores that the way we experience and remember events arises largely from our mental state, as opposed to properties of the events themselves. “You can change how the brain segments experience based on what you prime subjects with — what kind of information is relevant, whether it’s the social information or the location information,” Kuhl said. “That’s a novel contribution.”

And it suggests that people have some power over what they remember. “It’s not purely the stimulus activating these things,” Baldassano said. “You have some volitional control over how you choose to categorize the information that’s coming in.” That is, your preconceptions and goals shape your experiences and what you remember about them. It may be possible to adopt a mindset that predisposes you to remember particular details of an event.

Understanding these scripts and their nuances could shed light on memory disorders, Bird said. Scripts degrading or becoming inaccessible could account for some of the disorientation caused by memory disorders such as Alzheimer’s. “[The notion of scripts] gives us a window into how we normally make sense of the world, and why the world can be a very confusing place if you have a breakdown in these processes,” Bird said.

On the other hand, scripts can be leveraged to improve memory, as Mullen, the memory champion, does. In an ongoing study, Baldassano’s team taught 25 people to use the method of loci to memorize a list of 40 words. For example, they might use a restaurant script: To remember “waterfall,” they might imagine a waterfall behind a restaurant; for “microscope,” they might envision exceptionally tiny food.

At the start of the study, nobody could remember more than nine words. By the end, everyone recalled at least 35. Some earned a perfect score, though no one matched Mullen’s speed. “We were really successful in getting regular people to do this,” Baldassano said.

This script-writing process takes place throughout our lives: Experiences, routines and adventures comprise most of the raw material, but media such as books and movies also set up scaffolds for memories.

Baldassano has been fascinated to watch this play out with his sons. As toddlers, his children particularly liked books about Maisy Mouse. In the series, Maisy goes to the library, visits the dentist, learns to swim — and, yes, goes to the airport. There is little in the way of plot or incident: The mouse checks her suitcase, goes through security and boards the plane. Baldassano found the books “brutally boring” — but his children were riveted.

“It’s 100% just schema instruction,” he said. The world can be bewildering to children because they lack the mental scaffolds to make sense of what is happening. Books and other media help write the scripts, at the neurobiological level, that will help his kids navigate the rest of their lives. Baldassano knows better than most: “It’s a really important kind of knowledge to build.”