Can Brain Scans Read Your Mind? The 2016 Face Reconstruction Study

Originally published 2016-07-26.

Can a machine read your mind? Not in the science-fiction sense, but in 2016 two neuroscientists came closer than most people expected. Hongmi Lee and Brice Kuhl, then at the University of Oregon, used brain scans and a machine-learning model to rebuild rough images of faces that volunteers were looking at. They then tried something harder: reconstructing faces that people were only holding in memory. The results were blurry and often strange, but recognisable enough that strangers could guess basic features of the original faces better than chance.

Kuhl described the ambition vividly to Vox: "We can take someone's memory, which is typically something internal and private, and we can pull it out from their brains." Here is how the experiment worked, what it really showed, how far the field has moved since, and what it means for privacy.

How the face-reconstruction experiment worked

The study, published in The Journal of Neuroscience in June 2016, involved 23 volunteers and a library of about 1,000 colour photographs of different people's faces. Each volunteer lay in an fMRI scanner while the faces appeared on a screen.

To make faces understandable to a computer, the team used a technique called principal component analysis. It breaks a large set of face images into "eigenfaces," basic patterns of light and shade that can be mixed in different amounts to approximate any face. Each photo could then be described by a list of around 300 numbers, one for each component.

In the training phase, the model learned how patterns of brain activity related to those numbers. In the test phase, the volunteers saw new faces the model had never encountered. The model looked only at their brain activity, predicted the 300 numbers, and turned those back into an image.

What fMRI can and cannot see

Functional magnetic resonance imaging does not read neurons directly. It tracks changes in blood oxygen: active brain areas draw more oxygen-rich blood, which slightly changes the magnetic signal the scanner detects. This blood-oxygen-level-dependent (BOLD) signal was discovered by Seiji Ogawa and colleagues in 1990, and human studies followed in 1992.

The method has real limits. The blood response lags behind brain activity by a couple of seconds and peaks after four to six, so fMRI is slow. Each measurement point, or voxel, typically covers a few millimetres of tissue containing vast numbers of cells. The signal is also noisy, affected by breathing, heartbeat and head movement, so researchers need many repetitions and careful statistics. Any image built from fMRI is therefore a statistical estimate, not a photograph of a thought.

The two brain regions behind the images

Lee and Kuhl reconstructed faces from two areas. The occipitotemporal cortex (OTC), towards the back and underside of the brain, handles visual processing and includes regions specialised for recognising faces. The angular gyrus (ANG), in the lateral parietal lobe, is linked to language, number processing, attention and the vivid recollection of past experiences.

Finding that the angular gyrus carried enough information to rebuild faces was the study's most interesting result. It suggested this region is not just a hub that signals "I remember something" but actively represents the content of what is being perceived and recalled.

Below are examples of original faces alongside reconstructions from brain activity while volunteers were looking at them:

How well did it work?

The reconstructions are clearly not portraits. They look like smudged, ghostly averages. To test whether they captured anything real, the researchers showed them to a separate group of online participants and asked simple questions: is this person male or female, happy or sad, light- or dark-skinned? Answers matched the original faces more often than chance would predict. According to the published paper, emotional expression and skin colour were reconstructed especially well from the angular gyrus.

So a computer, trained on one person's brain data, produced images that let other people infer some features of what that person had seen. That is a genuine finding, even if it falls far short of reading detailed thoughts.

Rebuilding faces from memory

The harder task was reconstruction from memory. Volunteers viewed a face, then held it in mind while the scanner recorded their activity. The model, trained on activity from perception, was then asked to reconstruct the remembered face. The paper reports that a model trained on angular gyrus activity could reconstruct an independent set of faces held in memory, though the images were even rougher than before:

This matters scientifically because it shows that remembering a face reuses some of the same brain patterns as seeing it. It also shows how much information is lost between the original image and what a scanner can recover from memory.

Myths about "mind-reading machines"

• Myth: scanners can extract any private thought. The model only works for the kind of content it was trained on, here faces, and under controlled conditions.
• Myth: it works on anyone. Decoders are typically trained on each person's own brain data over hours of scanning. They do not transfer easily between people.
• Myth: it could be done secretly. fMRI requires lying still in a large, loud machine. Moving your head or thinking about something else ruins the signal.
• Myth: it could replace eyewitnesses or police sketches. Ideas like rebuilding a suspect's face from a victim's memory remain far off, and courts have not accepted fMRI-based lie detection as reliable.

How brain decoding has advanced since 2016

Image quality has improved dramatically. In a paper released in December 2022, Yu Takagi and Shinji Nishimoto of Osaka University combined fMRI recordings of people viewing photographs with the image-generating AI Stable Diffusion. Their method predicted a compressed representation of each image, plus text-like descriptions decoded from higher visual areas, and let the AI fill in detail. The results looked far closer to the originals than the blurry faces of 2016.

The basic limits remain the same, though. These systems still rely on volunteers viewing pictures inside a scanner, on large amounts of personal training data, and on powerful AI models that can add plausible detail which was never in the brain signal. Researchers and ethicists have begun discussing "neural privacy" so that rules are in place before the technology becomes portable. Earlier experiments that linked two people's brains over the internet to send simple signals, such as direct brain-to-brain connection studies, are another part of the same story: impressive in the lab, still far from everyday telepathy.

For now, your thoughts remain your own. What these studies offer is a new window into how the brain stores what we see and remember.

Sources: Lee & Kuhl, Reconstructing Perceived and Retrieved Faces from Activity Patterns in Lateral Parietal Cortex, The Journal of Neuroscience 36(22), 2016; ScienceAlert, Bec Crew (23 June 2016); Wikipedia: Functional magnetic resonance imaging; PetaPixel: Researchers use people's brain activity to generate AI images (March 2023). Kuhl quote as reported by Brian Resnick for Vox; reconstruction images from Lee & Kuhl (2016).