Your Face Is Big Data: Egor Tsvetkov's FindFace Experiment

First published April 26, 2017.

In early 2016, Russian photographer Egor Tsvetkov took pictures of strangers sitting opposite him on the St Petersburg metro, then used a face-recognition app called FindFace to look for them on VKontakte, Russia's largest social network. He reported finding the profiles of roughly 60 to 70 percent of the younger people he photographed. He called the project "Your Face Is Big Data".

The work was a piece of art, not a scientific study, but it made a point that has only grown more relevant: a single candid photo of your face can be enough for someone to connect you with your name, friends and life online. Here is what Tsvetkov did, how the technology behind it worked, what happened to FindFace afterwards, and what the experiment does and does not show.

What Egor Tsvetkov did

Tsvetkov, then a photography and art student, spent about six weeks on the project. According to Russia Beyond, which interviewed him in April 2016, he photographed passengers on the St Petersburg metro without their knowledge; PCWorld put the total at about 100 people. Most of them were looking down at their phones and did not notice the camera.

He then fed each picture into FindFace. The app compared the faces with profile photos on VKontakte (VK) and suggested likely matches. Tsvetkov told PCWorld that finding people aged roughly 18 to 35 was easy, with a success rate of 60 to 70 percent in that group, while older passengers, who tend to post fewer photos, were harder to find. Explaining his aim, he said my point in this art project is to show how technology breaks down the possibility of private life.

He said the idea came the moment he first heard about FindFace. He did not build the app himself, as some reports later claimed; it was made by a Moscow start-up.

Two faces of the same people

Tsvetkov presented his results as pairs: on the left, his own candid photo from the metro carriage; on the right, a picture the same person had chosen to share on their social profile. The contrast is the heart of the project.

In the metro shots, people are wrapped in winter coats and scarves, tired, absorbed in their screens. In the profile pictures they are smiling at birthday cakes, posing with sparklers, taking mirror selfies or, in one case, wearing a paratrooper's uniform and blue beret. The same woman in a fur coat appears twice in this series, once at the top of the page and again further down, with a glittering party photo beside her commute portrait.

That gap is exactly what makes the result striking. People curate their online image carefully, and the software still matched a tired face on a train to a polished party photo.

How FindFace worked, and what happened to it

FindFace was created by NtechLab, a Russian company founded in 2015 by Artem Kukharenko and Alexander Kabakov. The public web service launched in February 2016 and let anyone upload a photo to search for matching faces among VK profiles. It used neural-network algorithms trained to compare facial features rather than whole images, which is why it could cope with different lighting, angles and expressions. It passed one million visitors by May 2016.

The same feature that made Tsvetkov's project possible was soon misused. In 2016, users of the Russian imageboard Dvach used FindFace to identify women who had appeared in adult films and to harass them and their families, which drew widespread criticism.

FindFace's later history shows where this technology went next:

• In 2017, NtechLab's algorithms were built into Moscow's city video-surveillance network, matching camera footage against a police database.
• In September 2018, the public people-search service was shut down, and FindFace became a product for businesses and government agencies.
• In 2022, NtechLab was accused of helping Russian and Belarusian authorities track political activists.

So the experiment captured a brief moment when a powerful face-search tool was open to anyone with a phone, before it moved into the hands of companies and the state.

What the experiment shows, and what it does not

Tsvetkov's work is often shared under headlines claiming that privacy is over. That goes further than the evidence. A few points are worth keeping in mind:

• A match is a suggestion, not proof. Face-search tools return likely candidates. Some will be wrong, and checking them takes human judgement.
• The sample was small and informal. About 100 photos taken by one person is a striking demonstration, not a measure of accuracy. Reported accuracy figures for FindFace at the time varied widely, from around 70 percent in a 2016 Washington Post report to much higher claims from the company.
• It depended on public photos. The app could only find people whose profile pictures were visible on VK. Those with private or photo-free accounts were much harder to identify, which is one reason the success rate dropped for older people.

The real lesson is about combinations. One photo, a first name, a list of friends and a check-in at a café may each seem harmless. Put together and searched by a machine, they can reveal far more than any one of them alone.

Face recognition since 2016

Face search has become more capable and more common in the years since. Police forces in many countries use it, some private services have scraped billions of public images to build search databases, and regulators in Europe and elsewhere have responded with fines and new rules on biometric data. At the same time, smartphones use similar technology every day to unlock screens and sort photo albums.

That mix of convenience and risk is why the debate continues. Supporters point to finding missing people and suspects; critics point to mistaken identifications, surveillance of protesters and the end of being anonymous in a crowd. Tsvetkov's metro portraits remain a clear, human illustration of what is at stake.

Practical steps to protect your face online

You cannot stop someone from photographing you in public, but you can reduce how easily a photo leads back to you:

• Review your profile pictures. Profile and cover photos are often visible to everyone, even when the rest of an account is private.
• Limit who can see older photos and tags. Most social networks let you restrict past posts and approve tags before they appear.
• Separate accounts. Using a different photo, or no face at all, on accounts tied to your full name makes linking harder.
• Check app permissions. Apps with access to your photo library or camera may collect more than you expect.
• Search for yourself. Looking at what a stranger can find is the quickest way to see where to tighten settings.

None of this makes anyone invisible. It does make it harder for a single snapshot on a train to become a full profile of who you are.

Image credit: Egor Tsvetkov, "Your Face Is Big Data".

Sources: Russia Beyond, 12 April 2016: Russian photographer matches random people with social network photos; PCWorld, 13 April 2016: Your Face Is Big Data; Wikipedia, FindFace; original project via Bored Panda.