When a Super Bowl Ad Comes On, Advertisers May Know if You Watch or Click Away

Consumer Reports has no financial relationship with advertisers on this site.

Consumer Reports has no financial relationship with advertisers on this site.

If you watch Sunday’s Super Bowl on a Hisense television, a company called Alphonso may be watching alongside you. The company will use “automatic content recognition,” or ACR, technology embedded in the set to record whether you change the channel during ads.

Depending on everything from your age to your favorite soft drink brand, company executives explain, Alphonso might also help send specific ads to your smartphone during the game.

And if you take one of those online polls asking you to rate an ad like the Pepsi spot with Cardi B and Steve Carell, a brand research company, Survata, may use Alphonso data to confirm that you really did see that particular spot on TV. According to its representatives, Survata will be helping companies get feedback on a number of commercials.

Alphonso and Survata are just two companies in an industry that’s grown up to combine real-time TV-viewing data with information on consumer smartphone use, location-tracking technology, and advertising. (To learn how Consumer Reports collects and uses consumer data, please see our privacy policy.)

Consumer Reports has been reporting on ACR since 2015, and in 2018 we evaluated smart TVs to see how much data they were sharing with outside companies, and how easy it was for consumers to learn about and control the technology. (Here's how you can turn off this feature.)

“Virtually every smart TV you can buy has ACR, and it tries to grab as much information as possible about you and your viewing habits,” says Bobby Richter, who leads CR’s privacy and security testing program.

An ACR system takes milliseconds-long digital video samples from your TV screen and transmits them to company servers, where they are matched against a library of clips from movies, TV series, and commercials to determine what you’re watching. ACR works almost continuously, says Alphonso founder and CEO Ashish Chordia, sending out brief video samples one after the other.

Alphonso also has "audio-fingerprinting" technology built into many mobile games and other apps. As long as the user has granted permission, the apps can use a smartphone’s mic to capture tiny audio fragments, also measured in milliseconds, to identify what commercials or other content is playing on a user’s television. The company says the apps do not record or transmit speech.

“We can tell if you’re changing channels or fast forwarding through content,” says Chordia. “We see if you’re watching 10 seconds or 30 seconds of the ad.”

How the Data Is Used

Alphonso says it collects data from about 15 million sets in the U.S. Its technology also appears in other devices, including many set-top boxes. The company maintains a database of the results, so it knows, for instance, whether a certain household has played every episode of HBO’s “True Detective” or maintains a Thursday evening “The Good Place” habit.

The data isn’t linked to your name, the company explains, but it is associated with the IP address for your home WiFi network. Your IP address can also be associated with other devices that regularly connect to your home network, such as your family’s smartphones, laptops, and tablets.

However, it can be tricky for consumers to learn details such as who gets to see their TV-viewing data and exactly how it gets used.

Vizio in 2017 settled charges for collecting ACR data without users’ knowledge or consent, paying $1.5 million to the federal government and $700,000 to the state of New Jersey. The Federal Trade Commission has since made it clear that companies need your permission before collecting viewing data—but CR’s 2018 evaluation suggested the need for more transparency and consumer choice in smart TVs.

“While ACR is an opt-in technology—you sign up when you’re setting up the smart features on your TV—the privacy policies for the various platforms we evaluated were often hard to read,” says Richter. “That makes it hard to know exactly what you’re agreeing to.”

Alphonso is just one supplier of ACR technology. A number of Sony TVs incorporate ACR technology from a company called Samba TV. Nielsen, the TV ratings company, has an ACR subsidiary called Gracenote, and Vizio owns its own ACR company, Inscape. Samsung uses its own, in-house technology.

Wherever it comes from, TV-viewing data can let advertisers learn where their competitors are running ads, while helping them hone their own marketing campaigns.

A company called iSpot, which measures television ads using ACR data from other companies, can help analyze the information. The company recently studied a trailer for the 2019 movie “What Men Want,” a fantasy comedy about a woman who can hear men’s thoughts. (It’s a loose reimagining of the 2000 movie “What Women Want.”) According to Jason Damata, an industry analyst who works with iSpot, a sample of around 4 million households revealed that people who saw a version of the movie ad on the BET network typically watched it all the way through, while people who viewed the ads on other networks frequently switched away.

“With this kind of information, companies can make decisions in real time based on larger data samples rather than going by the reactions of a small group people in a focus group at a mall,” Damata says. (The movie studio did not hire iSpot to gather this information; like other companies interviewed for this story, iSpot declined to discuss work contracted for specific clients.)

What You Watch, Where You Shop

ACR has one primary benefit for consumers: It can enable your smart TV to make content suggestions based on your previous viewing patterns, a feature that may be familiar from Netflix. (“Like Jason Statham in ‘The Bank Job’? You might enjoy ‘The Mechanic.’”) The better the system gets to know you, the better the content suggestions get.

In exchange, Alphonso's Chordia says, “We, of course, make money with advertising.”

Heading into the Super Bowl, he explains that the company knows information such as whether your household has watched every Patriots game on TV or is a once-a-month NFL dabbler. And in addition to understanding what you watch on TV, companies like Alphonso know a lot about where you shop and what you buy. 

This information can come from a number of sources. One of Alphonso’s clients is the fantasy sports site FanDuel. Alphonso says it supplies tracking technology to FanDuel that reveals when a consumer responds to a FanDuel television ad with a visit to the site—and whether the consumer spends money there.

ACR companies can further supplement information about your TV viewing and your digital shopping with location data.

“Location [data] providers such as Placed and Foursquare are marrying location technologies to TV device data to help brands and agencies identify foot traffic against advertising impressions,” reads marketing material on Inscape’s website. Alphonso’s sources of location information have included the data brokers SafeGraph and NinthDecimal. Most of the information ultimately derives from consumers' smartphones—when you give a mobile app permission to access your phone's location, you may be supplying data that ends up with one of these vendors. 

Taco Bell is an Alphonso client. The data company might be able learn that you made a trip to a Taco Bell location after seeing that company’s commercials. It might also be able to find out whether you bought anything.

Depending on the project and client, Alphonso can learn about purchases from brokers who collect transaction data, such as IRI, Nielsen Catalina, Research Now, and Comscore. As Chordia explains, advertisers can then learn which ads got attention from viewers, which resulted in visits to websites or retail locations, and ultimately how it all translated into sales. This allows advertisers to decide where and when to show their ad, and how often—purchases can decline if a commercial is shown too often, Chordia says.

Alphonso TV wouldn’t discuss specific advertising plans for the Super Bowl, citing nondisclosure agreements. However, as an example, the company explained that a fast-food client would be able to target a customized set of consumers who meet very specific criteria, such as men ages 18 to 34, who are both serious football viewers and HBO subscribers, who watched a spot for the restaurant within the last month, and who then bought a meal there within the next 24 hours.

If all those characteristics describe you and you watch the big game on Sunday, you might then see a follow-up ad from the fast-food chain on your phone.

Maybe even around lunch hour on the Monday after the game.

More from Consumer Reports:
Top pick tires for 2016
Best used cars for $25,000 and less
7 best mattresses for couples

Consumer Reports is an independent, nonprofit organization that works side by side with consumers to create a fairer, safer, and healthier world. CR does not endorse products or services, and does not accept advertising. Copyright © 2019, Consumer Reports, Inc.