Bluefin Labs Reveals How It Is Tying Social Media To TV

On the Web, we have links, which makes all media trackable. But on TV there are no links. So how do you track the audience response to a TV show or an ad? It’s all guesswork, panels, and surveys pretty much. But Deb Roy thinks he has a better answer: treat social media as a realtime “focus group in the wild” and tie that commentary back to TV. He wants to infer links from what people are talking about. In the video above, he explains his approach with Bluefin Labs.

“Think about a switchboard that links realtime TV with social media,” he says. Roy is the founder and CEO of Bluefin Labs, a video and language analytics startup in Cambridge, Massachusetts. Bluefin is creating a console for advertisers and TV programmers to measure the social resonance of their content. Using sophisticated semantic analysis, Bluefin can determine what peopel are saying about a particular TV show or commercial across various social media, including Twitter, Facebook, and blogs.

The console (see screenshot below) still spits out fairly raw data right now and is in the process of getting a cleaner UI, but essentially it shows what looks like a digital program guide with shows and ads being tracked on different channels. For each show or ad, the grey bars represent how much commentary was sparked across various social media, with an actual sampling of Tweets and Facebook comments, along with a tag cloud summarizing what people are saying about that show or ad.

A brand introducing a new product could see how often the name of the product is mentioned by people talking about it versus the overall umbrella brand. Advertisers interested in actually measuring engagement could use this data to see how much buzz is created given the reach of a particular show. They could look at the response rate per airing and then rank order each TV network to see where their ad dollars are best spent.

“For every mass media action there is some sort of audience response,” says Roy. “This has always been the case. Because of the low barrier to entry to social media there are feedback loops. Those roll up to a significant new force which shifts how audiences view the mass media.”

Roy is a researcher at the MIT Media Lab and he founded the company in 2008 with one of his Ph.D students, Michael Fleischmann. ver the past 15 years, Roy’s research explored the nexus between video and language. He taught a robot named Toko the names of objects using video and language as complimentary feedback loops, and put his own family under video surveillance to capture how his son learned language over a period of 36 months.

Now with Bluefin, he is taking that deep machine learning and semantic analysis and applying it to TV. Last year, Bluefin raised a $6 million series A financing led by Redpoint Ventures. Other investors in that and a previous seed round include Lerer Ventures, Acadia Woods Ventures, Brian Bedol and Jonathan Kraft. The company has raised a total of $8.35 million, including a $1.15 million grant from the National Science Foundation. The company is piloting its console with nine large advertisers, brands, and media agencies including Pepsi.

But getting back to how social media can create effective links to TV. What is a link? It is a reference to something else on the Web. When you talk about a TV show or ad that is also a reference, but you usually don’t use links. Your comments, however, are increasingly being captured by social media. “You don’t need to interact with a piece of online media to create a response,” says Roy, “all you need is somebody to talk to.”

Bluefin is “treating social media like clickstreams: It determines when somebody is talking about a particular show or ad on TV using its language learning and semantic analysis engine, and then creates an implicit link to that content in its database. It is taking social media and mass media and mixing them together.