Sharon Machlis

Why I'm skeptical about the Twitter Political Index

August 03, 2012 1:21 PM EDT

Twitter is trying an interesting experiment with its Political Index, aiming to gauge "users' feelings towards the candidates as expressed in nearly two million Tweets each week," according to a company blog post. But "interesting" is not the same thing as "predictive;" and unfortunately this data stands a rather large chance of being misused. " 'Twitter Will Gauge Voter Sentiment in New Venture' was the headline at National Journal -- never mind the fact that this is neither a measure of voters or of sentiment," Micah Sifry at TechPresident.com observed.

Here are 3 reasons I'm skeptical about the Twitter political index as a tool for predicting the election:

* Potential for a wildly skewed sample set. I need some serious convincing that the universe of people tweeting about the U.S. presidential election accurately reflects the demographics and opinions of U.S. voters in 2012. Reputable political polls take great care to weight for general characteristics like age and gender, as well as specifics such as proportion of registered Democrats, Republicans and independents are likely to vote. On Twitter, we don't even know if tweets are coming from registered voters -- they could be politically involved students too young to vote or people overseas; we certainly don't know whether the pool of tweets accurately reflects likely voters.

This is especially dicey for the American Presidential election, where the voters who matter the most -- swing voters in swing states -- may be among those least likely to be heavily tweeting about the election. Given the vagaries of the Electoral College, it matters a lot less what people in New York or Utah are tweeting than those in Ohio and Pennsylvania. Even if the Twitter political index is accurately measuring overall public sentiment, it wouldn't be predictive: Just ask Al Gore, who won the popular vote by more than half a million in the 2000 election.

A fellow skeptic pointed me to a paper co-authored by two members of the Wellesley College Computer Science department with a colleague at Universidad de Oviedo in Spain, titled How (Not) To Predict Elections that concluded:

"[D]ata from social media did only slightly better than chance in predicting election results in the last US Congressional elections. We argue that this makes sense: So far, only a very rough estimation on the exact demographics of the people discussing elections in social media is known, while according to the state-of-the-art polling techniques, correct predictions requires the ability of sampling likely voters randomly and without bias. . . .

"Our results do not argue against having a strategy for involving social media in a candidate's election campaign. Instead, it argues that, just because a candidate is scoring high in some social media metrics (e.g., number of Facebook friends or Twitter followers), this performance does not guarantees electoral success."

* Pitfalls of measuring sentiment. It's simply not clear that algorithms can accurately measure sentiment, especially in 140-character text bursts that may contain irony, humor and satire. Does the index understand that "I can't wait to see [candidate] in the White House in 2013" with a link to a negative article, is actually a negative sentiment and not a positive one?

Topsy, which is partnering with Twitter to parse tweet sentiment, says its algorithms "agree with a randomly selected human 90% of the time on what a tweet means," Buzzfeed reported. That's something to keep in mind as you try to work out what the margin of error might be for the day's stated Twitter Political Index numbers.

* Incentive to game the data. There's not much you as an individual can do to try to tip results of a public opinion poll except hope to be called, other than perhaps have a lot of phone numbers for polls that random dial. But there's quite a lot that partisans can do to try to influence the Twitter index: Tweet more. One would have to assume that partisans of each candidate will do this in exactly the same amounts to conclude that it won't at all affect the data.

Granted, there's a large enough pool of election-related tweets -- 280,000 or so per day -- that someone adding 20 per day is unlikely to affect the index. But what about a group who each decides to each add 50 a day? Or more, via automated tools?

Conclusion? The Twitter Political Index is a fun and intriguing tool for data mining political tweets, but be wary about reports that try to make any connections between that data and likely election results.

See more from the Data Avenger blog series.