Sunday, November 11, 2012

A Last Look at Those Polls for Next Time

I was surprised by how many people seemed stunned by last Tuesday’s presidential election outcome. People like Beth Cox, for example, described on the front page of today's Washington Post. 

The thousands of polls taken over the past several months, or more likely the spin of those polls by pundits, apparently did an outstanding job of convincing everyone of exactly whatever it was they wanted to hear.

Newt Gingrich predicted a landslide win for Romney. Republican operative Dick Morris explained on Fox News that the polls showing Obama ahead erred because they seriously oversampled Democrats (wrong) and Obama’s core constituency wasn’t enthusiastic (wrong again). So rather than being behind by a few points, Morris thought Romney was actually ahead by 5 or 6 points.

Rush Limbaugh predicted a Romney win and threatened to move to Canada if he lost. He had also threatened to move to Costa Rica if the Affordable Care Act passed into law. Of course, he didn’t. 

(Why does Rush always threaten to emigrate to countries with universal health care? But, I digress . . .)

Meanwhile, over on CNN, progressive show host Chris Matthews said every night, “It’s razor thin out there!” 

It wasn’t.

And a pair of professors from the University of Colorado predicted a huge Romney win based on economic analyses, only to have to later apologize for their dismal performance.

Along the way, Nate Silver was widely abused by conservatives because his FiveThirtyEight blog’s model predicted a big Obama win. Silver correctly called every state in the 2008 presidential election. After Politico’s Dylan Byers called Silver a “One-Term Celebrity” and MSNBC’s “Morning Joe Scarborough” told Nate that he was looking at the election all wrong, Silver went on to correctly call every state in the 2012 election, as well, and had the closest prediction of all.

Sam Wang uses similar statistical techniques at Princeton Election Consortium and his predicted probabilities for an Obama win were much higher than Nate Silver’s, but Wang’s lower profile didn’t draw as much conservative ire. Wang had Obama with a 99% probability of winning weeks before the election (higher than Silver). His prediction of electoral votes was actually closer than Silver’s.

Prediction markets predicted Obama, too. Online services like Intrade have a pretty good prediction record. They allow people to bet actual money on the outcome and had Obama holding a 70% chance of winning for several weeks before the election. In the final few days, Intrade probabilities for an Obama win went up fairly dramatically to match those of Silver and Wang.

Slate has a nifty graphic showing prediction accuracy for the 2012 elections. Hover over the darts for the data.

So, what should you consider four years from now to understand how the election is actually going?

First, realize that there are several groups of players in this game. There are pollsters, like Gallup and Quinnipiac, who collect data from relatively small groups of potential voters and they sell this data. But individual polls are like pixels in a broader picture.

There are pundits, like Chris Matthews and Bill O'Reilly, who interpret poll results for their media audiences. Their primary objective is to influence voters, so their interpretations of the poll results can be quite biased.

Then there are statisticians like Nate Silver and Sam Wang. They don't poll potential voters themselves, they use statistics and probability to put together the larger picture I mentioned. Their goals, apparently, are simply to prove that they're smarter than everyone else and make money doing that. They have more incentive to be correct than to favor a party.

And, there are prediction markets. They take bets on who will win and their goal is to make money managing the market. They win no matter which party gets elected, so they have less reason to be biased, too.

Ignore Pundits

The most important thing you can do is to ignore political pundits on all sides. Whether they are conservative, progressive, or otherwise, they have terrible prediction skills. To paraphrase Warren Buffet, political pundits were created to make fortunetellers look good. (Economists are even worse forecasters.)

Even if they have some skill at prediction, they can’t tell you what they actually believe. They are paid to convince their followers that their candidate is winning but still needs your vote.

After the election, Chris Matthews apologized to his audience and said he and the other pundits were wrong and Nate Silver had had it right all along. A lot of people should apologize to Nate.

Look at the charts below from FiveThirtyEight and compare them to the story pundits were spinning on both sides. Romney momentum? It was gone before the second debate started. Christie and Sandy? Hardly caused a blip.

The pundits had it all wrong. They usually do.

Ignore Individual Polls

Hardly a day passed in the past few months when I didn’t hear Chris Matthews say something like, “A new poll out of Quinnipiac shows that President Obama is losing support among left-handed upland bird hunters.”

This is pure noise. It’s like watching a ticker tape of stock prices fly by and trying to figure out if the market is moving up or down as a whole. Ignore it.

Silver and Wang combine all the polls in a way that improves their predictive power. Their predictions will include the intentions of both left- and right-handed upland bird hunters, and most everyone else.

Perhaps the best-known poll, the Gallup Poll, performed exceptionally poorly in 2012 elections.

Forgive Silver and Wang for the Dems Having Won

If you read Nate Silver’s writing, you will conclude that he personally leans slightly left, but that he isn’t really all that political.  Still, many of my progressive friends love Silver because he consistently picked Democrats to win the past two elections.

But he did that because he was right, not because he is “left”.

The problem for my friends is that he will also pick the next Republican winner correctly, after which Democrats will love him less but he will still be just as accurate.

In his recent book, The Signal and the Noise, Silver suggests that he will probably sell his model to someone after the 2012 election and move on, having now conquered baseball and politics. The FiveThirtyEight model should still work with Silver gone, and presumably Sam Wang will still be around and using a similar process.

Ignore Strange Models

Kenneth Bickers and Michael Berry, professors at the University of Colorado, used an economics model to predict the 2012 presidential election and missed by 124 electoral votes. They predicted a huge Romney victory right to the bitter end. Only Karl Rove hung on longer.

They claimed that their model “would have predicted” the last eight elections. Good models would have predicted the present from the past, but they also need to predict the future.

You can find millions of patterns that “would have predicted” the past  —  like Redskin victories, hemlines, Super Bowl victories and the University of Colorado economics model  —  that don’t predict the future. 

Turns out the future is harder to predict than the past.

It’s hard enough to predict an election outcome by asking people how they will vote, as polls do. Assuming how they will vote based on how they might react to local economic conditions can only be much more difficult and unreliable.

Putting Their Money Where There Mouth Is

Then there are the prediction markets. At places like Intrade, people vote with their wallets instead of their hearts. Studies have shown that the prediction markets are pretty good. Some experts believe that Intrade actually outperformed Silver’s FiveThirtyEight blog in 2008. Intrade showed Obama with a 70% probability of winning in 2012 for several weeks before the election.

But, 70% isn’t certainty. It often rains when there is only a 30% chance of precipitation and this probability means that if the election were held 10 times, Romney would have won three of them.

Prediction markets don’t have the political pressures that pundits do to spin their findings and they have a fairly large sample size. When people “vote” on Intrade, they lose money if they’re wrong, so you tend to get honest opinions.

The Popular Vote Isn’t the Scoreboard

You could track both teams’ field goal percentages during a basketball game and often have a pretty good idea of who’s winning, especially if there is an unusually large disparity, but that isn’t the scoreboard, and the scoreboard decides who wins.

In presidential elections, the popular vote doesn’t determine the outcome; the Electoral College does. Al Gore won the popular vote and lost the presidency in the Electoral College. Nate Silver determined after the election that, given initial conditions in the states for this election, Romney could have won the popular vote by 2% and still lost the Electoral College.

So, when Chris Matthews was shouting, “It’s razor thin! It’s razor thin!” because the popular vote appeared close, he was unknowingly saying that Obama was ahead, because razor thin in this particular election would have been all Obama needed.

Though it appeared thin after considering the margin of error for the polls, the popular vote wasn’t actually terribly close. Obama won it by three million votes after all the results were in. “W” beat John Kerry by the same margin in 2004. Popular votes can be within the margin of error and still have large victories for one side.

What to Do Next Time

My suggestion would be to ignore the pundits next time and understand they are acting out of self-interest (keeping their jobs). I would bet on Silver and Wang’s interpretations, but if you have convinced yourself that they are left-leaning because they correctly called two presidential elections for Democrats, then go with the prediction markets.

Go with state polls and the Electoral College, which have proven to have more predictive power than national popular vote polls. The popular vote, if it is reasonably close, doesn’t tell you who will win in every election.

Look at a few prediction markets, including Intrade, and remember that a 70% chance of winning isn’t a certain chance of winning. On the other hand, if your favorite news network is predicting a landslide for your candidate and Intrade gives him a 20% chance, you might want to factor that into your thinking.

If the poll aggregators like Wang and Silver combined with the prediction markets like Intrade show some sort of consensus, they're probably onto something.

With Wang predicting a 100% chance for an Obama win, Silver predicting over 90% and Intrade predicting over 90% days before the election, I wasn’t surprised by the outcome, regardless of what Newt said he thought. I thought a Romney win was possible, but I would have been surprised.

The information you need is out there in black and white, but you have to filter out the red and blue.

Saturday, November 3, 2012

Clear Liquids

I spent last week at the National Institute of Health participating in a study.

I have an MRI every year or two so my doctor can tell me that nothing much has changed and that I should come back in a year or two for another MRI.

I have a crappy private insurance policy that costs about a million dollars a month with a five million dollar annual deductible (but limited to just ten million per family), so I pay the $3,500 for the MRI and the brief doctor’s visit out of pocket.

I have a working agreement with my insurer. I pay every premium on time or they will terminate my policy. In exchange for this premium, the insurer agrees to never, ever pay a claim.

So far, they’re holding up their end of the deal.

This year, needing both a colonoscopy and an MRI, I participated in this study and got those two tests for free — my favorite price for expensive services — and NIH threw in a P.E.T, a C.T., and an EKG.


They also ran several blood tests and a capsule endoscopy, but I’ll get back to that.

NIH told me I could bring my wife along for the week and that it would be fine to leave the clinic evenings to have dinner with her and maybe check out a movie. What they didn’t tell me is that I couldn’t eat anything at dinner unless clear broth or apple juice was on the menu.

Doesn’t seem to be all the rage in Washington restaurants at the moment.

Nearly all of the tests required a clear liquid diet the day before, and I had a test every day, so I got two actual meals all week. I could eat anything I wanted as long as it was clear broth, apple juice or a popsickle. 

But, I needed to drop a few pounds and I’m now down to 137 soaking wet and I can wear those skinny jeans, so there’s that.

Back to the capsule endoscopy. That’s the test where you swallow a small camera that takes pictures of your innards and transmits them to a small computer that you wear. I had one back in 2001, too.

At first, the doctors at NIH couldn’t believe that I had had one in 2001 because, as they noted, the capsules weren’t in hospitals, yet.

True that, but I participated in a trial of the technology a decade ago. They knew the doctor who ran the trial and are convinced I was one of the first people to have the test anywhere.

When I first took the test in 2001 in New York City, just months after 9-11, the prototype computer and antennas were built into a heavy black vest that I had to wear for eight hours. I wore it under a navy blazer and went to visit the nearby Guggenheim.

It kinda’ had a suicide-bomber vibe and I felt self-conscious so I walked over to the security guard and explained that the vest was part of a medical test that included swallowing a camera that took flash snapshots of my digestive system and transmitted them by Bluetooth to a computer in my vest.

After this thorough explanation, I realized that the guard spoke very little English. In retrospect, I’m probably lucky he didn’t just shoot me.

Heck, I would have.

I walked around the museum for about an hour convinced that everyone was staring at me and then decided to spend the rest of the day in my hotel room.

Nowadays, the computer looks like an oversized Walkman that you carry in a small shoulder bag for twelve hours. No more vest. It is wired to eight antennas that are attached to your abdomen by two-inch diameter adhesive patches, apparently using Crazy Glue.

They warned me to shave my abdomen and I did, but I wasn’t able to shave it perfectly. (It’s not like I practice.)

They promised to have a nurse remove the patches and the computer at 12:30 a.m., by which time I was sick of being wired to a shoulder bag. Besides, all those wires under my t-shirt made me look fat. 

A nurse walked into the room and laughed because I had been sitting there with my t-shirt pulled up for over an hour so as not to delay the removal process.

“You must be in a hurry to get rid of that thing,” she observed.

Ya’ think?

She removed the first two patches with tiny tugs and occasional swipes of alcohol on the adhesive. It hurt like hell.

But not as much as the third patch.

She pulled on the third patch twice, apparently in an area where I had not shaved as closely, and I wanted to scream. Each tug removed about 1/32nd of an inch of patch. She swabbed it with alcohol.

“Would you like me to pull it off in one tug?” she asked. “Some patients prefer that.”

“Absolutely not!” I replied. “Just keep doing what you’re doing.”

She grabbed the patch and ripped it. It felt like pulling off four square inches of skin.

I partially stifled a scream and stared at her wide-eyed.

The nurse smiled back at me gleefully and said, “All done!”

I left NIH with a perfectly waxed stomach and now I can wear those too-short belly shirts with my new skinny jeans, so there’s that.

My test results were all good. They routinely do a small biopsy and I got the results yesterday. It was normal.

Winston Churchill said, "There is nothing more exhilarating than to be shot at without result."

Maybe not, but a normal biopsy report has to come close.

Search This Blog