{"id":5074,"date":"2013-01-29T07:22:39","date_gmt":"2013-01-29T07:22:39","guid":{"rendered":"http:\/\/meeseeks:5080\/blog\/?p=5074"},"modified":"2013-01-29T07:22:39","modified_gmt":"2013-01-29T07:22:39","slug":"bayesianism-in-science","status":"publish","type":"post","link":"https:\/\/vukutu.com\/blog\/2013\/01\/bayesianism-in-science\/","title":{"rendered":"Bayesianism in science"},"content":{"rendered":"

Bayesians are so prevalent in Artificial Intelligence (and, to be honest, so strident) that it can sometimes be lonely being a Frequentist.\u00a0\u00a0 So it is nice to see a critical review of Nate Silver’s new book on prediction from a frequentist perspective.\u00a0\u00a0 The reviewers are Gary Marcus and Ernest Davis from New York University, and here are\u00a0some paras from their review<\/a> in The New Yorker:<\/em><\/p>\n

\n

Silver\u2019s one misstep comes in his advocacy of an approach known as Bayesian inference. According to Silver\u2019s excited introduction,
\nBayes\u2019 theorem is nominally a mathematical formula. But it is really much more than that. It implies that we must think differently about our ideas.<\/em>
\nLost until Chapter 8 is the fact that the approach Silver lobbies for is hardly an innovation; instead (as he ultimately acknowledges), it is built around a two-hundred-fifty-year-old theorem that is usually taught in the first weeks of college probability courses. More than that, as valuable as the approach is, most statisticians see it is as only a partial solution to a very large problem.
\nA Bayesian approach is particularly useful when predicting outcome probabilities in cases where one has strong prior knowledge of a situation. Suppose, for instance (borrowing an old example that Silver revives), that a woman in her forties goes for a mammogram and receives bad news: a \u201cpositive\u201d mammogram. However, since not every positive result is real, what is the probability that she actually has breast cancer? To calculate this, we need to know four numbers. The fraction of women in their forties who have breast cancer is 0.014, which is about one in seventy. The fraction who do not have breast cancer is therefore 1 – 0.014 = 0.986. These fractions are known as the prior probabilities. The probability that a woman who has breast cancer will get a positive result on a mammogram is 0.75. The probability that a woman who does not have breast cancer will get a false positive on a mammogram is 0.1. These are known as the conditional probabilities. Applying Bayes\u2019s theorem, we can conclude that, among women who get a positive result, the fraction who actually have breast cancer is (0.014 x 0.75) \/ ((0.014 x 0.75) + (0.986 x 0.1)) = 0.1, approximately. That is, once we have seen the test result, the chance is about ninety per cent that it is a false positive. In this instance, Bayes\u2019s theorem is the perfect tool for the job.
\nThis technique can be extended to all kinds of other applications. In one of the best chapters in the book, Silver gives a step-by-step description of the use of probabilistic reasoning in placing bets while playing a hand of Texas Hold \u2019em, taking into account the probabilities on the cards that have been dealt and that will be dealt; the information about opponents\u2019 hands that you can glean from the bets they have placed; and your general judgment of what kind of players they are (aggressive, cautious, stupid, etc.).
\nBut the Bayesian approach is much less helpful when there is no consensus about what the prior probabilities should be. For example, in a notorious series of experiments, Stanley Milgram showed that many people would torture a victim if they were told that it was for the good of science. Before these experiments were carried out, should these results have been assigned a low prior (because no one would suppose that they themselves would do this) or a high prior (because we know that people accept authority)? In actual practice, the method of evaluation most scientists use most of the time is a variant of a technique proposed by the statistician Ronald Fisher in the early 1900s. Roughly speaking, in this approach, a hypothesis is considered validated by data only if the data pass a test that would be failed ninety-five or ninety-nine per cent of the time if the data were generated randomly. The advantage of Fisher\u2019s approach<\/a> (which is by no means perfect) is that to some degree it sidesteps the problem of estimating priors where no sufficient advance information exists. In the vast majority of scientific papers, Fisher\u2019s statistics (and more sophisticated statistics in that tradition) are used.
\nUnfortunately, Silver\u2019s discussion of alternatives to the Bayesian approach is dismissive, incomplete, and misleading. In some cases, Silver tends to attribute successful reasoning to the use of Bayesian methods without any evidence that those particular analyses were actually performed in Bayesian fashion. For instance, he writes about Bob Voulgaris, a basketball gambler,
\nBob\u2019s money is on Bayes too. He does not literally apply Bayes\u2019 theorem every time he makes a prediction. But his practice of testing statistical data in the context of hypotheses and beliefs derived from his basketball knowledge is very Bayesian, as is his comfort with accepting probabilistic answers to his questions.<\/em>\u00a0
\nBut, judging from the description in the previous thirty pages, Voulgaris follows instinct, not fancy Bayesian math. Here, Silver seems to be using \u201cBayesian\u201d not to mean the use of Bayes\u2019s theorem but, rather, the general strategy of combining many different kinds of information.
\nTo take another example, Silver discusses at length an important and troubling paper by John Ioannidis, \u201cWhy Most Published Research Findings Are False<\/a>,\u201d and leaves the reader with the impression that the problems that Ioannidis raises can be solved if statisticians use Bayesian approach rather than following Fisher. Silver writes:
\n[Fisher\u2019s classical] methods discourage the researcher from considering the underlying context or plausibility of his hypothesis, something that the Bayesian method demands in the form of a prior probability. Thus, you will see apparently serious papers published on how toads can predict earthquakes\u2026 which apply frequentist tests to produce \u201cstatistically significant\u201d but manifestly ridiculous findings.<\/em>\u00a0
\nBut NASA<\/small>\u2019s 2011 study of toads<\/a> was actually important and useful, not some \u201cmanifestly ridiculous\u201d finding plucked from thin air. It was a thoughtful analysis of groundwater chemistry that began with a combination of naturalistic observation (a group of toads had abandoned a lake in Italy near the epicenter of an earthquake that happened a few days later) and theory (about ionospheric disturbance and water composition).
\nThe real reason that too many published studies are false is not because lots of people are testing ridiculous things, which rarely happens in the top scientific journals; it\u2019s because in any given year, drug companies and medical schools perform thousands of experiments. In any study, there is some small chance of a false positive; if you do a lot of experiments, you will eventually get a lot of false positive results (even putting aside self-deception, biases toward reporting positive results, and outright fraud<\/a>)\u2014as Silver himself actually explains two pages earlier. Switching to a Bayesian method of evaluating statistics will not fix the underlying problems; cleaning up science requires changes to the way in which scientific research is done and evaluated<\/a>, not just a new formula.
\nIt is perfectly reasonable for Silver to prefer the Bayesian approach\u2014the field has remained split for nearly a century, with each side having its own arguments, innovations, and work-arounds\u2014but the case for preferring Bayes to Fisher is far weaker than Silver lets on, and there is no reason whatsoever to think that a Bayesian approach is a \u201cthink differently\u201d revolution. \u201cThe Signal and the Noise\u201d is a terrific book, with much to admire. But it will take a lot more than Bayes\u2019s very useful theorem to solve the many challenges in the world of applied statistics.” [Links in original]<\/p><\/blockquote>\n

Also worth adding here that there is a very good reason experimental sciences adopted Frequentist approaches (what the reviewers call Fisher’s methods) in journal publications.\u00a0 That reason is that science is intended to be a search for objective truth using objective methods.\u00a0 Experiments are – or should be – replicable\u00a0 by anyone.\u00a0\u00a0 How can subjective methods play any role in such an enterprise?\u00a0 Why should the\u00a0 journal Nature<\/em> or any of its readers care what the prior probabilities of the experimenters were before an experiment?\u00a0\u00a0\u00a0 If these prior probabilities make a difference to the posterior (post-experiment) probabilities, then this is\u00a0the insertion of\u00a0a purely subjective element into something that should be objective and replicable. And if the actual numeric values of the prior probabilities\u00a0don’t matter to the posterior probabilities (as some Bayesian theorems would suggest),\u00a0then why does the methodology include them?\u00a0\u00a0
\n \n<\/div>\n","protected":false},"excerpt":{"rendered":"

Bayesians are so prevalent in Artificial Intelligence (and, to be honest, so strident) that it can sometimes be lonely being a Frequentist.\u00a0\u00a0 So it is nice to see a critical review of Nate Silver’s new book on prediction from a frequentist perspective.\u00a0\u00a0 The reviewers are Gary Marcus and Ernest Davis from New York University, and […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[23,86,74,82],"tags":[],"class_list":["post-5074","post","type-post","status-publish","format-standard","hentry","category-decision-theory","category-probability-theory","category-science","category-uncertainty","p1","y2013","m01","d29","h07"],"_links":{"self":[{"href":"https:\/\/vukutu.com\/blog\/wp-json\/wp\/v2\/posts\/5074","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/vukutu.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/vukutu.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/vukutu.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/vukutu.com\/blog\/wp-json\/wp\/v2\/comments?post=5074"}],"version-history":[{"count":0,"href":"https:\/\/vukutu.com\/blog\/wp-json\/wp\/v2\/posts\/5074\/revisions"}],"wp:attachment":[{"href":"https:\/\/vukutu.com\/blog\/wp-json\/wp\/v2\/media?parent=5074"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/vukutu.com\/blog\/wp-json\/wp\/v2\/categories?post=5074"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/vukutu.com\/blog\/wp-json\/wp\/v2\/tags?post=5074"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}