[MUSIC] Now, in the previous lecture, we talked about Bayesian statistics. Formally quantifying your prior belief, updating it based on data into a posterior belief. There's some debate in the literature about whether people should always use Bayesian statistics or not. Depends on how happy you are with incorporating mainly subjective prior information into your statistical reasoning. Here I want to talk a little bit about how it's important to at least use Bayesian statistics some of the time. Even if you don't want to formally quantify it - to think about statistical inferences. So even if you don't want to quantify your inferences using a subjective prior, it's always good to keep prior information into account in some way. Now taking prior probabilities into account is very often smart thinking. We can look at an example from medical science to see the point of thinking about priors. Now, let's take a situation where about 3% of the people in the population are sick. We have a test that can identify whether you have this illness or not, and this is correct 80% of the time. So this equals a statistical power of 80%. In 80% of the cases where you are sick, the test will tell you that you are actually sick. Of course we also have a Type I error rate. And we've seen that in medical science, this is often a little bit higher just to be sure, better safe than sorry. So in this example, there's a 13% false positive rate. Or, 13% of the people who don't have this illness will be identified by the test as maybe being sick. If you get the test, and there's a positive result - and remember, positive result in a test in medicine is not very positive. It means that you've been identified as you might have an illness. Given a positive test result, how probable is it that a patient is actually sick? Now let's visualize this by looking at a large population of individuals. And in this population, we see that about 3% of the individuals, the red ones in the corner, are actually sick. They actually have this illness. And it means that 97% of the people are not sick to begin with. So this is a situation where the prior probability of being sick is actually quite low. And then we collect some data; we perform a test. And in this test, we see that a number of individuals are identified as having a specific sickness. Now, four of the people that actually have this sickness - we can see that we had an 80% probability of exactly determining that they had this. So, the true positive rate, or the statistical power, was 80%, which means that four out of these five will be correctly identified as being ill. But we also had this large false positive probability of 13%. So we see that in this cluster of individuals, there's actually quite a large number of people that are identified as maybe being ill based on a Type I error. So, given that this is the data that we have observed, what is now the probability of actually being ill? And we can see that this is just the probability of being one of the red individuals, divided by the probability of one of the black individuals. We can also formally quantify this. So, again, as we did the previous time, we have the prior times the likelihood, which equals the posterior distribution. In this case, it means that before we started, we actually have 175 people here on this slide. 5 of them were actually ill to begin with, and 170 weren't ill. Then we have the data: 4 out of 5 were correctly identified as being sick, and 22 out of 170 were identified as maybe being sick. But these were all false positives. And if we combine this and then calculate the posterior probability, we actually see that it's - based on the outcome of the test - Now only 18% likely that you're actually ill. So this probability is surprisingly low. And we see that many medical doctors overestimate the probability that someone is really sick if the frequency of this illness is very low in the population. So at least some implicit Bayesian statistics is useful. Realizing that if something is pretty rare to begin with, a positive test result might not tell you so much yet, and follow up tests are definitely required. So, taking prior information into account can lead to better inferences. I'll show you a very intuitive way to do this when you're thinking about p-values, and when you're using p-values to more or less update your belief. Now p-values were not meant to do this. But there are some statistics that point out an easy approach to give you the most likely posterior probability of the hypothesis being true, given a p-value. And again, this is not formal Bayesian statistics, but it's a very easy way to at least use a little bit of Bayesian thinking. Now, this is known as a nomogram, this graph that we have. And there are three vertical lines, one in the left, one in the middle, and one in the right. This type of graph is very often used in medical institutions. Where for example, nurses cannot be expected to perform formal Bayesian statistics on the fly. When they see someone cough and want to think about the probability that this person is either just having the flu or having Ebola. But using a graph like this can very easily give you some indication of the posterior probability of a hypothesis, given a prior probability and an observed test result. Now, in this graph on the left, we see the prior probability that the null hypothesis is true. In the middle graph, we see the p-value that we've observed, and on the right, we can get the posterior probability that the null hypothesis is true. Now, let's start by thinking of a situation where we - before we collect some data - think that anything goes, so a person might be sick or not, and both of these are 50% probable. So here, I've pointed an arrow at the 50% prior probability that the null hypothesis is true. We collect some data, and then let's try to quickly use this graph to update our belief. In this case, we collected some data, and we found a p-value of 0.05. So, this is exactly on the border of statistical significance. We got lucky: It's just exactly statistically significant. But as many people who practice Bayesian statistics will tell you, such a high p-value, relatively high, even though it's statistically significant, is actually not a lot of evidence. And this graph nicely illustrates this. If you had a 50% prior belief that the null hypothesis is true, you collected some data. Then on the right, the point where the line hits the vertical axis is actually at about 29% posterior probability that the null hypothesis is true. So we started out with a prior belief that it was 50% likely that the null hypothesis is true. We observe some data, and we find a barely statistically significant p-value. And then, of course, it make sense that our prior belief that the null hypothesis is true is a little bit lower. But you see that it's not a lot lower. It's actually still 29% likely that the null hypothesis is true even though we've observed the p-value of 0.05. Now this is one of these ways in which you can use Bayesian thinking to at least acknowledge that a relatively high p-value doesn't mean that the null hypothesis is now completely unlikely. Let's say that you had the same prior belief, but you observed a much, much lower p-value of 0.001. In this case, we can see that the point where this line hits the right axis is around 2%. So, given this quite strong data, we now have a prior belief that the null hypothesis is true of only 2%. So that's a huge drop. We started out with th 50% prior belief that the null hypothesis might be true. We collect data, and now we think it's only 2% likely that the null hypothesis is true. So we've learned quite a lot. Now we can also used this graph in a very interesting way to think, "Okay, so what about a hypothesis that a priori - before I collected some data - I found really unlikely to be true?" Now, it's always a bit tricky, because we're talking about subjective belief. So what is an unlikely hypothesis? At least sometimes people in the literature will identify surprising results themselves. They say, "We published this result, and this is a very surprising finding". So that means that before they started, it was at least less likely to be untrue, than it was likely to be true. We're not at 50-50, but the prior probability that a null hypothesis is true was somewhat higher. Let's put this prior belief at about 75% - sounds about fair. So we have a 75% probability that a null hypothesis is true before we start. We're slightly skeptical that something is going on. We again collect some data, and we find a p-value of 0.05. Then the line continues on to the axis on the right, and again, this gives us the minimum posterior probability that the null hypothesis is true. And you see - because we started with a 75% prior probability that the null hypothesis is true - we observe a just significant result. But after this, it's actually still more likely that the null hypothesis is true than that the alternative hypothesis is true. So this is an interesting observation. Even though we observe a statistically significant result - because beforehand this result was actually very, very unlikely to be true - After collecting the data, we are still not convinced. So this is an easy way without very formal Bayesian statistics to at least give you a rough estimates of the best case scenario for the null hypothesis. Which might actually not be very good based on very high p-values. This is in line with a claim by Laplace, who says that extraordinary claims require extraordinary evidence. So very high p-values in a situation where we think that the hypothesis is extremely unlikely. Think about things like precognition. Do you think that precognition is true or not? Well, you might have some belief that it is possible. But you don't think that it's extremely likely to be true. Otherwise we would have seen some evidence for it by now. So your prior belief that precognition is real is pretty low. Or in other words, your prior belief in the null hypothesis that there is no precognition - this null hypothesis - your belief is very high. So if you want to show that there is indeed something like precognition, then you need to provide extraordinary evidence. Extremely compelling data that will lead you to slowly update your belief. Now, in this lecture I've talked about how you can use Bayesian thinking even when you don't want to use formal Bayesian statistics when report your results. You prefer to use frequentist statistics for example, and you don't want to quantify your prior belief and update it based on the data into a posterior belief. It still makes sense to use a little bit of Bayesian thinking. To keep in mind how likely the hypothesis was before the data was collected and weigh the evidence in relation to the extraordinariness of the claim. [MUSIC]
