[MUSIC] So we talked about publication bias and on the y-axis of that plot we had effect size. So what is effect size? Well, if you call P value was just the probability of, in repeated experiments getting an effect of at least that size or more extreme. And so all it tells you is yes or no, whether it was significant according to some threshold. It doesn't tell you how significant it was or how important it was and that's what effect size tries to capture. So the effect size in a situation where you're comparing the means of experimental group and control group, then the effect size is the mean of experimental group minus the mean of the control group divided by the standard deviation. So this tells you not just significant, but how significant and this measure's used prolifically in meta-analysis to combine results from multiple studies and it's useful to help standardize across studies with parameters, different sample sizes in particular, for example. And so in general, you've got to be careful with this, because average results in different experiments can produce nonsense if you sort of violate the assumptions under which the experiments were conducted, but it can be done. And so another caveat here is that there's other definitions of effect size exists, there's odds ratio and correlation coefficient, but we're just gonna talk about this one. And so there's an argument here that the effect size should always be reported along with the P value even though this is very rarely done. And that argument is delivered by Robert Coe in this paper presented at the annual conference of the British Educational Research Association 2002, that's cited here at the bottom of the slide. So a little more precisely, the effect size is the standardized mean difference where the mean of group 1 minus the group 2 divided by we said, the standard deviation. And here, I'm writing this as the standard deviation sub-pooled and there's lots of ways to compute a value for this notion of pooled. One simple way is just to take the standard deviation of the control group and one by one argument here. Hey, the variations is supposed to be the same, because the two groups are drawn from the same population. Another way is to compute the actual pooled, the standard deviation, which is an expression that looks like this. And so we're not going to go into much more detail that that, but just to point out that the overall notion is pretty simple and that it's not unreasonable to use a straightforward definition of standard deviation. So what's a big effect size? Well, because it is standardized, you can actually reason about what might be big and what might be small. So this is again, one of these cases where it sort of made up on the fly, but you can. One heuristic by Jacob Cohen is a small effect size is 0.20, a medium effect size is 0.50 and a large effect size is 0.80. So remember, this is dividing by the standard deviation. So this is sort for every bit of difference in mean, how much variance are you accounting for? And then finally, I'm not gonna talk too much detail about confidence intervals, but I do want to mention that the confidence interval of effect size just to give you the intuition for this. So, an advantage is that these are maybe easier to interpret in terms of actual decision-making. What is a 95% confidence interval of the effect size mean? It mean we have repeated the experiment 100 times, we expect that that interval would include this particular effect size measured in this experiment 95 out of 100 times. A corollary of this is that if that interval 95% confidence interval includes 0.0, that's equivalent to staying the result is not statistically Significant. That means that 90% of the time that interval would include no effect, what so ever. And equivalently, if that interval does not include 0.0, then it is statistically significant. So that's some more terminology here. [MUSIC]