5.5 Statistical significance

Loading...

En provenance du cours de The Hong Kong University of Science and Technology

Social Science Approaches to the Study of Chinese Society Part 2

3 notes

The Hong Kong University of Science and Technology

Social Science Approaches to the Study of Chinese Society Part 2

3 notes

This course is intended as a first step for learners who seek to become producers of social science research. It is organized as an introduction to the design and execution of a research study. It introduces the key elements of a proposal for a research study, and explains the role of each. It reviews the major types of qualitative and quantitative data used in social science research, and then introduces some of the most important sources of existing data available freely or by application, worldwide and for China. The course offers an overview of basic principles in the design of surveys, including a brief introduction to sampling. Basic techniques for quantitative analysis are also introduced, along with a review of common challenges that arise in the interpretation of results. Professional and ethical issues that often arise in the conduct of research are also discussed. The course concludes with an introduction to the options for further study available to the interested student, and an overview of the key steps involved in selecting postgraduate programs and applying for admission. Learners who complete the course will be able to make an informed decision about whether to pursue advanced studies, and should be adequately prepared to write an application for postgraduate study that exhibits basic understanding of key aspects of social science research paradigms and methodologies. Explore the big questions in social science and learn how you can be a producer of social science research. Course Overview video: https://youtu.be/QuMOAlwhpvU Part 1 should be completed before taking this course: https://www.coursera.org/learn/social-science-study-chinese-society

À partir de la leçon

Quantitative Analysis

Week 5 will give you a taste of the basic methods for quantitative analysis. From there you should be able to identify key issues when interpreting results and discuss implications for research.

5.1 Overview: Quantitative Analysis3:02

5.2 Tabulations10:53

5.3a Correlation and regression I12:06

5.3b Correlation and regression II6:18

5.4 Regression to the mean18:53

5.5 Statistical significance12:45

5.6 Type I errors9:31

5.7 Type II errors7:29

5.8 Summary Quantitative Analysis3:14

Rencontrer les enseignants

Cameron Campbell
Professor of Social Science
Associate Dean of the School of Humanities and Social Science

[MUSIC]

Hi, in this module I want to talk about statistical significance.

Now statistical significance is an important concept in statistics.

If you've taken any kind of statistic's class you've almost

certainly encountered it, and it's widely misunderstood.

Now for the purpose of this module, I'm not going to get into all the formulas,

you'll have to take a statistics class to learn how to compute a significance

test and so forth.

What I want to talk about is the ways in

which significance is sometimes misunderstood,

and help caution you against misapplication or misunderstanding.

It's maybe useful even if you have had a statistics class because statistics

classes tend to be heavily formal without thinking about the implications for

research.

Now a significance test or a measure of statistical significance,

all this assumes that we are making a measurement in

a sample that is drawn from some larger population.

So statistical significance hypothesis testing has always

been embedded in the framework of thinking about generalizing

to some large population that we cannot measure in its entirety.

By making measurements on a sample from that larger population, and

then making measurements and then somehow generalizing.

And it's quite important and often forgotten, that actually most of

the mathematics that underlies tests of statistical significance,

hypothesis testing and so

forth assumes that our sample is actually a random draw from the larger population.

In other words, in drawing our sample, we follow the procedures that

we talked about in previous lectures of randomly sampling or probability sampling

from some larger population in which we're actually interested.

So measurement in a sample, when we go out and, for example,

conduct an opinion poll of a few hundred people, we compute the percentage

of people that hold some attitude or believe in something.

The measurement we make in that sample is what we call

an estimate of a population parameter.

So out there in a population in which we are interested in, we think of there being

numeric values, parameters, that describe characteristics of that population.

Perhaps the total percentage of people with a particular attitude or

a preference for a particular party.

And then we draw our sample and we make a measurement.

And then we want to use that measurement to figure out how to say something about

the corresponding parameter that describes that in the larger population.

Now, this is very important, all of the mathematics

involved in the calculation of statistical significance

assumes that the measurement is influenced by chance.

That when we're drawing a sample at random from some larger population

in order to perform some sort of measurement on them,

measure the percentage of people that support a particular party and so forth.

The composition of that sample, again is based on luck of the draw, random chance.

So whatever the population parameter is,

whatever the true percentage of people who support one political party or another.

In the sample that we draw, it's like drawing colored balls from an urn

if you think about examples you may remember from probability classes.

And so even if in a urn we have 500 red balls and

500 blue balls, if we randomly draw 10 balls or

20 balls we may not always get an exact even distribution by color, right?

By the laws of chance,

occasionally you'll get perhaps more red balls or more blue balls.

A little bit different from the overall share of the urn from what we are drawing.

So the same reasoning applies to samples drawn from a larger population.

If we have some share of the population,

which has some affiliation with particular political party.

When we draw a sample by the laws of chance, yes, maybe on average that

sample will resemble the distribution in the larger population but

there's always a probability that it may tilt in one direction or another.

You may have people with affiliation with a particular party overrepresented in

the sample even if it's a well design sample.

Again, just by the laws of chance.

So a significance test, what it does is really trying to do is assess whether or

not the value that we measure in our sample, how likely that

value is prompts the proportion with the affiliation was a particular party.

How likely we are to see a value, right one we have observed

in a sample if the true population parameter the true value

in a larger population were some hypothesized value.

So typically, we put this into the format of a null hypothesis,

where we have a hypothesis that the parameter in the larger

population is some specific value.

And then a significance test works out the probability that just

by the luck of the draw, we could get a distribution like the one

that we have on our sample, purely as a result of random chance.

So going back to the voter analogy, we might hypothesize or

null hypothesis that the people are evenly divided between two parties,

or null hypothesis might be that the split is 50-50.

And then in drawing out our sample, measuring it,

we find that perhaps 48% of the people favor one party and 52% favor the other.

A significance test would be an assessment of

how likely we would be to get a split like that,

48-52 if the population parameter or

the hypothesized value of 50-50.

As you might imagine,

more skewed distributions in the sample get progressively less likely.

So we often talk about a p-value, that's the specific estimate of the probability

of getting a value like the one we observed if the null hypothesis were true.

So in our example, it would be the probability

of observing a 48-52 split, if the true split

was as hypothesized 50-50 in the larger population.

So when we talk about correlation or regression,

the null hypothesis is usually that there is no relationship between two values,

the correlation coefficient, zero, or the regression coefficient is zero.

The changes in one variable are not associated in a systematic

fashion with changes in the other variable.

So statistical significance simply means that if the null hypothesis were true,

chance variation would be unlikely to yield a sample that produces one

that looks like the value that we've actually measured.

So for example, if we run a correlation or we ran a regression, and we got some

estimate off a correlation coefficient when we carried out a significance test.

And we claim that the correlation coefficient was significant

at the 0.05 level.

Then, all that it's saying is that there's only a 0.05

chance that if there was no relationship between the two variables in

the original population from which we drew the sample.

There's only a 0.05% chance that in a sample of the size that we've drawn,

we would see a correlation as large as the one that we've calculated.

And we might think that gee, 0.05, that's quite small,

a 1 in 20 chance that are result can be the product of random variation and

the composition of our sample.

And we might decide that this correlation really is not zero.

That somewhere in the larger population,the true correlation must be

not zero.

We reject the null hypothesis.

So what influences statistical significance?

One is the magnitude of the effect in the population for the sample.

So essentially, the further a measured value is from the hypothesized value.

And for example, the larger coefficient, the larger correlation coefficient,

the more likely it is to show up as being statistically significant.

Sample size matters as well.

So if you're thinking about same measured effects or

same measured correlation coefficients across samples of different sizes.

It's more likely that in a large sample that correlation coefficient will be

reported as statistically significant.

The sample size feeds into the calculation of the p-value,

feeds into the calculation of the test statistics

that shape the determination of the p-value.

And finally, the amount of error.

So if we're measuring our outcomes, our variables, with a lot of error,

that adds noise to the system.

And then it turns out that it makes it harder to actually

show a statistically significant relationship, or effect.

Because again, we're introducing noise into the system and

it becomes harder to rule out the possibility

that something that we've observed is the product of random chance.

So what are the limitations of significance testing?

The most important is that significance test are not proof of causal relationship.

Yet some of though have nothing to do with causality by themselves anyway.

Proof of cause and effect can only come from an appropriate study design.

Ideally, perhaps a actual control of treatment design.

Might be a natural experiment like we talked about in previous lectures or

a quasi experiment or instrumental variables.

But one way or the other, statistical significance by itself

does not say anything about cause and effect.

Significance tests assume that a sample is drawn at random from a larger population.

As soon as you got a sample that doesn't hold that assumption or

doesn't follow that assumption.

You're actually violating a fairly important assumption about

the computational test of statistical significance and

it's become a little harder to interpret the results.

You have to think more carefully about what you are actually

doing in terms of claiming statistical significance.

So we need to exercise caution when we're interpreting test of

statistical significance from non-random samples.

So test will again only show that an absolute value was unlikely

if the parameter in the population was as hypothesized but statistical

significant test never approves that absolute value is impossible.

So there is always some chance,

it might be very very small that whatever we have observed in our sample,

when we measured it, is actually the product of random chance.

And that the population parameter is in fact different from what we claimed.

And we'll talk about some of the problems that that leads to in the next module,

on type one errors.

Explorer notre catalogue

Rejoignez-nous gratuitement et obtenez des recommendations, des mises à jour et des offres personnalisées.

Coursera propose un accès universel à la meilleure formation au monde, en partenariat avec des universités et des organisations du plus haut niveau, pour proposer des cours en ligne.

© 2018 Coursera Inc. Tous droits réservés.

Télécharger dans l'App Store

Disponible sur Google Play