0:00
[MUSIC]
I want to talk now about some key considerations in questionnaire design.
In fact, questionnaire design is as much of an art as a science.
Most students learn questionnaire design not so much from classes but
by actually joining ongoing research projects.
In which questionnaires are being designed and then looking at them and
learning about the process in a hands on way.
That said, I can at least outline a few key principles to keep in mind
if you design a questionnaire.
One is that you need to minimize the amount of thinking
that a respondent has to do to answer the question.
Basically, as much as possible, the questions should be clear, concise, short.
They should not require additional explanation.
Hopefully the respondent, upon hearing the question foreseeing it will immediately
know exactly what is needed from them without further interpretation or
explanation.
1:05
Each item in a questionnaire,
that is each question should only seek to measure one underlying dimension.
Don't mix apples and oranges.
So if you are interested in collecting information about age.
Then only ask about age.
If you're asking about some attitude about a particular topic,
the question should not refer to any other topics.
1:32
You also want to avoid too much reading or listening.
So it may be that respondents will tune out If they have to read too much or
listen to too much, there is a few special exceptions, but for
the most part the questions again should be short.
1:49
Make sure that the questions are relevant to the respondent's education,
experience, knowledge and so forth.
Ask questions that the respondent is capable of answering.
So one particular problem that I've seen is that sometimes,
people that are new to questionnaire design, they may ask respondents not about
their personal situation, but about their opinion of what people like them feel.
So rather than asking a respondent, what they think of a particular social topic.
They may ask a respondent what people like them think about that social topic,
in other words, asking the respondent to speak on behalf of some larger group.
Now, that's not normally a useful approach, because a respondent may
actually not know what other people that make up their group are actually thinking.
2:45
More technical issues, that questionnaires typically use skip patterns.
To ensure that respondents are only asked questions,
that are specifically relevant to their experience.
So in a survey, that includes a sample that has both men and
women, if there are questions about previous pregnancies normally
a skip pattern would be used according to which men who complete
the survey would be directed away from answering questions about pregnancy
because obviously they wouldn’t be able to answer them in a meaningful way.
3:21
There's also some issues that come up in the ordering of questions,
some general principles.
The ordering of questions is important.
Most surveys begin by putting the most straightforward questions about
non-sensitive issues, at the beginning of the survey.
This also helps build some trust or rapport with the interviewee because
the questions are fairly innocuous and straightforward.
Questionnaires then therefore begin with collection of information about social
demographic or economic characteristics of the interviewee.
And then questionnaires save the most sensitive questions for
the end of the survey.
By which time, hopefully, the interviewer has established a relationship with
the interviewee and the interviewee has opened up a bit become more trusting.
4:14
The ordering of questions is especially important when it comes to
attitudinal questions, for example on opinion polls.
We know that the ordering of questions about attitudes for
particular topics may influence the results that the respondents give.
4:33
I want to talk a little bit about online surveys.
They've gotten a lot of attention.
Now, you've probably,
at some point seen some online survey since you've been browsing the web.
You may have been invited to complete an online survey.
Again quite often, they are perhaps used more for entertainment than for
anything else.
Perhaps a news website may invite readers to
complete a very brief survey indicating their attitude about a particular poll.
Now, online surveys do seem attractive in that they promise to
allow the collection of an enormous amount of information very quickly.
So we see sometimes online surveys with hundreds of thousands, or
even millions of responses.
That come in very quickly.
The problem is that we really don't know how representative an online
population is, whether you can generalize from anything that you
learn online to some other real population out there in the world.
You think about what kind of people spend a lot of time on the internet and
might even have the time free to complete an online survey.
5:41
There are proposed solutions especially when it comes to academic
research involving online surveys, involving re-waiting and so forth.
So, for example if an online survey Includes questions about the demographic
characteristics of the respondent, their age, their sex, their race, and so forth.
People have suggested that these can be used to
reweight the answers to make the respondents on the online survey,
the data representative of some larger population.
Again, that's controversial, because online users,
even if you are able to re-weight the numbers to make it appear
representative of the larger population, the people that are online may still be
different from the quote real population in important ways.
What's most promising, especially for academic research, are hybrid approaches
in which subjects perhaps are initially contacted in person, or by mail and
perhaps even interviewed and then follow up is conducted online.
Or subjects are recruited with a piece of mail from an official address
perhaps especially for a government survey that explains the survey and
then gives a web address for the respondents to go to complete the survey.
There, because we have some control over the sample,
we can select the people we invite to participate in the survey
according to traditional sampling techniques.
And then the online portion is basically a convenience for
the people that have been selected and agree to participate in the interview,
then we have a lot less problem with representativeness, than an online survey
that consists of simply putting up a web page, and then asking people to click on
a link to complete a survey, again, with no attempt at traditional sampling.
7:40
So, this sort of online hybridization is especially important and
potentially useful for longitudinal surveys.
So, there are longitudinal surveys that are hybrids,
where perhaps at regular intervals, perhaps a few years apart, there
are in person surveys where interviewers go to visit people at their homes, and
they have long questionnaires but maybe takes several hours to complete.
But then in the time between those in person visits, the respondents
are invited to complete, perhaps brief update questionnaires online.
Perhaps providing information about recent activities like employment or
their relationships status.
So again, this integration of detailed data collection in person and
then more sporadic and less detailed data collection online in between those in
person visits is a very promising approach for the future of longitudinal surveys.
So hopefully, I've at least introduced you
to some of the basic considerations the practical considerations
you'll have to keep in mind if you're actually going to conduct a survey.
Assuming that you figured out how you're going to do your sampling and
you need to start thinking about how you're going to design your questionnaire
and what sort of format the survey will be in.
Cameron CampbellProfessor of Social Science
