Designing and Implementing Surveys in Research Projects


The speaker is Tony Harland, from the University of Otago.

Surveys are typically part of the suite of “qualitative” methods.
They compliment interviews, focus groups and observation. Tony does not view surveys as “quantitative” even though they produce qualitative products.

Mixed method research is a of great value as researchers learn from each method and gain skills and confidence.
Data should be triangulated to provide more knowledge on the subject.
Case studies are starting to dominate, because few researchers have the resources to do more than a case study.
The USA higher education publication market is unique and highly quantitative.

Hard copy
Personally administered
Telephone surveys

What data do you get?
Preferences, options and behaviours.

Mostly researchers use:
Free text

Ensure that the research and the claims made match. Surveys should be trialed before being widely used.

Terminology becomes very important in designing surveys. The middle option in a Likert scale is very neutral. Ensure you know your audience. You are trying to get your data in a few fields. People should largely agree with each other. What will 95% of the participants think? You have to try and anticipate this in advance. You should be capturing the crowd and the exceptions. Who is in the top 2.5% and who in in the bottom 2.5%? Interviews can help researchers to develop their research questions. The questions will then be relevant to the population surveyed.

Design principles:
Should have clear instructions and explanation of the research project.
Include instructions maybe even an example.
Ask the minimum number of questions possible.
Always have a free text option.
One idea per question.
Easier questions first.
Avoid weak items.
Text questions in a pilot.
No double negatives.
No negatives.
Give an indication of time, and the number of questions.

The problem of what people think, believe, perceive and what they actually do.
Surveys can be more honest that interviews. In interviews tend to be biased to the perceived view of the interviewer. Participants try to please. In long surveys the participants can become quite grumpy. When beliefs and actions do not match this is a good area for further research.

Behaviours are more important than beliefs. So the analytic data and the empirical data is of the most value.
Report on the different categories. What do teachers think? Staff developers think? Heads of Department think?

Tony does not think our local population is a good one to use to test a survey. Use people from the intended population to give you feedback.

Ranking gives participants the opportunity to select from a range. This does take thought about what are likely to be the key issues. Ranking questions are slower, and are more complex to analyse.

Headings do help participants understand the survey.

Online Surveys
Don’t know should be an option in knowledge, opinion type questions.
Are the questions mandatory.
Summary statistics in a case study. % only.
Nominal – Categories
Ordinal – median – non para metric, can be used to compare samples. Median is the most important, rather than the average- mean.
Interval – quantitative mean parametric.

A good response rate is over 100 participants. Remember these are case study projects.
Research proposals require numbers, 12-20 is still a good return in some areas. It depends on how many you sent out, and online surveys have a big drop off. 10% would be considered good. Journals need a 20-30% response rate. The relationship of the researcher to the participants does affect the response rate.

Subgroups will become very small.

Pragmatic response
Quality: How to you judge the quality of your own research work? Does this make a contribution to knowledge? Strive to do the best we can, that makes a difference.

Is there enough data to help you gain new knowledge, and better understand your research question?

Does the amount of data allow evidence-based claims that would seem reasonable or fair to you readers?

Research must comply with the hidden rules of research and publication.

Target the journal and find the things that they find acceptable.

This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s