Three Survey Design Best Practices

I recently finished a departmental survey. The results were a mix of interesting, confusing, and completely useless. In particular I was trying to determine which new services would be most beneficial to the department. For nearly every suggestion a lot of people thought it was great and should be implemented right away, others said it was useless and not worth the time. *Sigh*

In addition, I was asking how often people used resources. Those results were even more surprising. Since I’ve been collecting usage data on the site, I know how often individuals used these resources, but I was curious about the perceptions. On several resources I’ve recorded that they’ve been used, well, never. Maybe one or two clicks over the year. Nonetheless, several people say they used it weekly or monthly. Hmmm.

Regardless of any individual responses, it generated a lot of useful information. Since I’m the planning type, I reviewed a lot of best practices. For those who like lots of information. There are good guides from Survey Monkey, Purdue and The University of Texas. For those looking for quick information. Here are three best practices that I found most valuable.

1) Have a goal for each question

To design the best question, it helps to have an idea why the question is being asked. For example, I wanted to know how to best use staff time, so I tried to break down specific projects into each question. This way, I can see that 70% of staff want photo archives but only 40% want government document archives.

It is also important to share this goal with the audience. Another recent survey was exploring journal usage. We stated at the beginning that we need to remove journals due to space and budget constraints. I believe this framed the whole purpose so that staff members  really considered the merits of each title. The comments at the end were also much more direct and helpful than in other surveys.

2) Create Concrete Questions

A question that is more specific will generate better results than more open-ended abstract questions. Consider the following question:

How often do you use the library?

This is open to user interpretation. Do we mean how often do you go the building? How often do you look up information in the OPAC? How often do you download articles? Each user can look at this a different way. Contrast this with another question:

How often do you use an electronic database like Web of Science or Google Scholar?

Now the patron has much more clear information and a couple of examples to help the make quantitative estimates. It’s hard to tell if a question is concrete enough. Often there is no way to know until the results show wild swings in responses.

3) Avoid rating type questions when possible?

This is not fully agreed upon in the literature, but I have found it gives much better results. What is a rating question? Those include question such as:

The library should focus on collecting books instead of journals:

Strongly Disagree                                  Strongly Agree

1                                          3                                      5

These questions are also open to interpretation. Higher scales are even more difficult. On a rank of 1 – 10, what is the difference between 2 and 3?

Instead breaking these down to more specific objective values are preferred. Instead of the question above, we could ask:

How often do you go the library for journals?

Daily, Weekly, Monthly, etc.

Now we have results that are more factual.

Opinion questions are hard to completely eliminate, just reduce them when possible.

Piece of Cake

Now all that’s left is compiling the results, translating that into service design and then implementation. How hard can that be?

Photo credits: http://www.flickr.com/photos/bettyspics/2398443150/sizes/o/