No Short Cuts! Test and Pilot Data Collection Instruments

This blog post is written from my own unfortunate experience of not testing and piloting a data collection instrument properly. My failure to do so not only affected the quality of the data (as many persons did not answer the ambiguous questions on the survey) but also led to delays when some of the data had to be collected again.

How did this happen in the first place?

Well, in an attempt to reduce costs and to get things ‘moving as quickly as possible’, the client insisted that a questionnaire that was used in a previous baseline study, be adopted for this follow up study. Notwithstanding the fact that the questionnaire was fraught with misspellings, ambiguous sentences, incoherent flow and sequence of questions as well as culturally insensitive questions that probed religious affiliation and sexual identity.

As some of these issues were readily apparent, I did manage to negotiate a compromise with the client. The ‘solution’ we agreed on was to test the questionnaire with a sample of 6 persons. This would be quick, dirty and most of all, low cost (music to my client’s ears).

Afterwards I realized that this sample was not truly representative of the target population who would be the actual participants of the study. More specifically, the persons who tested the questionnaires were all residents of an institution and were mostly men. It therefore meant that their perspectives may have differed from other participants of the actual study that included women and who have never been institutionalized.

However, before we proceed further, what do we actually mean by ‘testing a data collection instrument’?

Testing a data collection instrument such as a questionnaire for a survey, involves using that questionnaire before the actual study or evaluation in conditions that are very similar to the official administering of your survey.

This test helps to eliminate certain faults on your questionnaire that I mentioned earlier. These include spelling and grammar issues, poorly constructed sentences, questions that are ambiguous/lack clarity and question that are not culturally sensitive, lengthy and redundant.

Remember, a data collection instrument may seem “perfect” from your point of view. The only way to be sure is to test it.

Apart from spotting grammatical issues, testing a data collection instrument helps you to determine its validity. In other words, during the testing phase you will get a chance to see if the data collection instrument actually measures what it is intended to measure.

In my case, it was only after the initial rounds of actual data collection did it become clear that a few questions that were meant to be measuring religiosity (e.g. ‘When was the last time you of attended a place of worship?’) were in fact measuring access to places of worship and not necessarily how 'devout' one was.

A focus group discussion later revealed that persons did not think frequent attendance to a place of worship meant a greater connection to a higher power. They felt just as religious and devout staying right at home. The frequency of their attendance to a place of worship was more dependent on how easy it was to get to the location from home. This is why mixed-methods and triangulation of data (in this case from the survey and focus group) is so important.

Testing the survey instrument also assists with determining reliability. That is, if the data collection instrument yields the same results every time it is used, or in other words, if the questions on the survey is answered by respondents the same way each time.

I learnt a valuable lesson that there is no short cut. Test and pilot the data collection instruments. I was fortunate that some of the issues with the questionnaire were captured before the data analysis started. Can you imagine what would have happened if there were missing data on hundreds of questionnaires because respondents were confused or entered 'false' information due to lack of understanding? The time and costs to redo weeks of data collection would have been substantial.

To recap, always test a data collection instrument for grammar, sentence construction, sequencing/logic flow of questions, validity and reliability. Plus, always ensure that the sample that is used to test the data collection is truly representative of the respondents who will be a part of the actual study.

Do you have any 'lessons learnt' to add? Please write them in the 'Comments' section below.