No Short Cuts! Test and Pilot Data Collection Instruments
- Ann-Murray Brown
- Aug 24, 2018
- 3 min read

This blog post is written from my own unfortunate experience of not testing and piloting a data collection instrument properly. My failure to do so not only affected the quality of the data (as many persons did not answer the ambiguous questions on the survey) but also led to delays when some of the data had to be collected again.
How did this happen in the first place?
Well, in an attempt to reduce costs and to get things ‘moving as quickly as possible’, the client insisted that a questionnaire that was used in a previous baseline study, be adopted for this follow up study. Notwithstanding the fact that the questionnaire was fraught with misspellings, ambiguous sentences, incoherent flow and sequence of questions as well as culturally insensitive questions that probed religious affiliation and sexual identity.
As some of these issues were readily apparent, I did manage to negotiate a compromise with the client. The ‘solution’ we agreed on was to test the questionnaire with a sample of 6 persons. This would be quick, dirty and most of all, low cost (music to my client’s ears).
Afterwards I realized that this sample was not truly representative of the target population who would be the actual participants of the study. More specifically, the persons who tested the questionnaires were all residents of an institution and were mostly men. It therefore meant that their perspectives may have differed from other participants of the actual study that included women and who have never been institutionalized.
However, before we proceed further, what do we actually mean by ‘testing a data collection instrument’?
Testing a data collection instrument such as a questionnaire for a survey, involves using that questionnaire before the actual study or evaluation in conditions that are very similar to the official administering of your survey.
This test helps to eliminate certain faults on your questionnaire that I mentioned earlier. These include spelling and grammar issues, poorly constructed sentences, questions that are ambiguous/lack clarity and question that are not culturally sensitive, lengthy and redundant.
Remember, a data collection instrument may seem “perfect” from your point of view. The only way to be sure is to test it.

Apart from spotting grammatical issues, testing a data collection instrument helps you to determine its validity. In other words, during the testing phase you will get a chance to see if the data collection instrument actually measures what it is intended to measure.
In my case, it was only after the initial rounds of actual data collection did it become clear that a few questions that were meant to be measuring religiosity (e.g. ‘When was the last time you of attended a place of worship?’) were in fact measuring access to places of worship and not necessarily how 'devout' one was.
A focus group discussion later revealed that persons did not think frequent attendance to a place of worship meant a greater connection to a higher power. They felt just as religious and devout staying right at home. The frequency of their attendance to a place of worship was more dependent on how easy it was to get to the location from home. This is why mixed-methods and triangulation of data (in this case from the survey and focus group) is so important.
Testing the survey instrument also assists with determining reliability. That is, if the data collection instrument yields the same results every time it is used, or in other words, if the questions on the survey is answered by respondents the same way each time.
I learnt a valuable lesson that there is no short cut. Test and pilot the data collection instruments. I was fortunate that some of the issues with the questionnaire were captured before the data analysis started. Can you imagine what would have happened if there were missing data on hundreds of questionnaires because respondents were confused or entered 'false' information due to lack of understanding? The time and costs to redo weeks of data collection would have been substantial.
To recap, always test a data collection instrument for grammar, sentence construction, sequencing/logic flow of questions, validity and reliability. Plus, always ensure that the sample that is used to test the data collection is truly representative of the respondents who will be a part of the actual study.
Do you have any 'lessons learnt' to add? Please write them in the 'Comments' section below.
google 优化 seo技术+jingcheng-seo.com+秒收录;
Fortune Tiger Fortune Tiger;
Fortune Tiger Fortune Tiger;
Fortune Tiger Fortune Tiger;
Fortune Tiger Slots Fortune…
站群/ 站群
gamesimes gamesimes;
03topgame 03topgame
EPS Machine EPS Cutting…
EPS Machine EPS and…
EPP Machine EPP Shape…
Fortune Tiger Fortune Tiger;
EPS Machine EPS and…
betwin betwin;
777 777;
slots slots;
Fortune Tiger Fortune Tiger;
무료카지노 무료카지노;
무료카지노 무료카지노;
google 优化 seo技术+jingcheng-seo.com+秒收录;
Fortune Tiger Fortune Tiger;
Fortune Tiger Fortune Tiger;
Fortune Tiger Slots Fortune…
站群/ 站群
gamesimes gamesimes;
03topgame 03topgame
EPS Machine EPS Cutting…
EPS Machine EPS and…
EPP Machine EPP Shape…
Fortune Tiger Fortune Tiger;
EPS Machine EPS and…
betwin betwin;
777 777;
slots slots;
Fortune Tiger Fortune Tiger;
EPS Machine EPS Cutting…
EPS Machine Eps Raw…
EPS Machine EPS Recycling…
EPS Machine EPS Mould;
EPS Machine EPS Block…
EPP Machine EPP Shape…
EPTU Machine ETPU Moulding…
EPS Machine Aging Silo…
EPTU Machine ETPU Moulding…
EPS Machine EPS and…
EPS Machine EPS and…
AEON MINING AEON MINING
AEON MINING AEON MINING
KSD Miner KSD Miner
KSD Miner KSD Miner
BCH Miner BCH Miner
BCH Miner BCH Miner