We recently conducted a pilot study for a national program involving 12 brands. We worked with one of those brands for our pilot phase, to determine the data collection and measurement strategy. Although it went relatively smoothly, it was the reporting portion of that phase that required the most work. Our initial approach failed to yield results meaningful to the purpose of the program, so we had to redefine success in that context.
A pilot study is a test run, designed to determine the feasibility of a similar study to be conducted on a larger scale. More specifically it is defined as:
“A small scale, preliminary study conducted in order to evaluate feasibility, time, cost, adverse events, and effect size (statistical variability) in an attempt to predict an appropriate sample size and improve upon the study design prior to performance of a full-scale research project.” (Hulley, Stephen B. Designing Clinical Research. Lippincott Williams & Wilkins, 2007, p.168-169.)
That seems straight-forward enough. But, in business, just mentioning the phrase “pilot study” can evoke all sorts of reactions: excitement, relief, consternation, and anxiety. So, why does something so innately objective evoke so many objective, emotional responses?
Why pilot studies evoke so much emotion
In business, a pilot study is inextricably linked to the idea of showing what worked. That means it also invariably exposes what did not work.
It’s important to keep in mind that, in a pilot program, “not successful” is not synonymous with “failure.”
By its very definition, a pilot study is meant to permit refinement and adjustment, so that the full phase of research will generated controlled and accurate results.
The advantages of conducting a pilot study
Ultimately, the benefits a pilot study offers far outweigh the risks of (perceived) failure. A pilot test affords the opportunity for:
- Buy-in from key stakeholders
- Identification of strengths or flaws in the research design
- Adjustment of the budget as needed
- Modification of expectations before the actual project begins
When it came time to present the pilot recap report to our clients and stakeholders, we reported what we learned from the pilot phase and how those facts should be applied to the roll-out.
By acknowledging the things that didn’t work as well as those that did, we were able to explain the need for changes in approach and offer a transparent look in to the research methodology.
Not only was the presentation of this report well received, the roll-out was a great success.
Photo Source: http://farm6.staticflickr.com/5122/5221699083_3c6284964c_n.jpg