What is a pilot study?

Posted on 31st July 2017 by Luiz Cadete

Pilot studies can play a very important role prior to conducting a full-scale research project

Pilot studies are small-scale, preliminary studies which aim to investigate whether crucial components of a main study – usually a randomized controlled trial (RCT) – will be feasible. For example, they may be used in attempt to predict an appropriate sample size for the full-scale project and/or to improve upon various aspects of the study design. Often RCTs require a lot of time and money to be carried out, so it is crucial that the researchers have confidence in the key steps they will take when conducting this type of study to avoid wasting time and resources.

Thus, a pilot study must answer a simple question: “Can the full-scale study be conducted in the way that has been planned or should some component(s) be altered?”

The reporting of pilot studies must be of high quality to allow readers to interpret the results and implications correctly. This blog will highlight some key things for readers to consider when they are appraising a pilot study.

What are the main reasons to conduct a pilot study?

Pilot studies are conducted to evaluate the feasibility of some crucial component(s) of the full-scale study. Typically, these can be divided into 4 main aspects:

• Process: where the feasibility of the key steps in the main study is assessed (e.g. recruitment rate; retention levels and eligibility criteria)
• Resources: assessing problems with time and resources that may occur during the main study (e.g. how much time the main study will take to be completed; whether use of some equipment will be feasible or whether the form(s) of evaluation selected for the main study are as good as possible)
• Management: problems with data management and with the team involved in the study (e.g. whether there were problems with collecting all the data needed for future analysis; whether the collected data are highly variable and whether data from different institutions can be analyzed together).

Reasons for not conducting a pilot study

A study should not simply be labelled a ‘pilot study’ by researchers hoping to justify a small sample size. Pilot studies should always have their objectives linked with feasibility and should inform researchers about the best way to conduct the future, full-scale project.

How to interpret a pilot study

Readers must interpret pilot studies carefully. Below are some key things to consider when assessing a pilot study:

• The objectives of pilot studies must always be linked with feasibility and the crucial component that will be tested must always be stated.
• The method section must present the criteria for success. For example: “the main study will be feasible if the retention rate of the pilot study exceeds 90%”. Sample size may vary in pilot studies (different articles present different sample size calculations) but the pilot study population, from which the sample is formed, must be the same as the main study. However, the participants in the pilot study should not be entered into the full-scale study. This is because participants may change their later behaviour if they had previously been involved in the research.
• The pilot study may or may not be a randomized trial (depending on the nature of the study). If the researchers do randomize the sample in the pilot study, it is important that the process for randomization is kept the same in the full-scale project. If the authors decide to test the randomization feasibility through a pilot study, different kinds of randomization procedures could be used.
• As well as the method section, the results of the pilot studies should be read carefully. Although pilot studies often present results related to the effectiveness of the interventions, these results should be interpreted as “potential effectiveness”. The focus in the results of pilot studies should always be on feasibility, rather than statistical significance. However, results of the pilot studies should nonetheless be provided with measures of variability (such as confidence intervals), particularly as the sample size of these studies is usually relatively small, and this might produce biased results.

After an interpretation of results, pilot studies should conclude with one of the following:

(1) the main study is not feasible;

(2) the main study is feasible, with changes to the protocol;

(3) the main study is feasible without changes to the protocol OR

(4) the main study is feasible with close monitoring.

Any recommended changes to the protocol should be clearly outlined.

Take home message

• A pilot study must provide information about whether a full-scale study is feasible and list any recommended amendments to the design of the future study.

References

Thabane L, Ma J, Chu R, et al. A tutorial on pilot studies: what, why and how? BMC Med Res Methodol. 2010; 10: 1.

Cocks K and Torgerson DJ. Sample Size Calculations for Randomized Pilot Trials: A Confidence Interval approach. Journal of Clinical Epidemiology. 2013.

Lancaster GA, Dodd S, Williamson PR. Design and analysis of pilot studies: recommendations for good practice. J Eval Clin Pract. 2004; 10 (2): 307-12.

Moore et al. Recommendations for Planning Pilot Studies in Clinical and Translational Research. Clin Transl Sci. 2011 October ; 4(5): 332–337.

Tags: