In this section we provide some resources on methods and guidelines for conducting feasibility and acceptability pilot studies, arguments against adjusting p values for multiple testing in RCTs, and power analyses for PIs.
Conducting feasibility and acceptability pilot studies
We provide several methodological resources for conducting feasibility studies. These include links to newly-published guidelines for conducting feasibility studies, resources on the role of pilot RCTs to assess feasibility and acceptability of full scale RCTs, and suggested language for pilot RCT study proposals.
Published guidelines for conducting feasibility and acceptability pilot studies: Recommends indicators for assessing the feasibility of assessments and data collection and intervention implementation, and presents advanced methods for estimating group differences.
Teresi JA, Yu X, Stewart AL, Hays RD. Guidelines for Designing and Evaluating Feasibility Pilot Studies. Med Care. 2022 Jan 1;60(1):95-103. PMCID: PMC8849521 PDF SUPPL MATERIAL
Published guidelines for evaluating feasibility of recruitment: Provides an organizing framework (8 steps) for assessing the feasibility of recruitment, including practical steps on how to design, track, and report measures of recruitment feasibility.
Stewart AL, Nápoles AM, Piawah S, Santoyo-Olsson J, Teresi JA. Guidelines for Evaluating the Feasibility of Recruitment in Pilot Studies of Diverse Populations: An Overlooked but Important Component. Ethn Dis. 2020 Nov 19;30(Suppl 2):745-754. PMCID: PMC7683033. PDF
Purpose and design of pilot RCTs: Concepts and strategies: Compares traditional with more recent perspectives of the goals of pilot RCTs, including the rationale for the new recommendation that they not be used to estimate effect sizes. Recommends instead to examine the feasibility and acceptability of methods for a larger trial (2018-19). SLIDE SET Video of Presentation
The role of pilot feasibility & acceptability studies in randomized controlled trials: This document provides three types of resources on this topic: web resources, proposal boilerplate language, and references for selected topics (2022). PDF
Argument against adjusting p values for multiple testing in RCTs
For RCTs that are testing the effect of clinical interventions on a corresponding set of outcomes, related policy decisions are often best served by tests that do not adjust alpha for multiple tests. We provide a slide set summarizing the rationale for rejecting adjustment for multiple testing, and a document with suggested language for responding to reviewers who request adjustment for multiple testing.
Rejecting Universal Adjustment for Multiple Testing in Public Health RCTs of Clinical/Behavioral Interventions: This slide set describes the rationale for recommending against adjusting alpha for multiple testing as well as describing a limited set of conditions under which alpha adjustments for multiple comparisons are recommended (2018).
Suggested language to respond to critiques asking for alpha level adjustments for multiple testing: Document includes examples of responses to reviewers and corresponding manuscript text to counter reviewer comments that alpha adjustments for multiple comparisons are required (2022). PDF
Control group design
Three types of control groups are explored with respect to RCTs examining the efficacy or effectiveness of an intervention: attention placebo control, time and attention control, and usual care (2018). SLIDE SET
Power analysis
Power analysis is a key aspect of quantitative research design. We provide two resources.
Power Analysis for PIs: This presentation includes a summary of basic types of input needed by statisticians to conduct power analysis (2019).
SLIDE SET
Power Analysis for Clustered Binary Data: This presentation covers different types of clustering and variance inflation and corresponding design effects arising from such clustering and how to incorporate those factors into making rigorous power calculations for logistic regression analyses of clustered data (2014). SLIDE SET