Levels of evidence: a vignette-study


Assume that we want to plan a study to answer the following question:

For a common metastatic cancer X, treatment A is the usual first-line treatment. Treatment B is also prescribed in routine practice but has never been compared to treatment A. We want to compare treatment A to treatment B.

The difference in treatment effect is expected to be moderate. Pharmaceutical companies are rarely interested in answering such therapeutic strategy questions, so the study would presumably be academic.

The ideal scenario for answering this question with high level evidence would be fast access to results from a large randomized controlled trial (i.e., the gold standard of causality) that would be funded and achieve planned recruitment. We know from experience that this is unlikely. Comparative observational studies can provide faster results but are subject to biases. However, new methods such as trial emulation* can reduce the risk of bias.

What is expected of you in this survey?

To answer this survey, you will take on the role of a study investigator. If only one study could be planned to provide evidence for the previous question, which study would you support?

To do so, we propose 6 different studies compared in a pairwise manner. You will have 5 comparisons to assess and you will need to choose one study each time.

This survey will take you about 10 minutes.

*Target trial emulation is a new method which uses observational data to mimic the design principles of a hypothetical randomized trial (i.e. the target trial). It requires adjustment for confounding factors (e.g. via matching).
Hernán MA, Robins JM. Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available. Am J Epidemiol. 2016 Apr 15;183(8):758-64. doi: 10.1093/aje/kwv254. Epub 2016 Mar 18. PMID: 26994063; PMCID: PMC4832051.