1.  Can you discuss some of the methods and issues relating to “real world” studies?

“Real-world” studies are designed to capture the effectiveness of health care interventions in community, routine, or general settings of health care. Such studies are intended to provide findings that are more likely to be applicable to health care decisions in real-world settings. A challenge to such studies is that, in contrast to carefully managed “efficacy” studies such as Randomized Controlled Trials (RCTs) with narrowly defined patient populations and specifically defined interventions, it is more difficult to isolate the causal impact of an intervention on health outcomes, and the same diversity of factors (e.g., variation among patients, application of treatments) that enriches such a study may confound the findings of the study. As such, “real-world studies” should be used to complement, though not necessarily substitute for, the more carefully managed efficacy studies. Real-world studies typically allow for more heterogeneous patient populations (e.g., with various comorbidities), different settings of care, and some variation in how interventions are delivered. Clinical trials intended to represent real-world experience are sometimes known as “practical clinical trials.” 

2.  Where does CAM fit into CER?

Complementary and alternative medicine (CAM) refers to nonstandard treatments that may be used along with standard care; examples include herbal medicines, acupuncture, yoga, chiropractic, and various dietary supplements. Many types of CAM have been studied in randomized clinical trials (RCTs) and other types of investigations. Similarly, CAM can be the subject of CER in the same manner as standard care. In its 2009 report on national priorities for CER, the Institute of Medicine noted the wide use of CAM and listed three broad priorities for CER of CAM. For example, one of these was: “Compare the effectiveness of mindfulness-based interventions (e.g., yoga, meditation, deep breathing training) and usual care in treating anxiety and depression, pain, cardiovascular risk factors, and chronic diseases.” 

3. How can Randomized Controlled Trials (RCTs) provide information to support personalized medicine?

For an RCT or other study to support personalized medicine, it must be designed to produce evidence pertaining to the differential responses to health care interventions (i.e., the heterogeneity of treatment effects) that are associated with particular patient attributes or traits, whether genomic, demographic, behavioral, or other. RCTs with prospectively identified patient subgroups can test hypotheses about relationships between those traits and patient outcomes. The findings of retrospectively identified associations between patient subgroups and patient outcomes generally are not as statistically robust as findings generated with prospectively identified subgroups, although they are useful for generating new hypotheses about such relationships. In either instance, however, the more subgroups that are identified for potential association with health outcomes, the more likely it is that any particular observed association is due to chance alone rather than to a true causal relationship. As such, the number of subgroups identified in any prospective trial or retrospective analysis should be limited. Certain “adaptive” clinical trials and other variations on clinical trials that enable modifications of patient assignment to treatment groups as information is compiled about patient subgroup responsiveness to interventions can improve the ability or efficiency of RCTs to detect important subgroup differences. RCTs also can be used to demonstrate that patients with a particular trait, whether genomic, demographic, behavioral, or other, are more or less likely to respond well or experience an adverse event attributable to a particular intervention. As the body of evidence develops for how particular patient subgroups respond to various interventions for a given disease or disorder, clinicians and patients will be able to “personalize” or “tailor” treatment decisions by drawing on evidence for how similar patients have responded to alternative treatments. Certainly, patient preferences should be considered as well. 

4. In one of the slides about heterogeneity of treatment effects, there are three sample distributions superimposed on the main population distribution. Can you say more about how those different samples would arise?

Slide #29 (from Kravitz et al. 2004) shows a broad population distribution representing the true response to a given therapy of the universe of the potential target patient population. Against that backdrop, three hypothetical sample distributions are shown, all of which could be drawn from the universe of the target patient population. As stated in the caption, Sample 1 is centered but fails to reflect the diversity of the population in terms of net treatment benefit. This could arise, for example, in an RCT in which the patient selection (inclusion and exclusion) criteria are narrowly defined, eliminating those patients that experience especially favorable or unfavorable (including harmful) treatment effects, though not being biased relative to the true average treatment effect. Sample 2 is composed of individuals who happen to derive much more net benefit from the treatment than does the average member of the population. This, too, could arise from an RCT in which inclusion and exclusion criteria yield a narrowly defined study population, though in this instance, that selected population is more likely to have a favorable treatment response, perhaps due to their particular set of risk factors (e.g., age, co-morbidities) resulting from the selection criteria. This treatment effect, biased relative to the true average treatment effect, could also appear in a single-arm clinical trial relying on a historical control group that was somehow inherently different from the group selected for the present study, for example, with a set of risk factors that resulted in a less favorable average treatment response. Sample 3 is broadly representative of the population in terms of risk, responsiveness, and vulnerability. This could arise from a randomized practical clinical trial that did not have restrictive inclusion criteria, resulting in a study population that was more representative of the universe of the potential target population, and not biased relative to the true average treatment effect.  

Last Reviewed: February 23, 2024