Challenges in EHR-Based Risk Models for TRD Across Health Systems

Transportability Issues in EHR-Based TRD Risk Models

Electronic health record (EHR)-based models for predicting treatment-resistant depression (TRD) face significant transportability challenges across different health systems. A recent study involving Mass General-Brigham, Vanderbilt University Medical Center, and Geisinger Clinic from 2004 to 2022 revealed poor performance when these models were applied to new settings. This underscores the complexity of TRD as an outcome and the limitations of current data sources.

Mechanisms Behind Model Failures

The study identified three primary mechanisms for the failure of these models to transport: the heterogeneity of TRD as an outcome, biases in source data, and site-/practice-level variation. TRD was defined by the provision of advanced treatments or the failure of more than two antidepressants. L1-regularized regression applied to sociodemographic features, medications, and ICD-10 diagnostic codes showed poor performance, with C-statistics ranging from 0.51 to 0.65 internally and 0.5 to 0.58 externally.

Policy and Research Implications

Addressing the intrinsic heterogeneity of clinical populations and biases in source data is crucial for improving risk stratification models. The study suggests that incorporating data beyond EHRs may be necessary to achieve clinically meaningful predictions. Policymakers and researchers must consider these factors when developing and implementing TRD risk models.

Risks and Unknowns

The limitations of current EHR-based models highlight the risks of relying solely on these data sources for clinical decision-making. The variability in model performance across different sites suggests that local practice patterns and patient demographics significantly impact model accuracy. This variability poses a challenge for standardizing TRD risk prediction across diverse health systems.

Future Directions in TRD Risk Modeling

Future research should focus on integrating diverse data sources, such as genetic information and patient-reported outcomes, to enhance the accuracy and transportability of TRD risk models. Collaborative efforts across health systems could help standardize data collection and model development, potentially leading to more robust and clinically useful predictions.