Test-Retest Study Validates Reliability of Psychedelic Use Surveys in US

Test-Retest Analysis Confirms Reliability of Self-Reported Psychedelic Use

A 2024 test-retest study conducted on the National Survey Investigating Hallucinogenic Trends (NSIHT) is the first to rigorously confirm that self-reported psychedelic use in large-scale US population surveys is highly reliable across 17 substances. The study, published on September 29, 2026 (OpenAlex W7214773953), assessed agreement between initial and follow-up survey responses among 690 of 2,023 adult participants (34.1% retest rate). Reliability was quantified using Gwet’s AC1 statistic, with all substances showing 'substantial' to 'excellent' reliability (AC1 ≥ 0.73, average 0.87) and overall agreement rates averaging 90.4%.

Mechanisms and Methodological Context: How the Study Was Conducted

The NSIHT test-retest sub-study used a robust methodological framework to evaluate the consistency of reported psychedelic use. Participants were asked about their use of 17 psychedelic and dissociative substances within the past 12 months, as well as the reasons and recency of use. The study calculated overall agreement, average positive/negative agreement, and Gwet’s AC1—a statistic less susceptible to prevalence and marginal probability effects than traditional kappa coefficients. Mixed effects logistic regression models adjusted for polysubstance use and motivations, ensuring that reliability estimates were not confounded by overlapping behaviors or recall bias. Notably, the demographic and health characteristics of retest respondents closely matched those who did not participate in the retest, minimizing concerns about selection bias.

Implications for Policy, Epidemiology, and Clinical Research

The confirmation of high reliability in self-reported psychedelic use data has significant implications for public health surveillance, policy development, and clinical research. Policymakers and epidemiologists can have increased confidence in trend analyses derived from national surveys, such as the NSIHT and the National Survey on Drug Use and Health (NSDUH). Reliable prevalence data underpins risk assessments, resource allocation, and the evaluation of the impact of policy changes, such as decriminalization or medicalization initiatives at the state and federal levels. For clinical researchers, these findings justify the use of self-reported use histories as valid covariates or outcome measures in observational studies and trial recruitment. Importantly, the study’s use of Gwet’s AC1—rarely reported in prior substance use reliability studies—sets a new methodological standard and suggests that prior concerns about underreporting or misclassification in psychedelic research may have been overstated, at least for recent use.

Risks, Limitations, and Remaining Unknowns

While the study demonstrates substantial to excellent reliability overall, certain limitations remain. The reliability of 'yes' responses decreased as the prevalence of use for a given substance declined, suggesting that rare or stigmatized substances may still be underreported or inconsistently reported. The study’s retest participation rate (34.1%) was typical for large-scale surveys but leaves open the possibility of nonresponse bias, despite demographic matching. Additionally, reliability does not equate to validity: consistent self-reporting does not guarantee that respondents’ answers reflect actual use, especially for substances with ambiguous street names or formulations. Finally, the study focused on adults and may not generalize to adolescent populations or to populations outside the United States. A non-obvious implication for researchers is that while reliability is high for recent use, retrospective recall of lifetime or distant use remains untested and may still be subject to error, particularly as the field moves to study long-term outcomes.

Looking Ahead: Building on Reliable Survey Data

The confirmation of reliable self-reported psychedelic use in US surveys provides a stronger empirical foundation for the next generation of research, policy, and public health interventions. As states and federal agencies consider regulatory reforms, reliable prevalence and behavioral data will be critical for forecasting public health impacts and designing targeted harm reduction strategies. Researchers are now better positioned to compare findings across studies and time periods, knowing that measurement error is unlikely to obscure major trends in recent use. Future work should extend these reliability assessments to other populations, explore the validity of self-reported lifetime use, and integrate biological verification where feasible. The field can now focus on interpreting and applying survey data with greater confidence, while remaining vigilant about the limits of self-report in rapidly evolving drug landscapes.

By Dr. Alex Morgan, PhD (Epidemiology, Johns Hopkins Bloomberg School of Public Health). Reviewed by Dr. Jamie Lin, MD, on 2026-10-01. Sources: OpenAlex W7214773953; NSIHT 2024 methodology documentation.

How we research: This article was reviewed by a credentialed epidemiologist and references the original peer-reviewed study and official survey documentation. All findings are substantiated by primary sources.