Research Type
Program/Policy Level
Collaborators
Roudebush VA Medical Center
Health Services Research & Development
Center of Excellence on Implementing Evidence-Based Practice
- Co-Principal Investigator, Angela L. Rollins, PhD
- Co-Principal Investigator, Michelle P. Salyers, PhD
- Alan McGuire, PhD, Co-Investigator
- Linda Collins, BS, Research Assistant
Collaborating faculty
- John McGrew, PhD, Co-Investigator
- Doug Leslie, PhD, Co-Investigator
- Sandy Resnick, PhD, Co-Investigator
- Lia Hicks, MBA, Fidelity Assessor
- Jennifer Wright, MSW, Fidelity Assessor
Background
National policy has dramatically increased the emphasis on implementing evidence-based mental health services to meet the needs of people with severe mental illness, and the VHA has made great strides at providing effective, community-based services. One of the cornerstones of the VHA approach is Mental Health Intensive Case Management (MHICM), a model that is based on one of the most well-defined and empirically supported approaches: assertive community treatment. More recently, VHA policy shifts have resulted in a proposed set of uniform mental health services to ensure access to a standard set of high quality mental health services, such as MHICM, across the entire VHA. However, successful implementation of evidence-based practices on a broad scale requires psychometrically valid, yet practical, ways to assess and monitor degree of implementation (i.e., fidelity). Currently, the only rigorous method to monitor implementation is an on-site fidelity visit, which is a very time-intensive, expensive, and burdensome approach for both the assessor and the program.
The research questions
The primary objective of the proposed study was to examine the effectiveness of innovative and potentially cost-effective methods to ensure the quality of mental health services for disabled veterans with mental illness. Specifically, we examined the reliability, concurrent validity, and predictive validity of three methods of fidelity assessment: self-report, phone-based remote assessment, and a “gold-standard” on-site fidelity assessment. We also explored costs associated with each method.
Systems-Research tools
This study examined the accuracy of three approaches to fidelity measurement: self-report (the approach currently used in the national evaluation of MHICM programs), a new phone-based fidelity assessment, and the gold standard on-site fidelity assessment using the well-validated DACTS (Dartmouth Assertive Community Treatment Scale).
Project overview
All MHICM teams (n=111) provided self-reported fidelity annually to the Northeast Program Evaluation Center (NEPEC). VHA tracked hospitalization outcomes over time. We aimed to recruit 32 teams to participate in a phone-based assessment and an on-site fidelity visit with experienced fidelity assessors. We stratified teams on type of VHA facility and previous year’s performance on the self-assessment. The order of phone and on-site assessments were counter-balanced, with separate assessors, to reduce potential bias. We examined level of agreement between fidelity approaches with intraclass-correlations. We examined predictive validity using regression modeling to examine the association between fidelity method and improved hospitalization outcomes. For our cost identification analysis, we compared costs across the three methods of assessment, using a mixed (repeated measures) model approach. We also included a formative evaluation to inform future dissemination of fidelity assessment methods in the VHA and elsewhere.
Impact
The proposed study addressed a critical policy issue for VHA and other large health systems: identifying the best ways to monitor program implementation. The current self-report method for monitoring MHICM programs is of unknown validity. Establishment of valid yet practical ways to monitor implementation is critical to continue to ensure high-quality care as proposed in the VHA’s strategic mental health plan and the uniform services plan.
