Improving identification of determinants of evidence-based practice implementation
With diversity supplement funding, Measurement Study Lead Jones set out to develop item banks and short forms for measuring intervention characteristics. Assessing implementation determinants (i.e., barriers and facilitators) before and during implementation is widely regarded critical for successfully integrating evidence-based interventions in cancer control into routine practice. Yet, available measures of implementation determinants lack desirable psychometric and pragmatic qualities, limiting their use and usefulness in both research and practice.
This study compiled a bank of items used to measure intervention characteristics in previous studies, assessed the content validity of the items in the item bank, and assessed the reliability, concurrent validity, and predictive validity of measures constructed from items demonstrating good content validity.
Implementation Science Knowledge Gained
Measures of 11 intervention characteristics that exhibit reliability, concurrent validity, and predictive validity that can be used in implementation research.
Research Team
The research team compiled items of determinants used in previous studies into an item bank. Those items were added to a spreadsheet and coded by which CFIR determinant they represented. After reviewing the items coded in a general “Other” category, three non-CFIR determinants were added: observability, risk, and burden. The research team sought copyright approval for items and eliminated 100 items for which approval could not be obtained. They then assessed the content validity of the item bank using a concurrent mixed-methods approach. First, they recruited 30 implementation scientists to complete an online survey and provide feedback on the 271 items in the bank. Study participants rated each item’s relevance using a 4-point Likert scale. The research team calculated the content validity index and retained items that 78% of participants rated as “quite relevant” (=3) or “highly relevant” (=4). They also obtained qualitative feedback from study participants on missing items, missing concepts, and item wording. Three experts then reviewed the items and provided feedback on item redundancy, representativeness, and clarity. The research team eliminated or reworded items based on the experts’ feedback.
Based on the content validity index, 179 items were eliminated, resulting in 92 items. However, based on suggestions from study participants, an additional 46 items were drafted and included in the item bank. Seven items were added after the content validity survey was completed due to a late-arriving copyright approval. Many retained items were revised based on study participants’ feedback. The item bank included 145 items across 11 intervention characteristics. Three clinical and implementation experts reviewed the 145 items and based on their feedback the number of items was reduced to 109.
In a subsequent study, the research team developed measures of 11 intervention characteristics using the 109 items from the content validity study. They assessed the reliability and validity of the measures through two surveys fielded to 175 healthcare personnel. All eleven measures exhibited good reliability, concurrent validity, and predictive validity.
- Jones S, Aditya S, Weiner B. Content validity of an item bank to assess intervention characteristic determinants of implementing evidence-based practices. Implement Res Pract 2023 Jan-Dec;4.
- Jones SMW, Weiner BJ. Reliability and validity of intervention characteristic measures for assessing barriers to evidence-based practice use. Transl Behav Med. 2024 Apr 29;14(5):304-309. doi: 10.1093/tbm/ibae013. PMID: 38518799; PMCID: PMC11056881.