Method of using RE-AIM framework to develop implementation outcome indicators_Chinese Journal of Evidence-Based Medicine

Authors：

LV Sensen ¹ , CAI Yiyuan ² , HE Wenjun ¹ , CAI Yefeng ³ , NI Xiaojia ³ , LUO Huanyuan ¹ , ZHONG Dongmei ¹ , LIU Siyuan ¹ , ZHAO Qing ¹ , ZHANG Lanping ¹ , LI Jiaqi ¹ , CHEN Xiaoshan ¹ , XIE Yunyun ¹ , LI Chunping ¹ , HUANG Wanqing ¹ ,  XU Dong ^1,4,5

1. Acacia Lab for Implementation Science, School of Health Management, Southern Medical University, Guangzhou 510515, P. R. China;
2. School of Public Health, Guizhou Medical University, Guiyang 550000, P. R. China;
3. Provincial Hospital of Chinese Medicine, Guangzhou 510120, P. R. China;
4. Center for World Health Organization Studies and Department of Health Management, School of Health Management of Southern Medical University, Guangzhou 510515, P. R. China;
5. Institute for Global Health, Dermatology Hospital of Southern Medical University, Guangzhou 510515, P. R. China;

Corresponding?author：

XU Dong, Email: romanxu@i.smu.edu.cn

Keywords：

Implementation science; Implementation research; Evidence-based practice; RE-AIM framework; Guideline

DOI：

10.7507/1672-2531.202211159

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

The RE-AIM framework is the most commonly used framework for implementation research outcome evaluation, with the original paper cited 2 800 times. However, in the specific operationalization process of the framework, there are often situations of misuse or overuse of various dimensions and indicators. This article provides a detailed introduction to the main content, application scenarios, and characteristics of the RE-AIM framework. Taking the implementation research of Evidence-based Practice Guidelines for Integrated Traditional Chinese and Western Medicine Treatment of Stroke as an example, it introduces how to use the silent & anonymous feedback technology, which combines the improved Delphi method and nominal group technique, to construct an implementation outcome indicator evaluation system to achieve the operability of the RE-AIM framework. This will provide a reference for other researchers to use implementation science theories, models, and frameworks to develop implementation plans or evaluate implementation outcomes.

Citation： LV Sensen, CAI Yiyuan, HE Wenjun, CAI Yefeng, NI Xiaojia, LUO Huanyuan, ZHONG Dongmei, LIU Siyuan, ZHAO Qing, ZHANG Lanping, LI Jiaqi, CHEN Xiaoshan, XIE Yunyun, LI Chunping, HUANG Wanqing, XU Dong. Method of using RE-AIM framework to develop implementation outcome indicators. Chinese Journal of Evidence-Based Medicine, 2023, 23(6): 695-701. doi: 10.7507/1672-2531.202211159 Copy

1.	Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: understanding time lags in translational research. J R Soc Med, 2011, 104(12): 510-520.
2.	Balas EA, Boren SA. Managing clinical knowledge for health care improvement. Yearb Med Inform, 2000, 9(1): 65-70.
3.	陳文嘉, 徐東, 李慧, 等. 實施科學理論的分類與介紹. 中國循證醫學雜志, 2020, 20(8): 986-992.
4.	屈智勇, 郭帥, 張維軍, 等. 實施科學對我國心理健康服務體系建設的啟示. 北京師范大學學報(社會科學版), 2017, (2): 29-36.
5.	Nutley S, Walter I, Davies HTO. From knowing to doing: a framework for understanding the evidence-into-practice agenda. Evaluation, 2003, 9(2): 125-148.
6.	Damschroder LJ. Clarity out of chaos: use of theory in implementation research. Psychiatry Res, 2020, 283: 112461.
7.	Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci, 2015, 10: 53.
8.	Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health, 1999, 89(9): 1322-1327.
9.	Proctor EK, Landsverk J, Aarons G, et al. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health, 2009, 36(1): 24-34.
10.	Green LW, Kreuter MW. Health promotion planning: an educational and ecological approach (3rd edition). Mountain View, CA: Mayfield Publishing, 1999.
11.	Glasgow RE, Harden SM, Gaglio B, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health, 2019, 7: 64.
12.	Tabak RG, Khoong EC, Chambers DA, et al. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med, 2012, 43(3): 337-350.
13.	Stange KC. Commentary: RE-AIM Planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health, 2020, 8: 245.
14.	King DK, Glasgow RE, Leeman-Castillo B. Reaiming RE-AIM: using the model to plan, implement, and evaluate the effects of environmental change approaches to enhancing population health. Am J Public Health, 2010, 100(11): 2076-2084.
15.	中華中醫藥學會腦病分會, 廣東省中醫藥學會腦病專業委員會, 廣東省中西醫結合學會卒中專業委員會. 中西醫結合腦卒中循證實踐指南(2019). 中國循證醫學雜志, 2020, 20(8): 901-912.
16.	Glasgow RE, Estabrooks PA, Ory MG. Characterizing evolving frameworks: issues from Esmail et al. (2020) review. Implement Sci, 2020, 15(1): 53.
17.	Bellg AJ, Borrelli B, Resnick B, et al. Enhancing treatment fidelity in health behavior change studies: best practices and recommendations from the NIH behavior change consortium. Health Psychol, 2004, 23(5): 443-451.
18.	婁曉民. RE-AIM模型. 余金明, 姜慶五, 主編. 現代健康教育學. 上海: 復旦大學出版社, 2019: 120-123.
19.	Holtrop JS, Estabrooks PA, Gaglio B, et al. Understanding and applying the RE-AIM framework: clarifications and resources. J Clin Transl Sci, 2021, 5(1): e126.
20.	田文華. 基于RE-AIM模型的城鄉居民大病保險模式評估. 復旦學報(社會科學版), 2021, 63(1): 152-160.
21.	Gaglio B, Shoup JA, Glasgow RE. The RE-AIM framework: a systematic review of use over time. Am J Public Health, 2013, 103(6): e38-e46.
22.	Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf, 2008, 34(4): 228-243.
23.	Liles EG, Schneider JL, Feldstein AC, et al. Implementation challenges and successes of a population-based colorectal cancer screening program: a qualitative study of stakeholder perspectives. Implement Sci, 2015, 10: 41.
24.	King DK, Shoup JA, Raebel MA, et al. Planning for implementation success using RE-AIM and CFIR frameworks: a qualitative study. Front Public Health, 2020, 8: 59.
25.	魯春麗, 曹卉娟, 徐東, 等. 實施科學產生的背景、概念和國內外發展近況. 中國中西醫結合雜志, 2020, 40(11): 1378-1380.
26.	Klesges LM, Estabrooks PA, Dzewaltowski DA, et al. Beginning with the application in mind: designing and planning health behavior change interventions to enhance dissemination. Ann Behav Med, 2005, 29 Suppl: 66-75.
27.	Kessler RS, Purcell EP, Glasgow RE, et al. What does it mean to "employ" the RE-AIM model. Eval Health Prof, 2013, 36(1): 44-66.
28.	Galaviz KI, Harden SM, Smith E, et al. Physical activity promotion in Latin American populations: a systematic review on issues of internal and external validity. Int J Behav Nutr Phys Act, 2014, 11: 77.
29.	Hoehner CM, Ribeiro IC, Parra DC, et al. Physical activity interventions in Latin America: expanding and classifying the evidence. Am J Prev Med, 2013, 44(3): e31-e40.
30.	Glasgow RE, Estabrooks PE. Pragmatic applications of RE-AIM for health care initiatives in community and clinical settings. Prev Chronic Dis, 2018, 15: E02.
31.	王擁軍, 李子孝, 谷鴻秋, 等. 中國卒中報告2019(中文版)(1). 中國卒中雜志, 2020, 15(10): 1037-1043.
32.	何莎, 張田, 孫偉, 等. 中國內地腦卒中治療多中心隨機對照試驗方法學質量現狀報告. 華西醫學, 2020, 35(6): 673-678.
33.	Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet, 1993, 342(8883): 1317-1322.
34.	Pronovost PJ. Enhancing physicians' use of clinical guidelines. JAMA, 2013, 310(23): 2501-2502.
35.	Liu M, Zhang C, Zha Q, et al. A national survey of Chinese medicine doctors and clinical practice guidelines in China. BMC Complement Altern Med, 2017, 17(1): 451.
36.	Cabana MD, Rand CS, Powe NR, et al. Why don't physicians follow clinical practice guidelines. A framework for improvement. JAMA, 1999, 282(15): 1458-1465.
37.	Working Group Members for Stroke. International consortium for health outcomes measurement (ICHOM). Stroke data collection reference guide. 2018.
38.	RANKIN J. Cerebral vascular accidents in patients over the age of 60. Ⅱ. Prognosis. Scott Med J, 1957, 2(5): 200-215.
39.	Quinn TJ, Langhorne P, Stott DJ. Barthel index for stroke trials: development, properties, and application. Stroke, 2011, 42(4): 1146-1151.
40.	Goldstein LB, Bertels C, Davis JN. Interrater reliability of the NIH stroke scale. Arch Neurol, 1989, 46(6): 660-662.
41.	Vernon W. The Delphi technique: a review. Int J Ther Rehabil, 2013, 16(2): 69-76.
42.	Mullen R, Kydd A, Fleming A, et al. A practical guide to the systematic application of nominal group technique. Nurse Res, 2021, 29(1): 14-20.
43.	柯惠新, 黃京華, 沈浩. 調查研究中的統計分析法. 北京: 北京廣播學院出版社, 1996: 351.
44.	黃敬亨. 健康教育學. 上海: 上海醫科大學出版社, 1997: 212.
45.	柯惠新, 沈浩. 調查研究中的統計分析方法. 北京: 北京傳媒大學出版社, 2005: 324.
46.	席仲恩, 杜玨 (譯). 量表編制: 理論與應用(第三版). 重慶: 重慶大學出版社, 2016.
47.	Schmidt RC. Managing Delphi surveys using nonparametric statistical techniques. Decision Sci, 1997, 28(3): 763-774.
48.	楊宗義. 教育統計學(第1版). 北京: 科學技術文獻出版社, 1990.
49.	魯曼, 劉虹, 鄭志杰. 醫學生全球勝任力指標體系構建的研究. 中華醫學教育雜志, 2021, 41(12): 1057-1061.
50.	Shen L, Yang J, Jin X, et al. Based on Delphi method and analytic hierarchy process to construct the evaluation index system of nursing simulation teaching quality. Nurse Educ Today, 2019, 79: 67-73.
51.	張天芳, 陳春芳. 關于Kendall協和系數的理解. 統計學與應用, 2020, 9(4): 578-581.
52.	Reilly KL, Kennedy S, Porter G, et al. Comparing, contrasting, and integrating dissemination and implementation outcomes included in the RE-AIM and implementation outcomes frameworks. Front Public Health, 2020, 8: 430.

1. Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: understanding time lags in translational research. J R Soc Med, 2011, 104(12): 510-520.
2. Balas EA, Boren SA. Managing clinical knowledge for health care improvement. Yearb Med Inform, 2000, 9(1): 65-70.
3. 陳文嘉, 徐東, 李慧, 等. 實施科學理論的分類與介紹. 中國循證醫學雜志, 2020, 20(8): 986-992.
4. 屈智勇, 郭帥, 張維軍, 等. 實施科學對我國心理健康服務體系建設的啟示. 北京師范大學學報(社會科學版), 2017, (2): 29-36.
5. Nutley S, Walter I, Davies HTO. From knowing to doing: a framework for understanding the evidence-into-practice agenda. Evaluation, 2003, 9(2): 125-148.
6. Damschroder LJ. Clarity out of chaos: use of theory in implementation research. Psychiatry Res, 2020, 283: 112461.
7. Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci, 2015, 10: 53.
8. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health, 1999, 89(9): 1322-1327.
9. Proctor EK, Landsverk J, Aarons G, et al. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health, 2009, 36(1): 24-34.
10. Green LW, Kreuter MW. Health promotion planning: an educational and ecological approach (3rd edition). Mountain View, CA: Mayfield Publishing, 1999.
11. Glasgow RE, Harden SM, Gaglio B, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health, 2019, 7: 64.
12. Tabak RG, Khoong EC, Chambers DA, et al. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med, 2012, 43(3): 337-350.
13. Stange KC. Commentary: RE-AIM Planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health, 2020, 8: 245.
14. King DK, Glasgow RE, Leeman-Castillo B. Reaiming RE-AIM: using the model to plan, implement, and evaluate the effects of environmental change approaches to enhancing population health. Am J Public Health, 2010, 100(11): 2076-2084.
15. 中華中醫藥學會腦病分會, 廣東省中醫藥學會腦病專業委員會, 廣東省中西醫結合學會卒中專業委員會. 中西醫結合腦卒中循證實踐指南(2019). 中國循證醫學雜志, 2020, 20(8): 901-912.
16. Glasgow RE, Estabrooks PA, Ory MG. Characterizing evolving frameworks: issues from Esmail et al. (2020) review. Implement Sci, 2020, 15(1): 53.
17. Bellg AJ, Borrelli B, Resnick B, et al. Enhancing treatment fidelity in health behavior change studies: best practices and recommendations from the NIH behavior change consortium. Health Psychol, 2004, 23(5): 443-451.
18. 婁曉民. RE-AIM模型. 余金明, 姜慶五, 主編. 現代健康教育學. 上海: 復旦大學出版社, 2019: 120-123.
19. Holtrop JS, Estabrooks PA, Gaglio B, et al. Understanding and applying the RE-AIM framework: clarifications and resources. J Clin Transl Sci, 2021, 5(1): e126.
20. 田文華. 基于RE-AIM模型的城鄉居民大病保險模式評估. 復旦學報(社會科學版), 2021, 63(1): 152-160.
21. Gaglio B, Shoup JA, Glasgow RE. The RE-AIM framework: a systematic review of use over time. Am J Public Health, 2013, 103(6): e38-e46.
22. Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf, 2008, 34(4): 228-243.
23. Liles EG, Schneider JL, Feldstein AC, et al. Implementation challenges and successes of a population-based colorectal cancer screening program: a qualitative study of stakeholder perspectives. Implement Sci, 2015, 10: 41.
24. King DK, Shoup JA, Raebel MA, et al. Planning for implementation success using RE-AIM and CFIR frameworks: a qualitative study. Front Public Health, 2020, 8: 59.
25. 魯春麗, 曹卉娟, 徐東, 等. 實施科學產生的背景、概念和國內外發展近況. 中國中西醫結合雜志, 2020, 40(11): 1378-1380.
26. Klesges LM, Estabrooks PA, Dzewaltowski DA, et al. Beginning with the application in mind: designing and planning health behavior change interventions to enhance dissemination. Ann Behav Med, 2005, 29 Suppl: 66-75.
27. Kessler RS, Purcell EP, Glasgow RE, et al. What does it mean to "employ" the RE-AIM model. Eval Health Prof, 2013, 36(1): 44-66.
28. Galaviz KI, Harden SM, Smith E, et al. Physical activity promotion in Latin American populations: a systematic review on issues of internal and external validity. Int J Behav Nutr Phys Act, 2014, 11: 77.
29. Hoehner CM, Ribeiro IC, Parra DC, et al. Physical activity interventions in Latin America: expanding and classifying the evidence. Am J Prev Med, 2013, 44(3): e31-e40.
30. Glasgow RE, Estabrooks PE. Pragmatic applications of RE-AIM for health care initiatives in community and clinical settings. Prev Chronic Dis, 2018, 15: E02.
31. 王擁軍, 李子孝, 谷鴻秋, 等. 中國卒中報告2019(中文版)(1). 中國卒中雜志, 2020, 15(10): 1037-1043.
32. 何莎, 張田, 孫偉, 等. 中國內地腦卒中治療多中心隨機對照試驗方法學質量現狀報告. 華西醫學, 2020, 35(6): 673-678.
33. Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet, 1993, 342(8883): 1317-1322.
34. Pronovost PJ. Enhancing physicians' use of clinical guidelines. JAMA, 2013, 310(23): 2501-2502.
35. Liu M, Zhang C, Zha Q, et al. A national survey of Chinese medicine doctors and clinical practice guidelines in China. BMC Complement Altern Med, 2017, 17(1): 451.
36. Cabana MD, Rand CS, Powe NR, et al. Why don't physicians follow clinical practice guidelines. A framework for improvement. JAMA, 1999, 282(15): 1458-1465.
37. Working Group Members for Stroke. International consortium for health outcomes measurement (ICHOM). Stroke data collection reference guide. 2018.
38. RANKIN J. Cerebral vascular accidents in patients over the age of 60. Ⅱ. Prognosis. Scott Med J, 1957, 2(5): 200-215.
39. Quinn TJ, Langhorne P, Stott DJ. Barthel index for stroke trials: development, properties, and application. Stroke, 2011, 42(4): 1146-1151.
40. Goldstein LB, Bertels C, Davis JN. Interrater reliability of the NIH stroke scale. Arch Neurol, 1989, 46(6): 660-662.
41. Vernon W. The Delphi technique: a review. Int J Ther Rehabil, 2013, 16(2): 69-76.
42. Mullen R, Kydd A, Fleming A, et al. A practical guide to the systematic application of nominal group technique. Nurse Res, 2021, 29(1): 14-20.
43. 柯惠新, 黃京華, 沈浩. 調查研究中的統計分析法. 北京: 北京廣播學院出版社, 1996: 351.
44. 黃敬亨. 健康教育學. 上海: 上海醫科大學出版社, 1997: 212.
45. 柯惠新, 沈浩. 調查研究中的統計分析方法. 北京: 北京傳媒大學出版社, 2005: 324.
46. 席仲恩, 杜玨 (譯). 量表編制: 理論與應用(第三版). 重慶: 重慶大學出版社, 2016.
47. Schmidt RC. Managing Delphi surveys using nonparametric statistical techniques. Decision Sci, 1997, 28(3): 763-774.
48. 楊宗義. 教育統計學(第1版). 北京: 科學技術文獻出版社, 1990.
49. 魯曼, 劉虹, 鄭志杰. 醫學生全球勝任力指標體系構建的研究. 中華醫學教育雜志, 2021, 41(12): 1057-1061.
50. Shen L, Yang J, Jin X, et al. Based on Delphi method and analytic hierarchy process to construct the evaluation index system of nursing simulation teaching quality. Nurse Educ Today, 2019, 79: 67-73.
51. 張天芳, 陳春芳. 關于Kendall協和系數的理解. 統計學與應用, 2020, 9(4): 578-581.
52. Reilly KL, Kennedy S, Porter G, et al. Comparing, contrasting, and integrating dissemination and implementation outcomes included in the RE-AIM and implementation outcomes frameworks. Front Public Health, 2020, 8: 430.

Chinese Journal of Evidence-Based Medicine

Method of using RE-AIM framework to develop implementation outcome indicators

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content