Interpretation of the Delphi critical appraisal tool (DCAT)_Chinese Journal of Evidence-Based Medicine

Authors：

FAN Qinlin ¹ ,  LI Sheyu ^1,2

1. Department of Endocrinology and Metabolism, West China Hospital, Sichuan University, Chengdu 610041, P. R. China;
2. MAGIC China Center, Chinese Evidence-Based Medicine Center, West China Hospital, Sichuan University, Chengdu 610041, P. R. China;

Corresponding?author：

LI Sheyu, Email: lisheyu@gmail.com

Keywords：

Delphi technique; DCAT; Critical appraisal tool; Consensus

DOI：

10.7507/1672-2531.202605154

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

The Delphi technique is a qualitative research method for building group consensus and has been widely adopted in the development of guidance documents. Published Delphi studies vary considerably in methodological quality, yet no dedicated critical appraisal tool existed until the Delphi Critical Appraisal Tool (DCAT) was published in 2025. DCAT comprises four core items (anonymity, iteration, statistical summary and controlled feedback) and twelve additional items (justification for Delphi, type of Delphi, panelist expertise, panelist retention, preliminary work, design choices, survey questions, response items, pilot test, data analysis, transparent reporting and ethical issues) and represents the first discipline-independent critical appraisal tool for Delphi studies, applicable to study protocol development, peer review and published study appraisal. This article interprets the item structure and appraisal logic of DCAT, aiming to facilitate quality appraisal and methodological improvement of Delphi studies.

Citation： FAN Qinlin, LI Sheyu. Interpretation of the Delphi critical appraisal tool (DCAT). Chinese Journal of Evidence-Based Medicine, 2026, 26(9): 1088-1094. doi: 10.7507/1672-2531.202605154 Copy

1.	Dalkey N, Helmer O. An experimental application of the DELPHI method to the use of experts[J]. Manage Sci, 1963, 9: 458-467.
2.	Hsu CC, Sandford BA. The Delphi technique: making sense of consensus[J]. Pract Assess Res Eval, 2007, 12(10): 1-8.
3.	Rinella ME, Lazarus JV, Ratziu V, et al. A multisociety Delphi consensus statement on new fatty liver disease nomenclature[J]. J Hepatol, 2023, 79(6): 1542-1556.
4.	Giacomelli R, Caporali R, Ciccia F, et al. Expert consensus on the treatment of patients with adult-onset Still's disease with the goal of achieving an early and long-term remission[J]. Autoimmun Rev, 2023, 22(12): 103400.
5.	Hoffmann-Vold AM, Maher TM, Philpot EE, et al. The identification and management of interstitial lung disease in systemic sclerosis: evidence-based European consensus statements[J]. Lancet Rheumatol, 2020, 2(2): e71-e83.
6.	Soriano JB, Murthy S, Marshall JC, et al. A clinical case definition of post-COVID-19 condition by a Delphi consensus[J]. Lancet Infect Dis, 2022, 22(4): e102-e107.
7.	Khodyakov D, Grant S, Kroger J, et al. Disciplinary trends in the use of the Delphi method: a bibliometric analysis[J]. PLoS One, 2023, 18(8): e0289009.
8.	Nasa P, Jain R, Juneja D. Delphi methodology in healthcare research: how to decide its appropriateness[J]. World J Methodol, 2021, 11(4): 116-129.
9.	Spranger J, Homberg A, Sonnberger M, et al. Reporting guidelines for Delphi techniques in health sciences: a methodological review[J]. Z Evid Fortbild Qual Gesundhwes, 2022, 172: 1-11.
10.	管慧, 戴國華, 高武霖, 等. 中醫藥臨床研究中應用德爾菲法的系統評價[J]. 中國循證醫學雜志, 2021, 21(2): 186-190.
11.	Sterne JAC, Savovi? J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials[J]. BMJ, 2019, 366: l4898.
12.	Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both[J]. BMJ, 2017, 358: j4008.
13.	Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique[J]. J Adv Nurs, 2000, 32(4): 1008-1015.
14.	Jünger S, Payne SA, Brine J, et al. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: recommendations based on a methodological systematic review[J]. Palliat Med, 2017, 31(8): 684-706.
15.	Niederberger M, Schifano J, Deckert S, et al. Delphi studies in social and health sciences-recommendations for an interdisciplinary standardized reporting (DELPHISTAR). Results of a Delphi study[J]. PLoS One, 2024, 19(8): e0304651.
16.	Gattrell WT, Logullo P, van Zuuren EJ, et al. ACCORD (ACcurate COnsensus Reporting Document): a reporting guideline for consensus methods in biomedicine developed via a modified Delphi[J]. PLoS Med, 2024, 21(1): e1004326.
17.	Hasson F, Keeney S, McKenna H. Revisiting the Delphi technique - research thinking and practice: a discussion paper[J]. Int J Nurs Stud, 2025, 168: 105119.
18.	Khodyakov D, Grant S, Kroger J, et al. RAND methodological guidance for conducting and critically appraising Delphi panels[M]. Santa Monica: RAND Corp, 2023: 1-72.
19.	Grant S, Khodyakov D. Proposal for a critical appraisal tool for studies using the Delphi method (DCAT)[J]. BMJ, 2025, 391: e084509.
20.	Dalal S, Khodyakov D, Srinivasan R, et al. ExpertLens: a system for eliciting opinions from a large pool of non-collocated experts with diverse knowledge[J]. Technol Forecast Soc Change, 2011, 78(8): 1426-1444.
21.	Keeney S, Hasson F, McKenna HP. A critical review of the Delphi technique as a research methodology for nursing[J]. Int J Nurs Stud, 2001, 38(2): 195-200.
22.	Woudenberg F. An evaluation of Delphi[J]. Technol Forecast Soc Change, 1991, 40(2): 131-150.
23.	Grant S, Booth M, Khodyakov D. Lack of pre-registered analysis plan allows unacceptable data mining for and selective reporting of consensus in Delphi studies[J]. J Clin Epidemiol, 2018, 99: 96-105.
24.	Fitch K, Bernstein SJ, Aguilar MD, et al. The RAND/UCLA appropriateness method user's manual[M]. Santa Monica: RAND Corp, 2001: 1-123.
25.	Avery AJ, Savelyich BSP, Sheikh A, et al. Identifying and establishing consensus on the most important safety features of GP computer systems: e-Delphi study[J]. Inform Prim Care, 2005, 13(1): 3-12.
26.	Gordon T, Pease A. RT Delphi: an efficient, "round-less" almost real time Delphi method[J]. Technol Forecast Soc Change, 2006, 73(4): 321-333.
27.	Lange T, Kopkow C, Lützner J, et al. Comparison of different rating scales for the use in Delphi studies: different scales lead to different consensus and show different test-retest reliability[J]. BMC Med Res Methodol, 2020, 20(1): 28.
28.	Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations[J]. BMJ, 2008, 336(7650): 924-926.
29.	Tugwell P, Knottnerus JA. When does a good practice statement not justify an evidence based guideline[J]. J Clin Epidemiol, 2015, 68(5): 477-479.

1. Dalkey N, Helmer O. An experimental application of the DELPHI method to the use of experts[J]. Manage Sci, 1963, 9: 458-467.
2. Hsu CC, Sandford BA. The Delphi technique: making sense of consensus[J]. Pract Assess Res Eval, 2007, 12(10): 1-8.
3. Rinella ME, Lazarus JV, Ratziu V, et al. A multisociety Delphi consensus statement on new fatty liver disease nomenclature[J]. J Hepatol, 2023, 79(6): 1542-1556.
4. Giacomelli R, Caporali R, Ciccia F, et al. Expert consensus on the treatment of patients with adult-onset Still's disease with the goal of achieving an early and long-term remission[J]. Autoimmun Rev, 2023, 22(12): 103400.
5. Hoffmann-Vold AM, Maher TM, Philpot EE, et al. The identification and management of interstitial lung disease in systemic sclerosis: evidence-based European consensus statements[J]. Lancet Rheumatol, 2020, 2(2): e71-e83.
6. Soriano JB, Murthy S, Marshall JC, et al. A clinical case definition of post-COVID-19 condition by a Delphi consensus[J]. Lancet Infect Dis, 2022, 22(4): e102-e107.
7. Khodyakov D, Grant S, Kroger J, et al. Disciplinary trends in the use of the Delphi method: a bibliometric analysis[J]. PLoS One, 2023, 18(8): e0289009.
8. Nasa P, Jain R, Juneja D. Delphi methodology in healthcare research: how to decide its appropriateness[J]. World J Methodol, 2021, 11(4): 116-129.
9. Spranger J, Homberg A, Sonnberger M, et al. Reporting guidelines for Delphi techniques in health sciences: a methodological review[J]. Z Evid Fortbild Qual Gesundhwes, 2022, 172: 1-11.
10. 管慧, 戴國華, 高武霖, 等. 中醫藥臨床研究中應用德爾菲法的系統評價[J]. 中國循證醫學雜志, 2021, 21(2): 186-190.
11. Sterne JAC, Savovi? J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials[J]. BMJ, 2019, 366: l4898.
12. Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both[J]. BMJ, 2017, 358: j4008.
13. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique[J]. J Adv Nurs, 2000, 32(4): 1008-1015.
14. Jünger S, Payne SA, Brine J, et al. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: recommendations based on a methodological systematic review[J]. Palliat Med, 2017, 31(8): 684-706.
15. Niederberger M, Schifano J, Deckert S, et al. Delphi studies in social and health sciences-recommendations for an interdisciplinary standardized reporting (DELPHISTAR). Results of a Delphi study[J]. PLoS One, 2024, 19(8): e0304651.
16. Gattrell WT, Logullo P, van Zuuren EJ, et al. ACCORD (ACcurate COnsensus Reporting Document): a reporting guideline for consensus methods in biomedicine developed via a modified Delphi[J]. PLoS Med, 2024, 21(1): e1004326.
17. Hasson F, Keeney S, McKenna H. Revisiting the Delphi technique - research thinking and practice: a discussion paper[J]. Int J Nurs Stud, 2025, 168: 105119.
18. Khodyakov D, Grant S, Kroger J, et al. RAND methodological guidance for conducting and critically appraising Delphi panels[M]. Santa Monica: RAND Corp, 2023: 1-72.
19. Grant S, Khodyakov D. Proposal for a critical appraisal tool for studies using the Delphi method (DCAT)[J]. BMJ, 2025, 391: e084509.
20. Dalal S, Khodyakov D, Srinivasan R, et al. ExpertLens: a system for eliciting opinions from a large pool of non-collocated experts with diverse knowledge[J]. Technol Forecast Soc Change, 2011, 78(8): 1426-1444.
21. Keeney S, Hasson F, McKenna HP. A critical review of the Delphi technique as a research methodology for nursing[J]. Int J Nurs Stud, 2001, 38(2): 195-200.
22. Woudenberg F. An evaluation of Delphi[J]. Technol Forecast Soc Change, 1991, 40(2): 131-150.
23. Grant S, Booth M, Khodyakov D. Lack of pre-registered analysis plan allows unacceptable data mining for and selective reporting of consensus in Delphi studies[J]. J Clin Epidemiol, 2018, 99: 96-105.
24. Fitch K, Bernstein SJ, Aguilar MD, et al. The RAND/UCLA appropriateness method user's manual[M]. Santa Monica: RAND Corp, 2001: 1-123.
25. Avery AJ, Savelyich BSP, Sheikh A, et al. Identifying and establishing consensus on the most important safety features of GP computer systems: e-Delphi study[J]. Inform Prim Care, 2005, 13(1): 3-12.
26. Gordon T, Pease A. RT Delphi: an efficient, "round-less" almost real time Delphi method[J]. Technol Forecast Soc Change, 2006, 73(4): 321-333.
27. Lange T, Kopkow C, Lützner J, et al. Comparison of different rating scales for the use in Delphi studies: different scales lead to different consensus and show different test-retest reliability[J]. BMC Med Res Methodol, 2020, 20(1): 28.
28. Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations[J]. BMJ, 2008, 336(7650): 924-926.
29. Tugwell P, Knottnerus JA. When does a good practice statement not justify an evidence based guideline[J]. J Clin Epidemiol, 2015, 68(5): 477-479.

Previous Article
Principle of adverse drug reaction signal detection methods and their applications
Next Article
Introduction to ACTIVE framework to describe stakeholder involvement in systematic reviews

Chinese Journal of Evidence-Based Medicine

Interpretation of the Delphi critical appraisal tool (DCAT)

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content