SINTA

2.5

Impact

Google Scholar

10

h-index

Indonesian Journal of Education Research (IJoER)

Open Access Journal

What the Content Validity Index Does Not Show: Rater Severity and Cut-Point Sensitivity in a Two-Round Delphi Validation and Implications for Instrument Validation in Education

Nguyen Dinh Tho
Thu Dau Mot University ROR
Viet Nam
thond@tdmu.edu.vn
Nguyen Phuong Chi
Hanoi National University of Education ROR
Viet Nam
Pham Tho Hoan
Hanoi National University of Education ROR
Viet Nam
Share

  • Purpose of the study: This study examines what the content validity index conceals rather than what it reports. Using a two round Delphi validation, it asks how far rater severity varies within one expert panel, how sensitive the consensus conclusion is to the dichotomisation point, and which findings survive across four index specifications.

    Methodology: The complete Round 2 rating matrix of a two round Delphi validation of a competence framework was re-analysed (20 experts, 169 items, 3,380 ratings). Analyses in base R comprised per rater severity summaries, cut point and threshold sensitivity, a leave one expert out jackknife, four index specifications, pairwise Spearman correlations against a permutation null model, and non-parametric facet comparisons.

    Main Findings: Expert mean ratings spanned 3.94 to 4.99 and top category use spanned 1.2% to 99.4%, yet the proportion of ratings at or above 4, the only quantity the index uses, spanned just 91.1% to 100%. Omitting any single expert left all 169 items in consensus, whereas restricting relevance to the top category cut qualifying items to 16. Aiken's V and the modified kappa left the conclusion intact. Rank agreement was real but minimal.

    Novelty/Originality of this study: Validation reports vary the panel and almost never vary the index specification. This study varies both on one dataset, shows that fragility is confined to stricter dichotomisation while indices using the full ordinal scale remain stable, settles by permutation whether weak rank agreement reflects disagreement or attenuation, and derives a diagnostic set requiring no new data. For education researchers and developers of assessment instruments, the practical implication is that a content validity index should be reported together with the rating distribution, per rater summaries and at least one index that uses the full scale before an instrument is used for curriculum, assessment or certification decisions.

  • How to cite

    [1]
    N. D. Tho, N. P. Chi, and P. T. Hoan, “What the Content Validity Index Does Not Show: Rater Severity and Cut-Point Sensitivity in a Two-Round Delphi Validation and Implications for Instrument Validation in Education”, Ind. Jou. Edu. Rsc, vol. 7, no. 5, pp. 865–700, Oct. 2026, doi: 10.37251/ijoer.v7i5.3787.

    More Citation Formats

  • 10
    Abstract views
    11
    Downloads

    Metrics — Badges

    1. D. F. Polit and C. T. Beck, “The content validity index: Are you sure you know what’s being reported? Critique and recommendations,” Research in Nursing & Health, vol. 29, no. 5, pp. 489–497, Oct. 2006, doi: 10.1002/nur.20147.
    2. D. F. Polit, C. T. Beck, and S. V. Owen, “Is the CVI an acceptable indicator of content validity? Appraisal and recommendations,” Research in Nursing & Health, vol. 30, no. 4, pp. 459–467, Aug. 2007, doi: 10.1002/nur.20199.
    3. M. R. Lynn, “Determination and quantification of content validity,” Nursing Research, vol. 35, no. 6, pp. 382–386, Nov. 1986, doi: 10.1097/00006199-198611000-00017.
    4. E. Almanasreh, R. Moles, and T. F. Chen, “Evaluation of methods used for estimating content validity,” Research in Social and Administrative Pharmacy, vol. 15, no. 2, pp. 214–221, Feb. 2019, doi: 10.1016/j.sapharm.2018.03.066.
    5. M. S. B. Yusoff, “ABC of content validation and content validity index calculation,” Education in Medicine Journal, vol. 11, no. 2, pp. 49–54, Jun. 2019, doi: 10.21315/eimj2019.11.2.6.
    6. F. E. Saal, R. G. Downey, and M. A. Lahey, “Rating the ratings: Assessing the psychometric quality of rating data,” Psychological Bulletin, vol. 88, no. 2, pp. 413–428, Sep. 1980, doi: 10.1037/0033-2909.88.2.413.
    7. C. M. Myford and E. W. Wolfe, “Detecting and measuring rater effects using many-facet Rasch measurement: Part I,” Journal of Applied Measurement, vol. 4, no. 4, pp. 386–422, 2003.
    8. C. M. Myford and E. W. Wolfe, “Detecting and measuring rater effects using many-facet Rasch measurement: Part II,” Journal of Applied Measurement, vol. 5, no. 2, pp. 189–227, 2004.
    9. S. A. Wind and M. E. Peterson, “A systematic review of methods for evaluating rating quality in language assessment,” Language Testing, vol. 35, no. 2, pp. 161–192, Apr. 2018, doi: 10.1177/0265532216686999.
    10. S. A. Wind, “Examining the Impacts of Rater Effects in Performance Assessments,” Applied Psychological Measurement, vol. 43, no. 2, pp. 159–171, Mar. 2019, doi: 10.1177/0146621618789391.
    11. C. H. Lawshe, “A quantitative approach to content validity,” Personnel Psychology, vol. 28, no. 4, pp. 563–575, Dec. 1975, doi: 10.1111/j.1744-6570.1975.tb01393.x.
    12. L. R. Aiken, “Content validity and reliability of single items or questionnaires,” Educational and Psychological Measurement, vol. 40, no. 4, pp. 955–959, Dec. 1980, doi: 10.1177/001316448004000419.
    13. L. R. Aiken, “Three coefficients for analyzing the reliability and validity of ratings,” Educational and Psychological Measurement, vol. 45, no. 1, pp. 131–142, Mar. 1985, doi: 10.1177/0013164485451012.
    14. V. Zamanzadeh, A. Ghahramanian, M. Rassouli, A. Abbaszadeh, H. Alavi-Majd, and A.-R. Nikanfar, “Design and implementation content validity study: development of an instrument for measuring patient-centered communication,” Journal of Caring Sciences, vol. 4, no. 2, pp. 165–178, Jun. 2015, doi: 10.15171/jcs.2015.017.
    15. D. M. Rubio, M. Berg-Weger, S. S. Tebb, E. S. Lee, and S. Rauch, “Objectifying content validity: Conducting a content validity study in social work research,” Social Work Research, vol. 27, no. 2, pp. 94–104, Jun. 2003, doi: 10.1093/swr/27.2.94.
    16. L. L. Davis, “Instrument review: Getting the most from a panel of experts,” Applied Nursing Research, vol. 5, no. 4, pp. 194–197, Nov. 1992, doi: 10.1016/S0897-1897(05)80008-4.
    17. S. G. Sireci, “Gathering and analyzing content validity data,” Educational Assessment, vol. 5, no. 4, pp. 299–321, Oct. 1998, doi: 10.1207/s15326977ea0504_2.
    18. S. G. Sireci, “The construct of content validity,” Social Indicators Research, vol. 45, no. 1–3, pp. 83–117, Nov. 1998, doi: 10.1023/A:1006985528729.
    19. S. N. Haynes, D. C. S. Richard, and E. S. Kubany, “Content validity in psychological assessment: A functional approach to concepts and methods,” Psychological Assessment, vol. 7, no. 3, pp. 238–247, Sep. 1995, doi: 10.1037/1040-3590.7.3.238.
    20. S. Messick, “Validity,” ETS Research Report Series, vol. 1987, no. 2, pp. i–208, 1987, doi: 10.1002/j.2330-8516.1987.tb00244.x.
    21. H. A. Von Der Gracht, “Consensus measurement in Delphi studies,” Technological Forecasting and Social Change, vol. 79, no. 8, pp. 1525–1536, Oct. 2012, doi: 10.1016/j.techfore.2012.04.013.
    22. I. R. Diamond et al., “Defining consensus: A systematic review recommends methodologic criteria for reporting of Delphi studies,” Journal of Clinical Epidemiology, vol. 67, no. 4, pp. 401–409, Apr. 2014, doi: 10.1016/j.jclinepi.2013.12.002.
    23. R. Boulkedid, H. Abdoul, M. Loustau, O. Sibony, and C. Alberti, “Using and reporting the delphi method for selecting healthcare quality indicators: a systematic review,” PLoS ONE, vol. 6, no. 6, p. e20476, Jun. 2011, doi: 10.1371/journal.pone.0020476.
    24. P. Nasa, R. Jain, and D. Juneja, “Delphi methodology in healthcare research: How to decide its appropriateness,” World Journal of Methodology, vol. 11, no. 4, pp. 116–129, Jul. 2021, doi: 10.5662/wjm.v11.i4.116.
    25. M. Niederberger, S. Köberich, and members of the DeWiss Network, “Coming to consensus: the Delphi technique,” European Journal of Cardiovascular Nursing, vol. 20, no. 7, pp. 692–695, Oct. 2021, doi: 10.1093/eurjcn/zvab059.
    26. J. Spranger, A. Homberg, M. Sonnberger, and M. Niederberger, “Reporting guidelines for Delphi techniques in health sciences: A methodological review,” Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen, vol. 172, pp. 1–11, Aug. 2022, doi: 10.1016/j.zefq.2022.04.025.
    27. N. Dalkey and O. Helmer, “An experimental application of the delphi method to the use of experts,” Management Science, vol. 9, no. 3, pp. 458–467, Apr. 1963, doi: 10.1287/mnsc.9.3.458.
    28. C.-C. Hsu and B. A. Sandford, “The delphi technique: making sense of consensus,” Practical Assessment, Research, and Evaluation, vol. 12, Art. no. 10, 2007, doi: 10.7275/pdz9-th90.
    29. M. T. Kane, “Validating the interpretations and uses of test scores,” Journal of Educational Measurement, vol. 50, no. 1, pp. 1–73, 2013, doi: 10.1111/jedm.12000.
    30. S. Messick, “Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning,” American Psychologist, vol. 50, no. 9, pp. 741–749, 1995, doi: 10.1037/0003-066X.50.9.741.
    31. American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, Standards for Educational and Psychological Testing. Washington, DC, USA: American Educational Research Association, 2014.
    32. S. Sireci and M. Faulkner-Bond, “Validity evidence based on test content,” Psicothema, vol. 26, no. 1, pp. 100–107, 2014, doi: 10.7334/psicothema2013.256.
    33. CC2020 Task Force, Computing Curricula 2020: Paradigms for Global Computing Education. New York, NY, USA: ACM, 2020, doi: 10.1145/3467967.
    34. A. N. Kumar et al., Computer Science Curricula 2023. New York, NY, USA: ACM, 2024, doi: 10.1145/3664191.
    35. S. Frezza et al., “Modelling competencies for computing education beyond 2020: A research based approach to defining competencies in the computing disciplines,” in Proc. Companion 23rd Annu. ACM Conf. Innovation and Technology in Computer Science Education (ITiCSE 2018 Companion), 2018, pp. 148–174, doi: 10.1145/3293881.3295782.
    36. A. Jonsson and G. Svingby, “The use of scoring rubrics: Reliability, validity and educational consequences,” Educational Research Review, vol. 2, no. 2, pp. 130–144, 2007, doi: 10.1016/j.edurev.2007.05.002.
    37. Y. M. Reddy and H. Andrade, “A review of rubric use in higher education,” Assessment & Evaluation in Higher Education, vol. 35, no. 4, pp. 435–448, 2010, doi: 10.1080/02602930902862859.
    38. M. J. Clayton, “Delphi: A technique to harness expert opinion for critical decision-making tasks in education,” Educational Psychology, vol. 17, no. 4, pp. 373–386, 1997, doi: 10.1080/0144341970170401.
    39. C. A. Wynd, B. Schmidt, and M. A. Schaefer, “Two quantitative approaches for estimating content validity,” Western Journal of Nursing Research, vol. 25, no. 5, pp. 508–518, 2003, doi: 10.1177/0193945903252998.
    40. R. C. MacCallum, S. Zhang, K. J. Preacher, and D. D. Rucker, “On the practice of dichotomization of quantitative variables,” Psychological Methods, vol. 7, no. 1, pp. 19–40, 2002, doi: 10.1037/1082-989X.7.1.19.
    41. D. G. Altman and P. Royston, “The cost of dichotomising continuous variables,” BMJ, vol. 332, no. 7549, p. 1080, 2006, doi: 10.1136/bmj.332.7549.1080.
    42. F. Hasson, S. Keeney, and H. McKenna, “Research guidelines for the Delphi survey technique,” Journal of Advanced Nursing, vol. 32, no. 4, pp. 1008–1015, 2000, doi: 10.1046/j.1365-2648.2000.t01-1-01567.x.
    43. C. Okoli and S. D. Pawlowski, “The Delphi method as a research tool: An example, design considerations and applications,” Information & Management, vol. 42, no. 1, pp. 15–29, 2004, doi: 10.1016/j.im.2003.11.002.
    44. S. Jünger, S. A. Payne, J. Brine, L. Radbruch, and S. G. Brearley, “Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review,” Palliative Medicine, vol. 31, no. 8, pp. 684–706, 2017, doi: 10.1177/0269216317690685.
    45. N. D. Tho, “Designing a SOLO-based analytic rubric for programming projects and a many-facet Rasch framework for its validation,” in International Conference on Educational Measurement and Assessment (VIETAME), Vietnam National University Press, to be published 2026.
    46. H. Baumgartner and J.-B. E. M. Steenkamp, “Response styles in marketing research: A cross-national investigation,” Journal of Marketing Research, vol. 38, no. 2, pp. 143–156, 2001, doi: 10.1509/jmkr.38.2.143.18840.
    47. Y. Van Vaerenbergh and T. D. Thomas, “Response styles in survey research: A literature review of antecedents, consequences, and remedies,” International Journal of Public Opinion Research, vol. 25, no. 2, pp. 195–217, 2013, doi: 10.1093/ijpor/eds021.
    48. B. Efron and R. J. Tibshirani, An Introduction to the Bootstrap. New York, NY, USA: Chapman & Hall, 1993.
    49. R. D. Penfield and P. R. Giacobbi, Jr., “Applying a score confidence interval to Aiken’s item content-relevance index,” Measurement in Physical Education and Exercise Science, vol. 8, no. 4, pp. 213–225, 2004, doi: 10.1207/s15327841mpee0804_3.
    50. P. I. Good, Permutation, Parametric and Bootstrap Tests of Hypotheses, 3rd ed. New York, NY, USA: Springer, 2005, doi: 10.1007/b138696.
    51. F. Wilcoxon, “Individual comparisons by ranking methods,” Biometrics Bulletin, vol. 1, no. 6, pp. 80–83, 1945, doi: 10.2307/3001968.
    52. D. Lakens, “Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs,” Frontiers in Psychology, vol. 4, Art. no. 863, 2013, doi: 10.3389/fpsyg.2013.00863.
    53. M. Friedman, “The use of ranks to avoid the assumption of normality implicit in the analysis of variance,” Journal of the American Statistical Association, vol. 32, no. 200, pp. 675–701, 1937, doi: 10.1080/01621459.1937.10503522.
    54. O. J. Dunn, “Multiple comparisons among means,” Journal of the American Statistical Association, vol. 56, no. 293, pp. 52–64, 1961, doi: 10.1080/01621459.1961.10482090.
    55. W. H. Kruskal and W. A. Wallis, “Use of ranks in one-criterion variance analysis,” Journal of the American Statistical Association, vol. 47, no. 260, pp. 583–621, 1952, doi: 10.1080/01621459.1952.10483441.
    56. R Core Team, R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing, 2025. [Online]. Available: https://www.R-project.org/
    57. V. Aryadoust, L. Y. Ng, and H. Sayama, “A comprehensive review of Rasch measurement in language assessment: Recommendations and guidelines for research,” Language Testing, vol. 38, no. 1, pp. 6–40, Jan. 2021, doi: 10.1177/0265532220927487.
    58. G. Engelhard, Jr., “Examining rater errors in the assessment of written composition with a many-faceted Rasch model,” Journal of Educational Measurement, vol. 31, no. 2, pp. 93–112, 1994, doi: 10.1111/j.1745-3984.1994.tb00436.x.
    59. J. M. Linacre, Many-Facet Rasch Measurement. Chicago, IL, USA: MESA Press, 1989.
    60. B. D. Wright and G. N. Masters, Rating Scale Analysis. Chicago, IL, USA: MESA Press, 1982.
    61. T. Eckes, Introduction to Many-Facet Rasch Measurement: Analyzing and Evaluating Rater-Mediated Assessments. Frankfurt am Main, Germany: Peter Lang, 2011, doi: 10.3726/978-3-653-04844-5.