×

Consistency of cluster analysis for cognitive diagnosis: the reduced reparameterized unified model and the general diagnostic model. (English) Zbl 1345.62141

Summary: The asymptotic classification theory of cognitive diagnosis (ACTCD) provided the theoretical foundation for using clustering methods that do not rely on a parametric statistical model for assigning examinees to proficiency classes. Like general diagnostic classification models, clustering methods can be useful in situations where the true diagnostic classification model (DCM) underlying the data is unknown and possibly misspecified, or the items of a test conform to a mix of multiple DCMs. Clustering methods can also be an option when fitting advanced and complex DCMs encounters computational difficulties. These can range from the use of excessive CPU times to plain computational infeasibility. However, the propositions of the ACTCD have only been proven for the Deterministic Input Noisy Output “AND” gate (DINA) model and the Deterministic Input Noisy Output “OR” gate (DINO) model. For other DCMs, there does not exist a theoretical justification to use clustering for assigning examinees to proficiency classes. But if clustering is to be used legitimately, then the ACTCD must cover a larger number of DCMs than just the DINA model and the DINO model. Thus, the purpose of this article is to prove the theoretical propositions of the ACTCD for two other important DCMs, the Reduced Reparameterized Unified Model and the General Diagnostic Model.

MSC:

62P15 Applications of statistics to psychology
62H30 Classification and discrimination; cluster analysis (statistical aspects)
62J20 Diagnostics, and linear inference and regression
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Ayers, E., Nugent, R., & Dean, N. (2008). Skill set profile clustering based on student capability vectors computed from online tutoring data. In R. S. J. de Baker, T. Barnes, & J. E. Beck (Eds.), Educational data mining 2008: Proceedings of the 1st International conference on educational data mining, Montreal, QC, Canada (pp. 210-217). Retrieved from http://www.educationaldatamining.org/EDM2008/uploads/proc/full
[2] Chiu, C.-Y., Douglas, J. A., & Li, X. (2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74, 633-665. · Zbl 1179.62087 · doi:10.1007/s11336-009-9125-0
[3] Chiu, C.-Y., & Köhn, H.-F. (2015a). Consistency of cluster analysis for cognitive diagnosis: The DINO model and the DINA model revisited. Applied Psychological Measurement, 39, 465-479.
[4] Chiu, C.-Y., & Köhn, H.-F. (2015b). The Reduced RUM as a logit model: Parameterization and constraints. Psychometrika. doi:10.1007/s11336-015-9460-2. · Zbl 1345.62142
[5] Chiu, C.-Y., & Ma, W. (2013). ACTCD: Asymptotic classification theory for cognitive diagnosis. R package version 1.0-0. Retrieved from the Comprehensive R Archive Network [CRAN] website http://cran.r-project.org/web/packages/ACTCD/. · Zbl 1284.62775
[6] de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76, 179-199. · Zbl 1284.62775 · doi:10.1007/s11336-011-9207-7
[7] DiBello, LV; Roussos, LA; Stout, WF; Rao, CR (ed.); Sinharay, S. (ed.), Review of cognitively diagnostic assessment and a summary of psychometric models, 979-1030 (2007), Amsterdam · Zbl 1255.91382
[8] DiBello, LV; Stout, WF; Roussos, LA; Nichols, PD (ed.); Chipman, SF (ed.); Brennan, RL (ed.), Unified cognitive/psychometric diagnostic assessment likelihood-based classification techniques, 361-389 (1995), Mahwah, NJ
[9] DiBello, L., Stout, W., Roussos, L., Templin, J., Chen, H., Zapata, D., et al. (2010). Arpeggiodocumentation and analysis manual. Chicago, IL: Applied Informative Assessment Research Enterprise (AIARE)-LLC.
[10] Feng, Y., Habing, B. T., & Huebner, A. (2014). Parameter estimation of the Reduced RUM using the EM algorithm. Applied Psychological Measurement, 38, 137-150. · doi:10.1177/0146621613502704
[11] Haberman, SJ; Davier, M.; Rao, CR (ed.); Sinharay, S. (ed.), Some notes on models for cognitively based skills diagnosis, 1031-1038 (2007), Amsterdam · Zbl 1256.91046
[12] Hartigan, J. A. (1975). Clustering algorithms. New York: Wiley. · Zbl 0372.62040
[13] Hartz, S. M. (2002). A Bayesian framework for the Unified Model for assessing cognitive abilities: Blending theory with practicality (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 3044108)
[14] Hartz, S. M., & Roussos, L. A. (October 2008). The Fusion Model for skill diagnosis: Blending theory with practicality. (Research report No. RR-08-71). Princeton, NJ: Educational Testing Service.
[15] Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning (2nd ed.). New York: Springer. · Zbl 1273.62005 · doi:10.1007/978-0-387-84858-7
[16] Henson, R. A., Templin, J. L., & Willse, J. T. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74, 191-210. · Zbl 1243.62140 · doi:10.1007/s11336-008-9089-5
[17] Hubert, L., & Arabie, P. (1985). Comparing partitions. Journal of Classification, 2, 193-218. · doi:10.1007/BF01908075
[18] Johnson, S. C. (1967). Hierarchical clustering schemes. Psychometrika, 32, 241-254. · Zbl 1367.62191 · doi:10.1007/BF02289588
[19] Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258-272. · doi:10.1177/01466210122032064
[20] Leighton, J., & Gierl, M. (2007). Cognitive diagnostic assessment for education: Theory and applications. Cambridge, UK: Cambridge University Press. · doi:10.1017/CBO9780511611186
[21] Lunn, D., Spiegelhalter, D., Thomas, A., & Best, N. (2009). The BUGS project: Evolution, critique, and future directions. Statistics in Medicine, 28, 3049-3067. · doi:10.1002/sim.3680
[22] Macready, G. B., & Dayton, C. M. (1977). The use of probabilistic models in the assessment of mastery. Journal of Educational Statistics, 2, 99-120. · doi:10.2307/1164802
[23] Muthén, L. K., & Muthén, B. O. (1998-2012). MplusUser’s guide (7th ed.). Los Angeles: Muthén & Muthén.
[24] Robitzsch, A., Kiefer, T., George, A. C., & Uenlue, A. (2015). CDM: Cognitive diagnosis modeling. R package version 3.1-14. Retrieved from the Comprehensive R Archive Network [CRAN] website http://CRAN.R-project.org/package=CDM
[25] Rupp, A. A., Templin, J. L., & Henson, R. A. (2010). Diagnostic measurement. Theory, methods, and applications. New York: Guilford.
[26] Steinley, D. (2004). Properties of the Hubert-Arabie Adjusted Rand Index. Psychological Methods, 9, 386-396. · doi:10.1037/1082-989X.9.3.386
[27] Tatsuoka, K. (1985). A probabilistic model for diagnosing misconception in the pattern classification approach. Journal of Educational Statistics, 12, 55-73.
[28] Templin, J., & Bradshaw, L. (2014). Hierarchical diagnostic classification models: A family of models for estimating and testing attribute hierarchies. Psychometrika, 79, 317-339. · Zbl 1288.62189 · doi:10.1007/s11336-013-9362-0
[29] Templin, J. L., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287-305. · doi:10.1037/1082-989X.11.3.287
[30] Vermunt, J. K., & Magidson, J. (2000). Latent GOLD’susers’s guide. Boston: Statistical Innovations Inc.
[31] von Davier, M. (2005). A general diagnostic model applied to language testing data (Research report No. RR-05-16). Princeton, NJ: Educational Testing Service.
[32] von Davier, M. (2011). Equivalency of the DINA model and a constrained general diagnostic model (Research report No. RR-11-37). Princeton, NJ: Educational Testing Service. · Zbl 1406.91392
[33] von Davier, M. (2006). Multidimensional latent trait modelling (MDLTM) [Software program]. Princeton, NJ: Educational Testing Service.
[34] von Davier, M. (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61, 287-301. · doi:10.1348/000711007X193957
[35] von Davier, M. (2014). The DINA model as a constrained general diagnostic model: Two variants of a model equivalency. British Journal of Mathematical and Statistical Psychology, 67, 49-71. · Zbl 1406.91392 · doi:10.1111/bmsp.12003
[36] Davier, M.; Cheng, C.; Cheng, CA; Yan, D. (ed.); Davier, AA (ed.); Lewis, C. (ed.), Multistage testing using diagnostic models, 219-227 (2014), Boca Raton, FL
[37] Ward, J. H. (1963). Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 58, 236-244. · doi:10.1080/01621459.1963.10500845
[38] Willse, J., Henson, R., & Templin, J. (2007). Using sum scores or IRT in place of cognitive diagnosis models: Can existing or more familiar models do the job? Paper presented at the Annual Meeting of the National Council on Measurement in Education, Chicago, IL.
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.