ANALYSIS AND EVALUATION OF MULTIPLE-CHOICE TEST ITEMS AND TEST DESIGN: A STUDY ON APPLICATION OF ITEM RESPONSE THEORY

Nguyễn Văn Cảnh; Phạm Văn Tặc

doi:10.34238/tnu-jst.4885

ANALYSIS AND EVALUATION OF MULTIPLE-CHOICE TEST ITEMS AND TEST DESIGN: A STUDY ON APPLICATION OF ITEM RESPONSE THEORY

About this article

Received: 12/08/21 Revised: 07/09/21 Published: 07/09/21

Authors

1. Nguyen Van Canh , Dong Thap University
2. Pham Van Tac, Dong Thap University

Abstract

This article presents the findings on applying the item response theory for a 2-parameter model in analyzing and evaluating question items and designing multiple-choice tests. Based on the results of data analysis by R (package ltm) of non-majored students’ performance in English 1 exam papers used in Dong Thap University from 2017 to 2021, the study identified the satisfactory items which could meet the exam requirements and unsatisfactory question ones for further adjustment or improvement. Among the unsatisfactory items, some abnormal characteristics, seriously violating the tests’ limitations in terms of difficulty and discriminate level, must have been definitely removed from the test papers. In addition, the study findings also show that the test items had a relatively low level of measuring students’ competence (below 0.0 according to the competence scale). Finally, the study introduced the way of applying the information curve tool to design test items to help accurately measure the students’ competence based on the characteristic parameters of items.

Keywords

Item Response Theory; Multiple – choice item; Information Curve; R software (package ltm); GeoGebra software

Full Text:

PDF

References

[1] T. Q. Lam, Measurement in Education - Theory and Application. Hanoi: Vietnam National University Press, Hanoi, 2011.

[2] G. Rasch, Probabilistic models for some intelligence and attainment tests. Copenhagen: Danish Institute for Educational Research, 1960.

[3] A. Birnbaum, "Some Latent Trait Models and Their Use in Inferring an Examinee’s Ability," in Statistical Theories of Mental Test Scores, F. M. Lord and M. R. Novick, Eds., Addison-Wesley, 1968, pp. 397-479.

[4] F. Baker, The basic of item response theory. Washington: ERIC Clearinghouse on Assessment and Evaluation, 2001.

[5] S. L. V. Bortolotti, R. Tezza, D. F. de Andrade, A. C. Bornia, and A. F. de Sousa Júnior, "Relevance and advantages of using the item response theory," Quality & Quantity, vol. 47, no. 4, pp. 2341-2360, 2013.

[6] S. Rakkapao, S. Prasitpong, and K. Arayathanitkul, "Analysis test of understanding of vectors with the three-parameter logistic model of item response theory and item response curves technique," Physical review physics education research, vol. 12, no. 2, 2016, Art. no. 020135.

[7] C. B. Essen, I. E. Idaka, and M. A. Metibemu, "Item level diagnostics and model-data fit in item response theory (IRT) using BILOG-MG v3.0 and IRTPRO v3.0 programmes," Global Journal of Educational Research, vol. 16, no. 2, pp. 87-94, 2017.

[8] G. C. Foster, H. Min, and M. J. Zickar, "Review of item response theory practices in organizational research: Lessons learned and paths forward," Organizational Research Methods, vol. 20, no. 3, pp. 465-486, 2017.

[9] F. A. Setiawati, R. E. Izzaty, and V. Hidayat, "Items parameters of the space-relations subtest using item response theory," Data in brief, vol. 19, pp. 1785-1793, 2018.

[10] M. S. Mu’iz, I. Kaniawati, and T. R. Ramalis, "Analyzing instrument characteristics of critical thinking skills and mastery of concepts based on item response theory," International Conference on Mathematics and Science Education of Universitas Pendidikan Indonesia, vol. 3, pp. 162-167, 2018.

[11] T. Q. Lam, M. N. Lam, T. M. Le, and B. D. Vu, "VITESTA software and analysis of test data," (in Vietnamese), Vietnam Journal of Education, vol. 176, pp. 10-12, 2007.

[12] M. H. T. Nguyen and T. D. Nguyen, "Measurement Assessment in the objective test: Question difficulty and Examinees’ ability," (in Vietnamese), Vietnam National University Journal of Science, vol. 4, pp. 34-47, 2006.

[13] V. A. Le, U. H. Pham, C. H. Doan, and H. T. Le, "Using Gibbs Sampler to evaluate item difficulty in Rasch model," (in Vietnamese), Ho Chi Minh City University of Education Journal of Science, vol. 14, no. 4, pp. 119-130, 2017.

[14] K. A. Bui and P. N. Bui, "Using IATA to analyze, evaluate and improve the quality of the multiple-choice items in chapter power functions, exponential functions and logarithmic functions," (in Vietnamese), Can Tho University Journal of Science, vol. 54, no. 9C, pp. 81-93, 2018.

[15] C. V. Nguyen and H. P. Nguyen, "Analyzing and selecting multiple-choice test items based on Classical Test Theory and Item Response Theory," (in Vietnamese), Ho Chi Minh city University of Education Journal of Science, vol. 17, no. 10, pp. 1804-1818, 2020.

[16] C. H. Doan, V. A. Le, and U. H. Pham, "Applying three-parameter logistic model in validating the level of difficulty, discrimination and guessing of items in a multiple-choice test," (in Vietnamese), Ho Chi Minh City University of Education Journal of Science, vol. 7, no. 8, pp. 174-184, 2016.

[17] T. H. B. Nguyen, "Using Quest software to analyze objective test questions," (in Vietnamese), Journal of Science and Technology - Da Nang University, vol. 2, pp. 119-126, 2008.

[18] Q. N. Bui, "Evaluation of the quality of multiple-choice test bank for the module of Introduction to Anthropology by using the RASCH model and QUEST software," (in Vietnamese), Science of Technology Development - Viet Nam National University Ho Chi Minh City, vol. 20, no. X3, pp. 42-54, 2017.

[19] C. V. Nguyen and T. Q. Nguyen, "Applying ConQuest software with the two-parameter IRT model to evaluate the quality of multiple-choice test," (in Vietnamese), HNUE Journal of Science, vol. 65, no. 7, pp. 230-242, 2020.

[20] H. P. Nguyen and N. T. Du, "The analysis and selection of objective test items based on S-P chart, Grey Relational Analysis, and ROC curve," (in Vietnamese), Ho Chi Minh City University of Education Journal of Science, vol. 6, no. 72, pp. 163-173, 2015.

[21] H. P. Nguyen, "Using GSP chart and ROC method to analyze and select multiple-choice items," (in Vietnamese), Dong Thap University Journal of Science, vol. 24, no. 2, pp. 11-17, 2017.

[22] D. Rizopoulos, "An R package for latent variable modeling and item response theory analysis," Journal of Statistical Software, vol. 17, no. 5, pp. 1-25, 2006.

[23] M. Hohenwarter and J. Preiner, "Creating mathlets with open source tools," The Journal of Online Mathematics and Its Applications, vol. 7, pp. 1-29, 2007.

"Times New Roman",serif;mso-fareast-font-family:Calibri;mso-fareast-theme-font:

minor-latin;mso-ansi-language:PT-BR;mso-fareast-language:EN-US;mso-bidi-language:

AR-SA'>

DOI: https://doi.org/10.34238/tnu-jst.4885

Refbacks

There are currently no refbacks.



Remember me