OJER Home

2023 - Volume 7 - Number 1

Standard Setting with Artificial Neural Networks: TIMSS 2015 Mathematics Case

Mahmut Sami Koyuncu * ORCID: 0000-0002-1670-4083
Afyon Kocatepe University, Faculty of Education, Afyonkarahisar, TURKEY

Open Journal for Educational Research, 2023, 7(1), 53-62 * https://doi.org/10.32591/coas.ojer.0701.05053k
Received: 3 April 2023 ▪ Revised: 20 July 2023 ▪ Accepted: 30 July 2023

LICENCE: Creative Commons Attribution 4.0 International License.

ARTICLE (Full Text - PDF)

ABSTRACT:
This study aims to demonstrate the optimal way to determine the cut-off score to be used to interpret the total scores obtained from an achievement test or scale using the Artificial Neural Networks method. To this end, the multiple-choice item responses in the Booklet-11 Mathematics subtest at the 8th grade level in the TIMSS 2015 Turkey sample dataset were used to determine the cut-off score for the achievement test. The item responses in the “Students Like Learning Mathematics Scale” in the TIMSS 2015 8th grade Mathematics Student Questionnaire were used to determine the cut-off score for the scale. The data were accessed from the TIMSS international database and the data were analyzed in MATLAB R2017b software. As a result of the study, the most appropriate cut-off score to be used for the evaluation of the total scores obtained from the TIMSS 2015 8th grade level Booklet-11 Mathematics subtest was determined as 45.5 out of 0-100 points with the Artificial Neural Network analysis method. The overall level of agreement between the cut-off score and the pass/fail classification based on 400 points, which is the lowest level of the TIMSS International Benchmark, was determined as 81%. The most appropriate cut-off score to be used for the evaluation of the scores obtained from the Students Like Learning Mathematics Scale (SLLSS) in the TIMSS 2015 8th grade student survey was determined as 19.6 out of 9-36 points. The overall level of agreement between the cut-off score and the classification of students who like/don’t like learning mathematics using the criterion based on the expression given in the original scale description was found to be 83%. The results concluded that the validity of the standard-setting studies conducted with the artificial neural network method was high. As a result, researchers are recommended to use the Artificial Neural Networks method to determine the cut-off score to be used in the interpretation of the total scores obtained from the achievement test or the total scale scores obtained from the scales.

KEY WORDS: artificial neural networks, standard setting, cut-off score, TIMSS 2015.

CORRESPONDING AUTHOR:
Mahmut Sami Koyuncu, Afyon Kocatepe University, Faculty of Education, Afyonkarahisar, TURKEY.

REFERENCES:

Bircan, H., Zontul, M., & Yüksel, A. G. (2010). SOM tipinde yapay sinir ağlarını kullanarak Türkiye’nin ihracat yaptığı ülkelerin kümelenmesi üzerine bir çalışma [A study on the clustering of countries to which Turkey exports using SOM type artificial neural networks.]. Atatürk Üniversitesi İktisadi ve İdari Bilimler Dergisi, 20(2), 219-239.

Gorr, W. L., Nagin, D., & Szczypula, J. (1994). Comparative study of artificial neural network and statistical models for predicting student grade point averages. International Journal of Forecasting, 10(1), 17-34.

Hardgrave, B. C., Wilson, R. L., & Walstrom, K. A. (1994). Predicting graduate student success: A comparison of neural networks and traditional techniques. Computers & Operations Research, 21(3), 249-263.

Haykin, S. (1999). Neural networks: A comprehensive foundation. New Jersey: Prentice Hall International.

Ibrahim, Z., & Rusli, D. (2007, September). Predicting students’ academic performance: comparing artificial neural network, decision tree and linear regression. In 21st Annual SAS Malaysia Forum (pp. 1-6).

Kohonen, T. (2001). Self-organizing maps. Berlin: Springer.

Mullis, I. V. S., Martin, M. O., Foy, P., & Hooper, M. (2016). TIMSS 2015 International Results in Mathematics. Retrieved from Boston College, TIMSS & PIRLS International Study Center website: http://timssandpirls.bc.edu/timss2015/international-results/.

Mullis, I. V. S., Martin, M. O., Foy, P., Kelly, D. L., & Fishbein, B. (2020). TIMSS 2019 International Results in Mathematics and Science. Retrieved from Boston College, TIMSS & PIRLS International Study Center website: https://timssandpirls.bc.edu/timss2019/international-results/.

Öztemel, E. (2003). Yapay sinir ağları [Artificial neural networks]. İstanbul: Papatya Yayıncılık.

Saraç, T. (2005). Estimating software project duration using artificial neural networks [Unpublished doctoral dissertation]. Gazi University.

Shah, J. R., & Murtaza, M. B. (2000). A neural network based clustering procedure for bankruptcy prediction. American Business Review, 18(2).

SubbaNarasimha, P. N., Arinze, B., & Anandarajan, M. (2000). The predictive accuracy of artificial neural networks and multiple regression in the case of skewed data: Exploration of some issues. Expert systems with Applications, 19(2), 117-123.