Measurement Invariance in the Graduate Record Examination: A Language-Based Analysis

Abd-Alelah Naji  Al-Khawaldeh; Yousef Mohammad  Sawalmeh

doi:10.35516/edu.v50i2.305

Authors

Abd-Alelah Naji Al-Khawaldeh Ministry of Education, Amman, Jordan https://orcid.org/0009-0005-5957-8098
Yousef Mohammad Sawalmeh Department of Counseling and Educational Psychology, Faculty of Education, Yarmouk University, Irbid, Jordan https://orcid.org/0009-0005-5957-8098

DOI:

https://doi.org/10.35516/edu.v50i2.305

Keywords:

Graduate record examination (gre) test, quantitative reasoning, measurement invariance, confirmatory factor analysis, multi-group confirmatory factor analysis

Abstract

Objectives: The study aimed to examine the measurement invariance in the GRE test for quantitative reasoning administered in both Arabic and English.

Method: A cluster random sample of 541 male and female MA students was selected from the University of Jordan, Yarmouk University, and Mutah University. Of these, 273 students (both male and female) took the test in Arabic, while 268 students (both male and female) took it in English. The data were analyzed using both confirmatory factor analysis (CFA) and multi-group conformity factor analysis (MGCFA) through the statistical program AMOS 24. The fit of the data to the factorial model was assessed using four indicators: χ²/df, CFI, RMSEA, SRMR.

Results: The results indicated that the data fit the four-factor model (arithmetic, algebra, geometry, and data analysis) for both the Arabic and English versions of the exam. However, the results also showed that while configural invariance was achieved, metric and scalar invariances were not achieved based on the language of the exam.

Conclusions: The quantitative reasoning exam presented in both English and Arabic versions had the same structure as assumed in the factorial model. However, the loadings on the factors and the intersection between the factors differed between the two versions. Preference was given to the Arabic version of the exam because it demonstrated a good fit with the factorial structure model, and proficiency in the English language did not significantly influence performance. Therefore, using the Arabic version with students whose mother tongue is Arabic allows for fair and unbiased comparisons between them.

Downloads

Download data is not yet available.

References

Piotr, S. (2015). American Educational Research Association, American Psychological Association, National Council on Measurement in Education, Standards for educational and psychological testing, American Educational Research Association, Washington 2014, 240 s. Kwartalnik Pedagogiczny, (60 (4 (238), 201-203.‏

Ariely, G., & Davidov, E. (2012). Assessment of measurement equivalence with cross-national and longitudinal surveys in political science. European Political Science, 11(3), 363-377.‏

Broer, M. (2005). Ensuring the fairness of GRE writing prompts: Assessing differential difficulty. ETS Research Report Series, (1), i-41.‏

Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. Testing structural equation models, 136-162.‏

Butcher, J. N. (2006). MMPI-2: A practitioner's guide. American Psychological Association.

Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement in variance. Structural equation modeling: a multidisciplinary journal, 14(3), 464-504.

Cheung, G. W. (2012). Mainstreaming culture in psychology. American Psychologist, 67, 721-730.

Davidov, E., Meuleman, B., Crecyuch, J., Schmidt, P., & Billiet, J. (2014). Measurement equivalence in cross- national research. Annual Review of Sociology, 40(1), 55-75.

Educational Testing Service. (2013). A snapshot of the individual who took the GRE revised General Test. Educational Testing Service.

Educational Testing Service. (2014). MBA programs that accept the GRE revised General. Educational Testing Service. http://www.ets.org/gre/revised_general/about/mba/programs?WT.ac=rx42

Educational Testing Service. (2015). ETS standards for quality and fairness. Educational Testing Service. http:// www.ets.org/gre/guide

Educational Testing Service. (2016). A Snapshot of the Individuals Who Took the GRE General Test. Educational Testing Service. http:// www.ets.org/gre/pdf/snapshot_test_taker_data_2016.pdf

Fischer, R. & Poortinga, H. (2018). Addressing methodical challenges in culture comparative research. Journal of Cross-Cultural Psychology, 49(5), 691-712.

Hair, J. F., Anderson, R. E., Tatham, R. L., & Black, W. C. (1995). Multivariate Data Analysis with Reading. (4th ed.). Plentice-Hall.

He, Q., Anwyll, S., Glanville, M., &Opposs, D. (2014). An investigation of measurement invariance of the Key Stage 2 National Curriculum science sampling test in England. Research Papers in Education, 29(2), 211-239.

Horn, J. & Mcardle, J. (1992). Apractical and theoretical guide to measurement invariance in aging research. Experimental Aging Research, 18(3), 117-144.

Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural equation modeling: a multidisciplinary journal, 6(1), 1-55.

International Test Commission. (2017). The ITC guidelines for translating and adapting tests.‏ http:// www.InTestCom.org

Kim, H., & Sasaki, J. Y. (2017). Intercultural similarities and differences in personality development. In Personality Development Across the Lifespan (pp. 419-434). Academic Press.‏

Kline, R. B. (2015). Principles and practice of structural equation modeling. Guilford Press.

Kuncel, N. R., Wee, S., Serafin, L., & Hezlett, S. A. (2010). The validity of the Graduate Record Examination for master’s and doctoral programs: A meta-analytic investigation. Educational and Psychological Measurement, 70(2), 340-352.

Marlene, E., Henerson, L. M. & Carol, T. G. (1978). How to Measure Attitudes. (7th ed.). Sage Publications.

Marsh, H. W., & Hocevar, D. (1985). Application of confirmatory factor analysis to the study of self-concept: First-and higher order factor models and their in variance across groups. Psychological bulletin, 97(3), 562.

Reeves, J. (2002). Aptitude assessment for career and educational guidance. The Work Suite.

Sekercioglu, G., & Kogar, H. (2018). The Examination of Measurement Invariance and Differential Item Functioning of PISA 2015 Cognitive Tests in Terms of the Commonly Used Languages. Novitas-ROYAL (Research on Youth and Language), 12(2), 152-172.‏

Sinkovics, R. R., Henseler, J., Ringle., C.M., & Sarstedt, M. (2016). Testing measurement invariance of composites using partial least squares. International marketing review.

Sousa, K., West, S., Moser, S., Harris, J. & Cook, S. (2012). Establishing measurement invariance: English and Spanish Pediatric Asthma Quality of Life Questionnaire. Nurse Research, 61(3), 171-180.

Steenkamp, J. B. E., & Baumgartner, H. (1998). Assessing measurement invariance in cross-national consumer research. Journal of consumer research, 25(1), 78-90. ‏

Tsaousis, I. (2015). Factorial invariance and latent mean differences of scores on SAAT across gender. http://www.qiyas.sa/MAndAssesment/Researches/Pages/Researches.aspx

Wendler, C., Bridgeman, B., & Ezzo, C. (2014). The Research Foundation for the GRE revised General Test: A compendium of studies. Educational Testing Service. http://www.ets.org/gre/compendium

West, S. G., Finch, J. F., & Curran, P. J. (1995). Structural equation models with nonnormal variables: Problems and remedies.‏

Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational research method, 3(1), 4-70.

Van de Vijver, F., & Tanzer, N. K. (2004). Bias and equivalence in cross-cultural assessment: An overview. European Review of Applied Psychology, 54(2), 119-135.‏

Varsity Tutors. (2019). GRE Study Guide: Lessons, Strategies, and Diagnostic Tests. Varsity Tutors LLC. https://www.google.com/url?sa=t&scource=web&rct=j&url=https://varsitytutors.com/varsity-tutors-gre-book-first-edition.pdf&ved=2ahUKEwiatO-5ocj0AhU77uAKHTgqCLsQFnoECDIQAQ&usg=AOvVaw2BBD-opn4tBmwDAb4oQPGE

Veenhoven, R. (1996). Developments in satisfaction-research. Social Indicators Research, 37(1), 1-46.