Building a Rubric for Peer Review of Manuscripts in Psychological and Behavioral Sciences
Keywords:
Peer review, refereeing, research manuscripts, rubric, educational & psychological sciencesAbstract
The purpose of this research is to build a rubric that can be used in peer review of research manuscripts in educational and psychological sciences. This rubric is built according to the Helvoort model for performance tasks, and its validity in the construction phase was confirmed by opinions of referees and its reliability was calculated by interrater reliability. The results of the construction stages showed the consensus of studies, methodologies in educational and psychological sciences and review forms used in journals on (6) criteria, namely: introduction, methodology, results, discussion, conclusions and recommendations, references and documentation. The results of the evaluation of rubric by judgments showed the comprehensiveness of its criteria and its relevance to the assessment of manuscripts, assisting the reviewers in focusing on research components, shortening review time, and achieving interrater reliability. It makes the beneficiaries (editor, reviewer, and author) trust its effectiveness and use it in review. Although the rubric is based on best practices in reviewing research manuscripts in educational and psychological sciences, it may not be generalized to other fields. It also provides a general and important vision that should be discussed further in the future.
Downloads
References
Albarracín, D. (2015). Psychological Bulletin, 141(1), 1–5. https://doi.org/ 10.1037/bul0000007
Albarracín, D., Cuijpers, P., Eastwick, P. W., Johnson, B. T., Roisman, G. I., Sinatra, G. M., & Verhaeghen, P. (2018). Psychological Bulletin, 144(3), 223–226. https://doi.org/10.1037/bul0000147
Ali, P., & Watson, R. (2016). Peer review and the publication process. Nursing Open, 3(4), 193-202, 2016. https://doi.org/10.1002/nop2.51
AlRwaili, E. (2016). Method of (SCORING RUBRICS) in Performance Assessment and Its Impact in The Achievement of First Secondary Students in Mathematics. Dirasat: Educational Sciences, 43(5), 1903-1914.
Andrade, H. G. (2005). Teaching with Rubrics: The Good, the Bad, and the Ugly. College Teaching, 53(1), 27–31. https://doi.org/10.3200/CTCH.53.1.27-31
Arter, J. (2000). Rubrics, scoring guides, and performance criteria: Classroom tools for assessing and improving student learning. In The annual meeting of the American Educational Research Association, New Orleans, LA.
Baldwin, M. (2017). In referees we trust? Phys Today, 70(2)44. doi: 10.1063/PT.3.34 63
Bhattacherjee, A. (2012). Social Science Research: Principles, Methods, and Practices. Textbooks Collection, 3. http://scholarcommons.usf.edu/oa_ textbooks/3
Bordage, G., & Caelleigh, A.S. (2001). A tool for reviewers: Review criteria for research manuscripts. Academic Medicine,76(9), 904–951.doi:10.1097/ 00001888 -200 109000-00013
Brookhart, S. M. (2013). How to Create and Use Rubrics for Formative Assessment and Grading. Alexandria, VA: ASCD.
Brookhart, S., & Chen, F. (2015). The quality and effectiveness of descriptive rubrics. Educational Review, 67(3), 343-368. doi: 10.1080/00131911.2014. 929 565
Chubin, D., & Hackett, E. (1989). Peerless Science: Peer Review and U. S. Science Policy.
Cicchetti, D. (1991). The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation. Behavioral and Brain Sciences, 14(1), 119-135. doi: 10.1017/S0140525X00065675
Clement, L., Chauvot, J., Philipp, R., & Ambrose, R. (2003). A method for developing rubrics for research purposes. In N. A. Pateman, B. J. Dougherty, & J. T. Zilliox (Eds.), Proceedings of the 2003 joint meeting of PME and PMENA (Vol. 2, pp. 221–227). Honolulu: CRDG, College of Education, University of Hawaii.
Cockett, A., & Jackson, C. (2018). The use of assessment rubrics to enhance feedback in higher education: An integrative literature review. Nurse Education Today, 69, 8-13.
Cox, G.C., Morrison, J., & Brathwaite, B.H. (2015). The Rubric: An Assessment Tool to Guide Students and Markers. In The 1st International Conference on Higher Education Advances. Valencia, Spain. http://dx.doi.org/10.4995/HEAd 15.2015.414
Devine, E., & Frass, W. (2016). Peer review: A global view. Retrieved from: http://authorservices.tayl orandfrancis.com/peer-review-global-view/
Dickinson, P., & Adams, J. (2017). Values in evaluation–The use of rubrics. Evaluation and Program Planning, (65), 113-116. doi: 10.1016/j. evalprog plan.2017.07.005
Fraile, J., Panadero, E., & Pardo, R. (2017). Co-creating rubrics: The effects on self-regulated learning, self-efficacy and performance of establishing assessment criteria with students. Studies in Educational Evaluation, 53, 69-76. http://doi.org/10.1016/j.stueduc.2017.03.003
Gastel, B., & Day, R. E. (2016). How to write and publish a scientific paper. Santa Barbara: Greenwood.
Graue, E & Gallego, M. (2006). The Craft of Reviewing Manuscript for Journals: Building Skill and Learning Through the Process. In The American Educational Research Conference, Chicago, IL, April.
Gunning, T. G. (2006). Assessing and correcting reading and writing difficulties. Boston: Pearson Education Inc.
Helvoort, J. V. (2010). A scoring rubric for performance assessment of information literacy in Dutch Higher Education. Journal of Information Literacy, 4(1), 22-39. http://dx.doi.org/10.11645/4.1.1256
Horbach, S., & Halffman, W. W. (2018). The changing forms and expectations of peer review. Research Integrity and Peer Review, 8(3). doi:10.1186/s41073-018-0051-5
Jonathan, B., Harvey, C., & Hirshleifer, D. (2017). How to Write an Effective Referee Report and Improve the Scientific Review Process. Journal of Economic Perspectives, 31 (1): 231-44. doi: 10.1257/jep.31.1.231
Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130–144.
Katarina, U. (2014). Measuring essay assessment: Intra-rater and inter-rater reliability. Eurasian Journal of Educational Research, 57, 113-136.doi.org/10.14689/ejer.2014.57.2
Kerlinger, F. N. (1986). Foundations of Behavioral Research. New York: Holt, Rinehart and Winston.
Laming, D. (1991). Why is the reliability of peer review so low? Behavioral and Brain Sciences, 14(1), 154-156. doi:10.1017/S0140525X00065882
Leung, D., law, R., Kucukusta, D., & Guillet, B. (2014). How to review journal manuscripts: a lesson learnt from the world’s excellent reviewers. Tourism Management Perspectives, 10, 46–56. https://doi.org/10.1016/ j.tmp.2014.01.003
Lönngren, J., Adawi, T., & Svanström, M. (2019). Scaffolding strategies in a rubric-based intervention to promote engineering students’ ability to address wicked problems. European Journal of Engineering Education, 44(1-2), 196-221.
Marsh, H. W., Jayasinghe, U. W., & Bond, N. W. (2008). Improving the peer-review process for grant applications: Reliability, validity, bias, and generalizability. American Psychologist, 63(3), 160-168.http://dx.doi. org/ 10.1037/0003-066X.63.3.160
Martens, K. (2018). Rubrics in program evaluation. Evaluation Journal of Australasia, 18 (1), 21-44. https://doi.org/10.1177/ 1035719X17753961
McKercher, B., Law, R., Weber, K., Song, H., & Hsu, C. (2007). Why Referees Reject Manuscripts. Journal of Hospitality & Tourism Research, 31(4), 455–470. https://doi.org/10.1177/1096348007302355
McNair, R., Le Phuong, H. A., Cseri, L., & Szekely, G. (2019). Peer review of manuscripts: A valuable yet neglected educational tool for early-career researchers. Education Research International. http://dx.doi.org/10.1155/2019/1359362
Menéndez-Varela, JL., & Gregori-Giralt, E. (2018). The reliability and sources of error of using rubrics-based assessment for student projects. Assessment & Evaluation in Higher Education, 43(3), 488–99. doi: 10.1080/02602 938.2017.1360838
Mertler, C. A. (2001). Designing scoring rubrics for your classroom. Practical Assessment. Research & Evaluation, 7(25). Retrieved April 23, 2019 from http://edresearch.org/pare/getvn.asp?v=7&n=25.
Nguyen, V. M., Haddaway, N. R., Gutowsky, L. F., Wilson, A. D., Gallagher, A. J., Donaldson, M. R., & Cooke, S.T. (2015). How long is too long in contemporary peer review? Perspectives from authors publishing in conservation biology journals. PLoS ONE, 10(8), e0132557.
Nitko, A. J. (1996). Educational Assessment of Students. Englewood Cliffs, NJ: Prentice-Hall.
Panadero, E., & Jonsson, A. (2013). The use of scoring rubrics for formative assessment purposes revisited: A review. Educational Research Review, 9(0), 129-144. doi: http://dx.doi.org/10.1016/j.edurev.2013.01.002
Popham, J. (1997). What's Wrong - and What's Right - with Rubrics. Educational Leadership, 55 (2),72–75.
Prins, F., de Kleijn, R., & van Tartwijk, J. (2017). Students’ use of a rubric for research theses. Assessment & Evaluation in Higher Education, 42(1), 128-150. doi: 10.1080/02602938.2015.1085954
Provenzale, J. M., & Stanley, R. J. (2005). A Systematic Guide to Reviewing a Manuscript. AJR, 185, 1–7.
Raamkumar, A.S., Foo, S., & Pang, N. (2016). Survey on inadequate and omitted citations in manuscripts: a precursory study in identification of tasks for a literature review and manuscript writing assistive system. Information Research, 21(4). Retrieved from http://InformationR.net/ir/21-4/paper733.html (Archived by WebCite® at http://www.webcitation. org/6m5HZifOk).
Reddy, Y., & Andrade, H. (2010). A review of rubric use in higher education. Assessment & Evaluation in Higher Education, 35(4), 435-448, doi: 10.1080/026 02930902 86 2859
Rennie, D. (2003). Editorial peer review: Its development and rationale. Peer Review in Health Sciences. 1-13.
Rezaei, A., & Lovorn, M. (2010). Reliability and validity of rubrics for assessment through writing. Assessing Writing, 15 (1), 18-39. https://doi.org/10.1016/ j.asw.2010.01.003
Rockwell, S. (2005). Ethics of Peer Review: A Guide for Manuscript Reviewers. Retrieved 16-7-2012 from: http//medicine. yale.edu/therapeutic radiology/ Images/ Ethical_ Issues_in_Peer_Review_tcm307- 34211.pdf.
Rosenfeld, R. M. (2010). How to review journal manuscripts. Otolaryngology-Head and Neck Surgery, 142, 472-486.
Shamari, E. (2018). Evaluating the Status of Possession and Practice of Basic Grade’s Teachers of Alternative Assessment Strategies in Schools of Hail Region. Dirasat: Educational Sciences, 45(7), 537-551.
Suskie, L. (2009). Assessing student learning: A common sense guide. San Francisco: Jossey-Bass.
Teddlie, C., & Tashakkori, A. (2009). Foundations of Mixed Methods Research: Integrating Quantitative and Qualitative Approaches in the Social and Behavioral Sciences. London: Sage.
Wicherts, J. M. (2016). Peer Review Quality and Transparency of the Peer-Review Process in Open Access and Subscription Journals. PLOS ONE, 11(1), e0147913. https://doi.org/10.1371/journal.pone.0147913
Wood, M., Roberts, M., & Howell, B. (2004). The Reliability of Peer Reviews of Papers on Information Systems. Journal of Information Science, 30(1), 2–11. htt ps://doi.org/10.1177/0165551504041673
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
