Journal of Language, Literature, and Educational Research

an Open Access Journal

Exploring Rater Perceptions in Argumentative Writing Assessment: A Qualitative Case Study of Experience, Nativeness, and Rhetorical Variability

Gül KARAAGAC ZAN
Nigde Omer Halisdemir University
Türkiye
glkaraagac@gmail.com
Share

  • Purpose of the study: This study aims to explore how linguistic background and teaching experience influence evaluation criteria, rhetorical interpretation, and scoring decisions during assessment of argumentative writing. It also examines how raters perceive rhetorical variation in essays written by students from different cultural backgrounds.

    Methodology: This study employed an exploratory qualitative case study design using Concurrent Think-Aloud Protocols (TAP). Participants consisted of four English as a Foreign Language instructors with different linguistic backgrounds and teaching experience. Data were collected through audio-recorded assessment sessions and analyzed using structural coding and pattern coding to identify recurring evaluation themes and scoring behaviors.

    Main Findings: Findings revealed that native English-speaking raters emphasized coherence, audience awareness, and rhetorical organization, whereas non-native English-speaking raters focused primarily on grammatical accuracy and content. Rhetorical variation influenced evaluation decisions, producing differences in assigned scores across participants. Teaching experience and linguistic background contributed to variation in assessment criteria, rhetorical interpretation, and scoring practices during evaluation of argumentative writing.

    Novelty/Originality of this study: This study integrates linguistic nativeness, teaching experience, and rhetorical variation within one qualitative investigation using Think-Aloud Protocols. Unlike previous studies emphasizing score comparison, it explores cognitive processes underlying writing assessment decisions. Findings provide deeper understanding of how rater characteristics shape evaluation practices and support development of more inclusive and reliable writing assessment in multilingual educational contexts.

  • How to cite

    Exploring Rater Perceptions in Argumentative Writing Assessment: A Qualitative Case Study of Experience, Nativeness, and Rhetorical Variability. (2026). Journal of Language, Literature, and Educational Research, 3(1), 153-159. https://doi.org/10.37251/jolle.v3i1.2039

    More Citation Formats

  • 4
    Abstract views
    0
    Downloads

    Metrics — Badges

    1. [1] M. S. Ibarra-sáiz, G. Rodríguez-gómez, D. Boud, and G. Rodr, “The quality of assessment tasks as a determinant of learning,” Assess. Eval. High. Educ., vol. 46, no. 6, pp. 943–955, 2021, doi: 10.1080/02602938.2020.1828268.
    2. [2] A. Zarbafian, G. Abbasian, and A. Mohseni, “Synthesizing transformative education with dynamic assessment in developing efl learners’ productive skills,” Cogent Arts Humanit., vol. 7, no. 1, pp. 1–26, 2020, doi: 10.1080/23311983.2020.1750840.
    3. [3] S. E. Kassab et al., “Construct validity of an instrument for assessment of reflective writing based portfolios of medical students,” Adv. Med. Educ. Pract., vol. 1, no. 1, pp. 397–404, 2020, doi: 10.2147/AMEP.S256338.
    4. [4] J. Golzar, S. E. Momenzadeh, and M. A. Miri, “Afghan english teachers and students perceptions of formative assessment: A comparative analysis,” Cogent Educ., vol. 9, no. 1, pp. 1–19, 2022, doi: 10.1080/2331186X.2022.2107297.
    5. [5] L. Allal, “Involving primary school students in the co construction of formative assessment in support of writing,” Assess. Educ. Princ. Policy Pract., vol. 28, no. 5–6, pp. 584–601, 2021, doi: 10.1080/0969594X.2021.1951164.
    6. [6] A. Fletcher, “Australia’s national assessment programme rubrics: An impetus for self-assessment?,” Educ. Res., vol. 63, no. 1, pp. 43–64, 2021, doi: 10.1080/00131881.2020.1850207.
    7. [7] G. B. Skar and A. Huebner, “Predicting first grade students’ writing proficienc,” Assess. Educ. Princ. Policy Pract., vol. 29, no. 2, pp. 219–237, 2022, doi: 10.1080/0969594X.2022.2057424.
    8. [8] A. Reinhart et al., “Think Aloud Interviews: A tool for exploring student statistical reasoning,” J. Stat. Data Sci. Educ., vol. 30, no. 2, pp. 1–35, 2022, doi: 10.1080/26939169.2022.2063209.
    9. [9] T. G. Kormelink and T. G. Kormelink, “Seeing, thinking, feeling: A critical reflection on interview-based methods for studying news use,” Journal. Stud., vol. 21, no. 7, pp. 863–878, 2020, doi: 10.1080/1461670X.2020.1716829.
    10. [10] C. Hogarth, C. Matthiesen, J. Bakken, and C. Matthiesen, “Advancing citizenship through language arts education: Conceptions of rhetoric in Scandinavian national curricula,” J. Curric. Stud., vol. 54, no. 4, pp. 559–575, 2022, doi: 10.1080/00220272.2021.1989050.
    11. [11] Y. Khajavi and A. Rasti, “A discourse analytic investigation into politicians’ use of rhetorical and persuasive strategies: The case of us election speeches,” Cogent Arts Humanit., vol. 7, no. 1, pp. 1–15, 2020, doi: 10.1080/23311983.2020.1740051.
    12. [12] I. K. Ellasante, “Radical sovereignty, rhetorical borders, and the everyday decolonial praxis of indigenous peoplehood and two spirit reclamation,” Ethn. Racial Stud., vol. 44, no. 9, pp. 1507–1526, 2021, doi: 10.1080/01419870.2021.1906437.
    13. [13] J. Schaafsma, M. Zoodsma, and T. Sagherian-dickey, “Closing chapters of the past? Rhetorical strategies in political apologies for human rights violations across the world,” J. Hum. Rights, vol. 20, no. 5, pp. 582–597, 2021, doi: 10.1080/14754835.2021.1977919.
    14. [14] M. Bayat, S. K. Banihashem, and O. Noroozi, “The effects of collaborative reasoning strategies on improving primary school students’ argumentative decision making skills,” J. Educ. Res., vol. 115, no. 6, pp. 349–358, 2022, doi: 10.1080/00220671.2022.2155602.
    15. [15] E. Bjerkholt, T. Ørbæk, T. Kindeberg, and E. Bjerkholt, “An outline of a pedagogical rhetorical interactional methodology researching teachers responsibility for supporting students desire to learn as well as their actual learning,” Teach. High. Educ., vol. 28, no. 2, pp. 238–253, 2023, doi: 10.1080/13562517.2020.1792876.
    16. [16] A. Leander, “Militarization matters: Rhetorical resonances and market militarism,” Crit. Mil. Stud., vol. 10, no. 2, pp. 147–170, 2024, doi: 10.1080/23337486.2022.2081300.
    17. [17] A. A. Wornyo, “Rhetorical strategy preferences in newspaper editorials,” Cogent Arts Humanit., vol. 9, no. 1, pp. 1–16, 2022, doi: 10.1080/23311983.2022.2146636.
    18. [18] M. Marttunen, T. Salminen, and J. Utriainen, “Student evaluations of the credibility and argumentation of online sources,” J. Educ. Res., vol. 114, no. 3, pp. 294–305, 2021, doi: 10.1080/00220671.2021.1929052.
    19. [19] S. Latifi, O. Noroozi, and E. Talaee, “Worked example or scripting? Fostering students’ online argumentative peer feedback, essay writing and learning,” Interact. Learn. Environ., vol. 31, no. 2, pp. 655–669, 2023, doi: 10.1080/10494820.2020.1799032.
    20. [20] M. Firoozjahantigh, E. F. Alamdari, and A. Marzban, “Investigating the effect of process based instruction of writing on the ielts writing task two performance of iranian efl learners: Focusing on hedging & boosting,” Cogent Educ., vol. 8, no. 1, pp. 1–15, 2021, doi: 10.1080/2331186X.2021.1881202.
    21. [21] K. Oparinde, M. Rapeane-mathonsi, and G. Mheta, “Exploring manipulative rhetorical choices in Nigerian political speeches,” Cogent Arts Humanit., vol. 8, no. 1, pp. 1–13, 2021, doi: 10.1080/23311983.2020.1864913.
    22. [22] M. Zare, M. S. Bagheri, and F. Sadighi, “An investigation of the linguistic complexity of ielts writing topics based on the levels of discourse representation and the degree of meaning coding,” Cogent Educ., vol. 8, no. 1, pp. 1–20, 2021, doi: 10.1080/2331186X.2020.1868235.
    23. [23] D. Tillery and E. F. Bloomfield, “Hyperrationality and rhetorical constellations in digital climate change denial: A multi methodological analysis of the discourse of watts up with that,” Tech. Commun. Q., vol. 31, no. 4, pp. 356–373, 2022, doi: 10.1080/10572252.2021.2019317.
    24. [24] L. Lepistö, S. Lepistö, K. Kallio, L. Lepistö, and S. Lepistö, “Unpacking data analytics: Rhetorical analysis,” Technol. Anal. Strateg. Manag., vol. 35, no. 8, pp. 993–1004, 2023, doi: 10.1080/09537325.2021.1990876.
    25. [25] Y. Hashlamon, “Rhetoricity at the end of history: Defining rhetorical debility under neoliberal colonialism,” Rhetor. Soc. Q., vol. 52, no. 1, pp. 18–31, 2022, doi: 10.1080/02773945.2021.1990378.
    26. [26] T. Bruhn, J. Doona, T. Bruhn, and J. Doona, “Serious grappling with satire: Rhetorical genre affordances and invitations to participation in public controversy,” J. Eur. Inst. Commun. Cult., vol. 29, no. 3, pp. 284–300, 2022, doi: 10.1080/13183222.2021.1983303.
    27. [27] M. Bahmani, A. Chalak, and H. H. Tabrizi, “The effect of evaluative language on high- and low-graded post-graduate students’ academic writing ability across gender,” Cogent Educ., vol. 8, no. 1, pp. 1–15, 2021, doi: 10.1080/2331186X.2021.1905229.
    28. [28] N. Roderick, “Form from form: The case for exaptation in rhetorical genre evolution,” Q. J. Speech, vol. 107, no. 4, pp. 398–417, 2021, doi: 10.1080/00335630.2021.1983193.
    29. [29] N. T. Kerman, O. Noroozi, S. K. Banihashem, M. Karami, and H. J. A. Biemans, “Online peer feedback patterns of success and failure in argumentative essay writing,” Interact. Learn. Environ., vol. 32, no. 1, pp. 614–626, 2024, doi: 10.1080/10494820.2022.2093914.
    30. [30] C. Vögelin et al., “Organisational quality of esl argumentative essays and its influence on pre service teachers’ judgments,” Cogent Educ., vol. 7, no. 1, pp. 1–21, 2020, doi: 10.1080/2331186X.2020.1760188.