IULC Indiana University Linguistics Club

Investigating the rater bias patterns in reading-to-write tasks using FACETS

Time: 02:30pm - 04:00pm 

Place: Ballantine Hall 205

Sunyoung Shin

Reading-to-write tasks have become more common recently in university settings as more valid and authentic means to assess academic writing skills than a traditional, impromptu composition test (Plakans, 2009; Weigle, 2004). However, to date, little research has been conducted on the reliability of ratings in the assessment of reading-to-write tasks particularly in terms of rater behavior in relation to test takers and scoring domains. In this vein, this paper aims to explore rater severity and consistency, and scoring domain difficulty and consistency along with rater bias patterns towards examinee levels and domains in the reading-to-write tasks using FACETS. In the study, six experienced raters scored 83 ESL learners' written responses to two texts about the same topic, but with different points of view, using analytic rating rubrics comprising five domains: 1) Recognition; 2) Organization; 3) Development; 4) Language Use; and 5) Text Engagement. Findings revealed that raters were not equally severe despite their high inter-rater reliability, and "Recognition" category was found to be the most harshly rated compared to other scoring domains. Additionally, raters tended to score more harshly high-level test takers and more leniently low-level ones, which is in line with the findings of previous research on rater bias patterns in independent writing tasks (Kondo-Brown, 2002; Schaefer, 2008). The analysis also showed that reading-related domains, such as "Recognition" and "Text Engagement", are significantly either harshly or leniently scored by different raters. This study thus provides an insight into analytic rating rubric development and rater training for the reading-to-write tasks.

In category: Second language acquisition