Comprehensibility in Language Testing
Parvaneh Tavakoli [+–]
University of Reading
View Website
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
Sheryl Cooke [+–]
British Council
Sheryl Cooke is Director of the British Council’s East Asia Assessment Solutions Team, leading on language assessment projects and stakeholder engagement across the region, including needs analyses, language assessment literacy training, test development, post-test services and teacher support. Sheryl has 20 years’ experience in language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on automated assessment of spoken English and the potential implications for English as a Lingua Franca, and she has presented at leading regional and international conferences on this topic. Research interests include assessment of speaking, particularly accent and pronunciation, the use of new technologies in language testing and the ethics of language assessment in the global context.
Comprehensibility, or the extent to which a listener understands L2 speakers’ spoken output/performance, is a fundamentally important aspect of communication, and a construct the language testing discipline has been interested in for a few decades. Comprehensibility is considered central to successful communication (Munro & Derwing, 1995, 2007) as it highlights the listener’s perceptions of the level of difficulty involved in understanding a speaker. Despite the crucial role it plays in communication and the contribution it makes towards the assessment of spoken language ability, comprehensibility has mostly been assessed holistically in language tests, if at all, with little reference to an analytic framework in which comprehensibility can be objectively analysed and reliably assessed. The lack of such an analytical framework for defining and analysing comprehensibility has motivated us to propose the current manuscript as a research-oriented and evidence-based monograph that provides an analytic framework for understanding, defining and assessing comprehensibility. In this monograph, we argue that conceptualising comprehensibility as a multidimensional construct and adopting a broader perspective to understanding and analysing it for communication purposes would benefit the fields of second language assessment and second language acquisition.
Series: British Council Monographs on Modern Language Testing
Table of Contents
Chapter 1
Introduction [+–]
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
British Council
Sheryl Cooke is Director of the British Council’s East Asia Assessment Solutions Team, leading on language assessment projects and stakeholder engagement across the region, including needs analyses, language assessment literacy training, test development, post-test services and teacher support. Sheryl has 20 years’ experience in language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on automated assessment of spoken English and the potential implications for English as a Lingua Franca, and she has presented at leading regional and international conferences on this topic. Research interests include assessment of speaking, particularly accent and pronunciation, the use of new technologies in language testing and the ethics of language assessment in the global context.
This chapter provides an overview of the scope and aims of the book and presents a summary of current approaches to the construct of comprehensibility and the related concepts of intelligibility and accentedness (Levis, 2005; Munro & Derwing, 2011; Isaacs, 2014). A brief synopsis of previous research articles is followed by consideration of the difficulties inherent in defining, teaching and measuring a concept that is characterised by complexity and further confounded by broader socio-linguistic factors such as the role of the listener and the existence of multiple variations of languages (Kachru, 1985; Lindemann, 2017; Harding & Mcnamara, 2017) . The outline of a framework for analysing comprehensibility from a micro- to macro-level is presented. Following chapters focus on describing and providing a rationale for this new approach.
Chapter 2
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
British Council
Sheryl Cooke is Director of the British Council’s East Asia Assessment Solutions Team, leading on language assessment projects and stakeholder engagement across the region, including needs analyses, language assessment literacy training, test development, post-test services and teacher support. Sheryl has 20 years’ experience in language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on automated assessment of spoken English and the potential implications for English as a Lingua Franca, and she has presented at leading regional and international conferences on this topic. Research interests include assessment of speaking, particularly accent and pronunciation, the use of new technologies in language testing and the ethics of language assessment in the global context.
This chapter focuses on the more traditional scope of the comprehensibility/intelligibility construct and takes an analytical approach to examining evidence related to phonemic and prosodic linguistic components affecting comprehensibility of spoken language. An overview of the features that have been identified by previous studies is presented and categorised (e.g. Isaacs & Harding, 2017; Isaacs & Trofimovich, 2012), including those occurring at phoneme level (articulation), word level (segmental units, word stress) and sentence level (intonation, pitch, fluency). The features are then examined from three key perspectives: a) reported impact on comprehensibility b) how the features are reflected in rating scales c) whether the features are L1-dependent
Chapter 3
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
British Council
Sheryl Cooke is Director of the British Council’s East Asia Assessment Solutions Team, leading on language assessment projects and stakeholder engagement across the region, including needs analyses, language assessment literacy training, test development, post-test services and teacher support. Sheryl has 20 years’ experience in language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on automated assessment of spoken English and the potential implications for English as a Lingua Franca, and she has presented at leading regional and international conferences on this topic. Research interests include assessment of speaking, particularly accent and pronunciation, the use of new technologies in language testing and the ethics of language assessment in the global context.
Spoken texts and the features associated with longer turns of monologic speech are the focus of this chapter. These typically include monologues such as presentations or speeches, narration and extended turns that might include characteristics of argumentation or description. Potentially impactful elements identified in the literature (e.g. Isaacs & Trofimovich, 2012) and in research are examined and, as in chapter 2, analysed from three perspectives: a) reported impact on comprehensibility b) how the features are reflected in rating scales c) impact of L1 discourse conventions on comprehensibility
Chapter 4
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
British Council
Sheryl Cooke is Director of the British Council’s East Asia Assessment Solutions Team, leading on language assessment projects and stakeholder engagement across the region, including needs analyses, language assessment literacy training, test development, post-test services and teacher support. Sheryl has 20 years’ experience in language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on automated assessment of spoken English and the potential implications for English as a Lingua Franca, and she has presented at leading regional and international conferences on this topic. Research interests include assessment of speaking, particularly accent and pronunciation, the use of new technologies in language testing and the ethics of language assessment in the global context.
This chapter focuses on the pragmatic aspect of comprehensibility, i.e., when the speaker’s sounds and words are clear, but their intended meaning is not because of their limited pragmatic knowledge. The discussions of this chapter will include the speaker’s perspective and highlights the dynamic nature of comprehensibility. After defining comprehensibility from a communication and intercultural perspective, the chapter will discuss the key aspects of pragmatic knowledge that affect comprehensibility. Summarising research in this area (e.g., Purpura, 2004; Rover, 2011, Taguchi, 2005, 2007, 2012) we will discuss issues such as use of formulaic expressions, implicatures and indirect speech acts as some potential areas in which comprehensibility issues arise. We will also focus on cultural norms and differences in pragmatic aspects of language use, e.g., politeness and backchannelling, that affect comprehensibility. More importantly, we will argue that to have a full understanding of the effects of pragmatic knowledge on comprehensibility, it is necessary to examine the speaker’s participation in extended discourse during both monologic and dialogic types of performance (Tavakoli, 2016; Tavakoli, 2018).
Chapter 5
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
British Council
Sheryl Cooke is Director of the British Council’s East Asia Assessment Solutions Team, leading on language assessment projects and stakeholder engagement across the region, including needs analyses, language assessment literacy training, test development, post-test services and teacher support. Sheryl has 20 years’ experience in language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on automated assessment of spoken English and the potential implications for English as a Lingua Franca, and she has presented at leading regional and international conferences on this topic. Research interests include assessment of speaking, particularly accent and pronunciation, the use of new technologies in language testing and the ethics of language assessment in the global context.
The chapter’s main aim is to discuss the relationship between fluency and comprehensibility. After defining the construct of fluency in terms of cognitive, perceived and utterance fluency (Segalowitz, 2010; Tavakoli & Wright, 2020) and measureable features of speed, breakdown and repair fluency (Skehan, 2003, Tavakoli, et al., 2020), the chapter discusses in what ways these features of speech affect comprehensibility. More specifically, the chapter will discuss the effects of fluency on listeners and their judgements. By providing examples of how these different features of fluency are represented in rating descriptors of different language tests, the chapter will argue that the relationship between fluency and comprehensibility is not clearly represented in rating descriptors and rating materials.
Chapter 6
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
Technological advances are having a significant impact on language assessment in general and this chapter examines the opportunities machines present to better measure comprehensibility, as well as the threats that constrained tasks demanded by automated assessment might present to the construct. An overview of current models for automated rating of speech is presented (e.g. Xi, et al. 2008; Van Moere & Downey, 2016; Isaacs, 2014) and the approaches are examined from the perspective of the comprehensibility framework presented in the book, drawing on the features extrapolated in each of the preceding chapters to examine how technology does and does not offer solutions for automated assessment and feedback for learning on particular components of the comprehensibility inventory.
Chapter 7
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
British Council
Sheryl Cooke is Director of the British Council’s East Asia Assessment Solutions Team, leading on language assessment projects and stakeholder engagement across the region, including needs analyses, language assessment literacy training, test development, post-test services and teacher support. Sheryl has 20 years’ experience in language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on automated assessment of spoken English and the potential implications for English as a Lingua Franca, and she has presented at leading regional and international conferences on this topic. Research interests include assessment of speaking, particularly accent and pronunciation, the use of new technologies in language testing and the ethics of language assessment in the global context.
This chapter provides a teaching perspective to the discussion of comprehensibility. Highlighting the importance of teaching the concept to L2 teachers, it will raise some key questions about the extent to which L2 teaching curricula focus on comprehensibility, or whether comprehensibility is actually taught and practiced in L2 classrooms. Analysing some national curricula for English language teaching, we will examine the focus on comprehensibility in these documents and strengthen our arguments by research in this area. We will provide some examples of what L2 teachers can do to highlight the role of comprehensibility.
Chapter 8
Concluding Remarks [+–]
University of Reading
Parvaneh Tavakoli is a Professor of Applied Linguistics at the University of Reading. Parvaneh’s main research interest lies in the interface of second language acquisition, language teaching, and language testing. Parvaneh has led several international research projects investigating the effects of task and task design on performance, acquisition, assessment and policy in different contexts. Her research has been published in prestigious peer-reviewed journals and books.
British Council
Sheryl Cooke is Director of the British Council’s East Asia Assessment Solutions Team, leading on language assessment projects and stakeholder engagement across the region, including needs analyses, language assessment literacy training, test development, post-test services and teacher support. Sheryl has 20 years’ experience in language assessment and her qualifications include an MA Language Testing (Lancaster University) and an MA Linguistics (SOAS). She is currently a PhD candidate at the University of Jyväskylä (Finland), focusing on automated assessment of spoken English and the potential implications for English as a Lingua Franca, and she has presented at leading regional and international conferences on this topic. Research interests include assessment of speaking, particularly accent and pronunciation, the use of new technologies in language testing and the ethics of language assessment in the global context.
In this chapter, we will provide an overview of issues related to comprehensibility and its key features that are central to successful communication. This chapter will also draw together the analytical framework discussed in the monograph to demonstrate the key aspects of the framework that benefit different language professionals. We highlight the contexts that potentially influence comprehensibility in significant ways and highlight the dynamic relationship between speakers and listeners. We will argue that adopting the broader perspective we have suggested will enable language professionals to have a more objective assessment of comprehensibility.
ISBN-13 (Hardback)
9780000000000
Price (Hardback)
£75.00 / $100.00
ISBN-13 (Paperback)
9780000000000
Price (Paperback)
£24.95 / $32.00
ISBN (eBook)
9780000000000
Price (eBook)
Individual
£24.95 / $32.00
Institutional
£75.00 / $100.00
£24.95 / $32.00
Institutional
£75.00 / $100.00
Publication
01/05/2023
Pages
200
Size
234 x 156mm
Readership
scholars and language teachers
