Tradicionalmente, los profesores de idiomas hemos evaluado la competencia léxica a través de diversos métodos que incluyen actividades de traducción, textos incompletos, reformulación del mensaje original y ejercicios de redacción entre otros; todos ellos, sin embargo, se centran en la evaluación del vocabulario fuera de contexto como complemento de la gramática. El presente artículo pretende ofrecer una amplia perspectiva sobre la evaluación del componente léxico en contexto a través de la expresión oral y escrita. Para ello, analizaremos las rúbricas que los profesionales de las Escuelas Oficiales de Idiomas de Madrid usan en la actualidad.

Traditionally, language teachers have been compelled to assess learners’ awareness of vocabulary by a multiplicity of methods ranging from translation activities, cloze tests, and rephrasing exercises to writing whole sentences which were aimed at evaluating vocabulary mastery out of context as to complement grammar structures. The aim of this article is to offer a deep look at how language teachers can assess vocabulary in context by the use of rubrics for productive skills- writing and speaking; for that purpose, the rubrics currently used by EOI professionals for Certificate Exams in Comunidad de Madrid will be analyzed as a case study.

1. ASSESSING VOCABULARY IN CONTEXT

Due to the marked changes in the goals of language education programs, language teachers have moved away from the goal of accuracy towards a focus on fluency andcommunicative effectiveness. Consequently, the teaching of speaking and writing skills have become increasingly important. As it is stressed in the Common European Framework, communicative competence entails firstly, grammatical or linguistic competence, secondly, sociolinguistic competence- which addresses how utterances are appropriately produced and understood in different sociolinguistic contexts- discourse competence or the ability to sustain coherent discourse with another speaker and finally, strategic competence- the means by which learners deal with potential breakdowns in communication. Thus, our task as language teachers is to try and evaluate language holistically rather than in discrete areas. In terms of oral production, it refers to the evaluation of the spoken input around the message itself or tasks related to functional units addressed throughout the course contents such as expressing opinion, comparison, contrast, narrative, description, process and short question-answer. In terms of written production it deals with the mechanisms the user has in order to get the reader’s attention and achieve a concrete communicative purpose such as applying for a job (writing a cover letter) or organizing a baby shower by writing a short invitation via email. Communicative ability therefore is not something which can be broken down into separate categories such as grammar, vocabulary, pronunciation and fluency, as it has been traditionally done but luckily, a new era is upon us in which a holistic view of language teaching and learning is being recognized where vocabulary is seen as respectable and prominent.

Regarding the need to assess vocabulary in context, Nagy and Scott (2000, p.575) mention interrelatedness- the knowledge of a given word which depends on the knowledge of other words- as one of the factors of word knowledge that should be taken into account when assessing vocabulary. Assessment of word knowledge is then not an easy task as it entails considering words as open entities in that they form part of larger discourse and consequently, they should be tested as such. Although lexical knowledge has traditionally been assessed in Reading Comprehension, it is time for us teachers to select those texts that fulfill our goals in terms of what Read (2000: 9) calls the Three Dimensions of Vocabulary Tests as embedded, comprehensive and context-dependent, which we will use later on to comment the rubrics we aim to analyze.

Apart from that, assessing lexical competence in context also benefits from the use of authentic materials. The current use of corpora- the latest addition to the resources available for vocabulary input- makes it particularly useful for providing attested examples of language in use (an approach which might have been missing by traditional practitioners), as well as frequency and collocational information that we can find, namely in concordancing and key word programs. The use of large collections or databases of language incorporating stretches of discourse has revolutionized the way we view language. According to the CEFR, authentic texts foster real acquisition of vocabulary in that they provide salient tokens of the vocabulary we want to present to students as in real-life situations, not simulated ones, depending on students’ needs and characteristics of the syllabus in particular. Vocabulary instruction is much more effective when it is situated in a meaningful context, embedded in authentic (or semi-authentic) discourse, and motivated by getting learners to achieve a goal or complete an interesting task. Thus, the texts we use in order to evaluate learners’ competences should be part or contain part of authentic materials whenever possible.

​2. THE ROLE OF VOCABULARY ACCORDING TO THE COMMON EUROPEAN FRAMEWORK

The Framework highlights the role of vocabulary as one of the essential components in linguistic competence along with phonetics and syntax to name just a few:

Linguistic competence includes lexical, phonological, syntactic knowledge and skills and other dimensions of language as a system (…) This [lexical] component considered here from the point of view of a given individual’s communicative language competence, relates not only to the range and quality of knowledge (e.g. in terms of phonetic distinctions made or the extent and precision of vocabulary) but also to cognitive organization and the way this knowledge is stored (e.g. the various associative networks in which the speaker places a lexical item) and to its accessibility, activation, recall and availability. (2001: 13)

But, as we will see later on when we analyze the role vocabulary plays in EOI rubrics, linguistic competence is not sufficient in order to become a competent user of a language. Taking into account the action-oriented approach adopted by the CEFR, it is worth recalling the concept of the language learner as a user of the language as well as in the domains in which learners will have to operate, the knowledge of the culture they will have to call on, and the most likely situations in which they will use the language among other variables. In fact, learners will need to be prepared to use the language they need for specific situations in terms of content- linguistic competence or how they select vocabulary- , appropriateness- strategic competence as in the use of “Nice to meet you” as opposed to “Pleased to meet you” depending on the formality of the encounter- and communicative efficiency- discourse competence when choosing different connectors such as “Nowadays” or “Currently” depending on the type of medium. Therefore, all three components of Communicative Competence- linguistic, discursive and strategic- deal with the use of words and how we adapt the language we have at our disposal in order to suit our convenience.

Furthermore, according to the CEFR, in evaluating a text for use with a particular group of learners, factors such as linguistic complexity, text type, discourse structure, physical presentation, length of text and its relevance for the learners need to be considered. All those factors are currently taken into serious consideration by EOI test designers.

3. THE USE OF RUBRICS TO EVALUATE PRODUCTIVE SKILLS: THE CASE OF OFFICIAL SCHOOL OF LANGUAGES.

A scoring rubric is an attempt to communicate our expectations of quality around a task. Because consistent criteria are made public in rubrics, it allows both teachers and learners to evaluate those criteria which can be sometimes complex and/or subjective. Official schools of Languages in Madrid Regional Government have been formally using rubrics in order to assess productive skills since 2008- when the new system of Certificate Exams was implemented- and have suffered some minor changes from that time. Rubrics are common for all the languages in EOI and are divided into three different grading sheets according to the three different levels, i.e. Basic, Intermediate and Advanced. Speaking Tests (Expresión oral) consist of two different tasks: oral interaction and oral production which are subsequently divided into different descriptors the examiner must grade qualitatively. The same accounts for written tests, in which both written interaction and production are examined. Consequently, in both tests students are expected to produce texts using cohesive devices, vocabulary richness (lexical variation/density, weighed lexical variation and sophistication according to the level), syntactic complexity (dependent clauses per clause among other descriptors) and grammatical accuracy providing teachers with a description of language development that can be used for diagnosis and later curriculum planning.

As it is stated in the Framework, learners/users’ communicative language competence is activated in the performance of the various language activities involving reception, production, interaction or mediation. Each of these types of activity is possible in relation to texts in oral, written form, or both. Here, we will just concentrate on oral and written production as it is the focus of our analysis.

Examiners mark tasks using assessment scales that were developed with explicit reference to the Common European Framework of Reference for Languages (CEFR). The scales, which are used across the spectrum of all languages in Official School of Languages in Madrid depend on the type of text (oral and written) and the task in question where responses can be marked on each subscale from 0 to 3 except for Content or “alcance en el uso de la lengua” which can exceed up to 4 in writing production- task 2 of the Writing Part.

The scales are the following:

• Content, which is analyzed in terms of comprehensibility- communicative functions and sociolinguistic accuracy, that is, how well the candidate has fulfilled the task or if s/he has done what s/he was expected to do and whether the appropriate register has been used for that purpose.
• Communicative achievement or how appropriate the writing is for the task and how the information is organized taking into account its coherence, discursive development and how relevant it is for the task.
• Organization: the way the candidate puts together the piece of writing in terms of language content, grammatical devices, cohesive devices and coherence in the use of tenses; in other words, if the message is presented in a logical and ordered way and also considerations about fluency in the case of Speaking.
• Language. It focuses on vocabulary and grammar as well as spelling and punctuation in writing, and pronunciation and intonation in Speaking. This includes the range of the language and how accurate it is.

Besides, when marking the task, examiners take into account length of responses and varieties of English: Guidelines on length are set according to the general instructions provided each academic year by Madrid Regional Government. Responses which are too short may not have an adequate range of language and may not provide all the information that is required, while responses which are too long may contain irrelevant content and can possibly have a negative effect on the reader. These may affect candidates’ marks on the relevant subscales.

Apart from that, regarding writing production and interaction, the text needs to be efficiently organized, that is, be generally well-organized and coherent, using a variety of linking words and cohesive devices. We need to remember that mastery of vocabulary exceeds choosing appropriate words for specific situations as it also relates to giving the text ‘texture’ by providing lexical cohesion and coherence. In this sense, linking words and connectors in general become essential in certain levels where students are expected to produce texts in a variety of situations with a certain degree of autonomy, that is, as independent users. Let us take English B2 as a case study.Considering B2 level on a global scale, users of B2 need to show a good range of vocabulary for matters connected to his/her field and most general topics. (S/he) can also vary formulation to avoid frequent repetition, but lexical gaps can still cause hesitation and circumlocution. As for lexical accuracy, this is generally high, though some confusion and incorrect word choice does occur without hindering communication.

According to the scales used by EOI professionals, users of B2 level need to be able to produce a text in terms of Communicative Achievement, Organization, Content and Language both for Writing and Speaking. Furthermore, a user of a language has to be efficient in terms of language use: using a range of everyday vocabulary appropriately, with occasional inappropriate use of less common lexis, and if containing errors, those should not impede communication as in the following subscales:

CONTENT/ “ALCANCE EN EL USO DE LA LENGUA”

• Vocabulary: Uses a range of everyday vocabulary appropriately although occasional inappropriate use of less common lexis could be present.
• Cohesive devices: text is connected and coherent, using basic linking words and a sufficient number of cohesive devices.

LANGUAGE/ “CORRECCIÓN FORMAL”

• Vocabulary and grammar: Reasonable control of a wide repertoire of simple lexis and grammar in order to fulfill the most common communicative functions.
• Complex grammatical forms and less common lexis: uses a range of vocabulary, including less common lexis, appropriately along with a range of complex grammatical forms with control and flexibility.
• Spelling and punctuation (Writing): Basic spelling and punctuation without big mistakes; mistakes can occur in less frequent words and punctuation conventions.
• Pronunciation and Intonation (Speaking): pronunciation is clearly intelligible although mother tongue can be noticed at times as well as sporadic mistakes. Intonation is generally appropriate and sentence and word stress are generally accurately placed not to impede communication.

As we can easily observe throughout the analysis of Official School of Languages rubrics, the mastery of vocabulary becomes essential in productive skills in the sense that lexical errors tend to impede communication more than grammar inaccuracy; therefore, lexical errors are rated as more serious than grammar errors since they are measured in all three subscales whereas grammar is merely observed in the last two ones. It is also worth considering the fact that, when asked about what rates an exam task as appropriate or inappropriate, EOI teachers often refer to ‘lexical richness’ in terms of how accurate the vocabulary is and whether it really suits the required level.

The rubrics work in assessing vocabulary control and range holistically in that they adapt to Read’s Three Dimensions of Vocabulary Assessment:

Firstly, we consider vocabulary as embedded not just in discourse where it naturally belongs but also in a specific task, a measure of vocabulary which forms part of the assessment of some other, larger construct. In fact, in the case of writing, examiners have a deep look at the whole task before analyzing the subscales in which the test is divided in order to assess it qualitatively and try to guess whether it fulfills the expected functions or goals of the task (”Impresión global en primera lectura”). In this sense, examiners can have a first approach to the test in general terms and mark it positively or negatively before other concrete aspects of the assessment process are tackled. As for speaking, vocabulary measure is considered as a useful indicator of the fluency of the learner’s speech as it is shown in the third subscale: content or “alcance en el uso de la lengua”- divided into lexis, grammar, cohesion and fluency- which is graded up to four points whereas the rest are graded up to three. Apart from that, all the descriptors in the subscales make reference to very specific conditions depending on the type of text which the examiner needs to bear in mind during the process of evaluation not to lose sight of the texts constraints- which mark its own discursive development, its cohesion and coherence- along with the task itself- which determines features such as length and lexical range.

Secondly, we refer to a comprehensive measure in that it takes account of the test taker’s response (Writing/Speaking tasks) as it is the case of our study. As opposed to ‘selective’ measure, where specific vocabulary items are the focus of the assessment, here, lexis is included as part of the general learners’ response to stimuli, be it spoken or written.

Finally, it is context-dependent. It stands for a vocabulary measure which assesses the test taker’s ability to take account of contextual information in order to produce the expected response, depending, as we stated before, on the type of text and the kind of task required.

Carter, R. (1989) Vocabulary. London: Unwin Hyman.

Common European Framework of Reference for the Teaching and Learning of Languages. available online

Nagy, W. E., & Scott J. A. (2000). “Vocabulary processes”. In M. L. Kamil, P. B. Mosenthal, P. D. Pearson,R. Barr (Eds.), Handbook of reading research: Volume III (pp. 269-284). Mahwah, NJ: Erlbaum.[NICHD] National Institute of Child Health and Human Development. (2000). Report of the National Reading Panel: Teaching children to read. (NIH Publication No. 00-4754). Washington, DC: U.S. Government Printing Office.

Nation, ISP (1990) Teaching and Learning Vocabulary. New York: Newbury House.

Read, J. (2000) Assessing Vocabulary. Cambridge: Cambridge University Press.

Schmitt, N. (2000) Vocabulary in Language Teaching. Cambridge Language Education. Cambridge University Press.

Thornbury, S (2002). How to Teach Vocabulary. Pearson: Longman.