An investigation of the intellectual skill in the assessment of selected subjects in the Leaving Certificate in Ireland
Author(s):
Denise Burns (presenting / submitting) Joe O'Hara Gerry McNamara
Conference:
ECER 2016
Format:
Paper

Session Information

09 SES 08 B, Formative and Summative Assessments

Paper Session

Time:
2016-08-25
09:00-10:30
Room:
NM-F103a
Chair:
Jennifer Lorenz

Contribution

Numerous media reports in Ireland state that the Leaving Certificate, on exit of secondary schooling, is “all rote learning and memory recall.”   The purpose of this study was to investigate the intellectual skills implied in the written final assessment of 23 subjects from 2005 to 2010.  The key research question was: 

  • What intellectual skills are assessed in the examination papers in the subjects included in the study; and what intellectual skills should be developed by students of the Leaving Certificate, based on the developmental capacities and challenges of students in the 16 – 19 age group? 

The study adopted a definition of “intellectual skill” as a construct, in recognition that the words are representations of processes that are internal and not fully understood.  The definition is sufficiently broad to include a range of skill in different subject disciplines:

All mental processes, some more complex than others, which develop with maturity and can be enhanced with training/practice, including the acquiring of information, the storing and organising of knowledge, all aspects of use of knowledge such as thinking, reasoning, judgement, problem-solving, creative faculties, affective processes, language, expertise. 

Works of authors such as Gavin (1998), Anderson (2000/2005/2010), Schraw (2006/2009), Benjafield (2010), Pelligrino and Hilton (2012), Sternberg and Sternberg (2009/2012) attest to the international use of the abstract concepts of intellectual skill and intellectual development. While the focus of this study is the Leaving Certificate in Ireland, and assessment systems may differ (LeMétais 2003), the issue of the development of intellectual skill of students exiting secondary education is applicable in an international context.  

The research was exploratory, investigating a socially constructed abstract, intellectual skill.  The study makes the assumption that interpretation for intellectual skill can be discerned within the educational community involved in the Leaving Certificate.  The study makes the claim that the findings of this study constitute interpreted, contingent knowledge, bounded by history and culture, rather than certain, absolute, objective knowledge.  This stance implies a philosophical framework of constructivism.  Although interpretation is at the heart of this study, the study claims credibility, as the methodology is documented, systematic, transparent and reproducible.    

The study adopts an approach of investigation of language for intellectual skill.   In alignment with the theoretical approach of constructivism, is a “willingness to use data of different types and from different sources and combine into an analysis and interpretation of a situation” (Newby 2010, p.116). The sources of language investigated for intellectual skill were two-fold:  a corpus of documents seminal to the assessment of the Leaving Certificate subjects and the spoken language of students who had recently completed the Leaving Certificate. 

While many studies use text analysis for the purpose of studying language use per se, a growing number of studies are using document analysis for the purposes of analysing social practices (Santini 2009, p.105).     Documents can be considered as living objects and not “inert objects” (Prior 2011, p.106), separate from their creators and their users.  Documents are “social facts” (Atkinson and Coffey 2011, p.79).   

Complementary to the document analysis was the analysis of interviews of students where students described the intellectual skills they exercised in completing the written examination papers.  A research procedure that invites participants to recall their thinking at the time of an event is called “stimulated recall” (Lyle 2003, p.861).  While the stimulant for recall used by researchers is often video recording, Burden et al (2015) used assessment documents to stimulate recall.  They concluded that using an artefact (document) to stimulate recall can elicit description of thought processes “which may be difficult to obtain in a normal, semi-structured interview” (p.26). 

 

Method

A survey of literature revealed several key points relevant to the hierarchy of intellectual skill, progression within a discipline and the critical time of young adulthood in the development of intellectual skill. A corpus of examination documents immediately relevant to each of the 23 subjects over the years, and available in the public domain, was compiled: examination papers, marking schemes and examiners’ reports. While some subjects have a percentage of coursework assessment, for which instructions were not all available, all subjects have a written final external examination. While Bloom’s taxonomy of intellectual skill is ubiquitous in the literature on the framing of curriculum and assessment, the “revised” taxonomy developed by Anderson and Krathwohl (2001) was used as it addressed the criticisms and empirical testing of Bloom’s taxonomy. The revised taxonomy was two-dimensional, including six categories of intellectual skill: remember, understand, apply, analyse, evaluate and create; and four knowledge domains: factual, conceptual, procedural and metacognitive. Several sources were used to compile the command verbs used in the six categories of intellectual skill. A software programme, called SketchEngine, was then employed to search for these verbs in the examination papers. 14,910 instances of the most frequently used command verbs were assigned values for intellectual skill and knowledge domain. This annotation considered the context of the verb: the object of the verb, the location of the verb within the whole question and the presence of stimulus material. In a validating exercise four experienced secondary teachers assigned values to 100 samples each of five different verbs. A percent agreement index was used to give the proportion of agreement with the researcher’s values for the same instances. The Statistical Package for Social Science (SPSS) was used to analyse the verb data. This process yielded findings of the most frequent intellectual skills and knowledge domains occurring in each subject examination paper. This facilitated a comparison of subjects and subject groups. The transcriptions of interviews of thirty students who had just completed the Leaving Certificate were analysed into recurrent words and patterns for themes in intellectual skill. The comments that exemplified the significant themes and the comments that were contrary to the recurrent themes were noted.

Expected Outcomes

Overall, the analysis of the examination papers revealed that the three lowest intellectual skills dominated significantly over the three higher skills. The distribution of intellectual skills suggested three groupings: 1. Nine subjects with emphasis on remember and understand 2. Seven subjects with an emphasis on performance of the techniques of the discipline (apply) 3. Eight subjects with a spread of intellectual skills. All subjects were low in the skill, analyse. English and Art had the greatest allocation to the two highest skills: evaluate and create. The conceptual and procedural knowledge domains were dominant, with almost no metacognitive knowledge. The occurrence of apply and procedural knowledge was concentrated in five subjects and almost absent in other subjects. The quality of assessment of understand was of concern. While literature acknowledges the difficulty of assessing understanding, literature on theory of learning as construction of knowledge focuses attention on understanding as the development of meaning or as Moseley et al (2005) express it, “concept formation” (p.378). Instances of understand were often snippets of detail in a piecemeal, fragmented manner. The findings of student interviews were something of a counterpoint to the findings of the corpus analysis. Questions that aimed to examine understand were actually assessing remember as student preparation for the examination was dominated by predicting, preparing answers and memorising answers, despite some evidence of genuine deep learning. The findings of the document analyses and the student interviews were reviewed in light of the literature search, with particular reference to intellectual development, progression within a discipline and the challenges of students in the 16-19 age group. This resulted in recommendations for inclusions in assessment and marking schemes. The empirical experience of Anderson and Krathwohl’s taxonomy resulted in a recommendation for an adaptation of the taxonomy.

References

Anderson, John R. (2000/2005/2010): Cognitive Psychology and its Implications. 7th edition. New York. Worth Publishers. Anderson, L. W. and Krathwohl, D.R. (Editors) (2001): A Taxonomy for Learning, Teaching and Assessing. Allyn & Bacon. Boston, Ma. Atkinson, P. and Coffey, A. (2011): “Analysing Documentary Realities” in David Silverman (Editor): Qualitative Research: Issues of Theory, Method and Practice. 3rd edition. Sage. Pp. 77 – 92. Benjafield, J. G.; Smilek, D. and Kingstone, A. (2010): Cognition. 4th edition. Oxford University Press. Burden, S., Topping, A and O’Halloran, C. (2015): “The value of artefacts in stimulated-recall interviews” in Nurse Researcher. Volume 23 (1) Pp. 26 – 33. DOI: 10.7748/nr.23.1.26.e1324 Gavin, Helen (1998): The Essence of Cognitive Psychology. The Essence of Psychology Series. Prentice Hall Europe. LeMétais, J. (2003): International Developments in Upper Secondary Education: Context, Provision and Issues. National Foundation for Educational research. Sponsored by the National Council for Curriculum and Assessment. Lyle, John (2003): “stimulated recall: a report on its use in naturalistic research” in British Educational Research Journal. Wiley. Volume 29, No. 6. December. Pp. 861 – 878. DOI: 10.1080/0141192032000137349 Moseley, D., Elliott, J., Gregson, M. And Higgins, S. (2005): “Thinking Skills Frameworks for Use in Education and Training” in British Educational Research Journal. Routledge. Volume 31, No. 3. Pp. 367 – 390. Newby, Peter (2010): Research Methods for Education. Longman Pearson. Pelligrino, J. W. and Hilton, M. L. (2012): Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century. Committee on defining Deeper Learning and 21st Century Skills. National Research Council. Division on Behavioral and Social Sciences and Education. Washington D.C. National Academies Press. Prior, Lindsay (2011): “Using Documents in Social Research” in David Silverman (Editor): Qualitative Research: Issues of Theory, Method and Practice. 3rd edition. Pp. 93 – 110. Sage. Santini, Marina (2009): Book Review: Discourse on the Move: Using Corpus Analysis to Describe Discourse Structure by D. Biber, U. Connor and T.A. Upton. In Computational Linguistics. MIT Press. Volume 25, No. 1. Pp. 105 – 107. Schraw, Gregory (2006/2009): “Knowledge: Structures and Processes” in P. Alexander and P. Winne (Editors): Handbook of Psychology. 2nd edition. Routledge. Taylor and Francis Group. American Psychological Association. Pp. 245 – 263. Sternberg, R.J. and Sternberg, K. (2009/2012): Cognition. 6th edition. Wadsworth.

Author Information

Denise Burns (presenting / submitting)
Dublin City University
Education
Dublin
Dublin City University, Ireland
Dublin City University, Ireland

Update Modus of this Database

The current conference programme can be browsed in the conference management system (conftool) and, closer to the conference, in the conference app.
This database will be updated with the conference data after ECER. 

Search the ECER Programme

  • Search for keywords and phrases in "Text Search"
  • Restrict in which part of the abstracts to search in "Where to search"
  • Search for authors and in the respective field.
  • For planning your conference attendance, please use the conference app, which will be issued some weeks before the conference and the conference agenda provided in conftool.
  • If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.