The Evaluation of the Impact of a New, Broad IT Bachelor Curriculum amongst Freshmen and Sophomores
Author(s):
Monique Bijker (presenting / submitting) Chris Kockelkoren (presenting)
Conference:
ECER 2017
Format:
Paper

Session Information

Paper Session

Time:
2017-08-24
13:30-15:00
Room:
K5.19
Chair:
Tamás Kozma

Contribution

In 2008 the first European e-Competence Framework was launched. The e-Competence Framework (EC-F, 2014) was used by the Dutch IT-Foundation of Universities of Applied Sciences (HBO-I Stichting, 2014) to formulate guidelines, together with employers and the professional field, to redesign IT-curricula accordingly. Their aim was to better prepare graduates for a complex and dynamic practice. At Zuyd University of Applied Sciences this resulted into a new core strategy (Zuyd, 2014), and a new broad IT Bachelor program. The curriculum design process was structurally based on the Neo-Piagetian framework of Biggs and Collis (1982), and co-created by IT-teachers and the work field. The new IT-curriculum was introduced in September 2013. 

The taxonomy of cognitive development of Biggs and Collis (1982), the Structure of the Observed Learning Outcome (SOLO), distinguishes five main phases. The first three main phases are characterized by a quantitative increase of knowledge. Prior to any cognitive development there is a phase of ignorance (SOLO Prestructural), which is followed by a second phase in which isolated facts or concepts are mentioned (SOLO Unistructural). The third quantitative phase is characterized by listing facts (SOLO Multistructural). In the fourth main phase a qualitative change in the knowledge structure occurs. Relationships between different facts and concepts are presented coherently, creating a context for logical arguments, thereby increasing the chance that reliable and valid conclusions are drawn (SOLO Relational). The fifth main phase includes the transfer of the structured knowledge from the relational phase to a whole new context (e.g. the development of a new theory; the formulation of hypotheses; the creation of new models or designs; SOLO Extended Abstract). Between the quantitative main phases transitional phases can occur (e.g. SOLO X transitional; SOLO D transitional), which show characteristics of both adjacent main phases, but have insufficient substance of the upper phase.

The SOLO-taxonomy allows domain-independent criterion-based evaluation of cognitive progress in a reliable and valid way, and can easily be combined with domain-specific competence rubrics. Many different domains have already used the taxonomy (e.g. writing essays (Campbell, Watson, & Collis, 1992); counselling (Burnett, 1999); statistical literacy (Watson & Moritz, 2000); swallowing processes (Scholten, Keeves, & Lawson, 2002); knowledge development in a computer-supported-collaborative learning environment (Veldhuis-Diermanse, 2002); nursing (Trigwell & Prosser, 1991); and academic writing (Tynjälä, 1999). 

In the development process of the new, broad IT curriculum the IT-teachers translated the intended SOLO-phases into the five HBO-I task types (managing; analysing; consultancy; design; and delivery), and intended HBO-I competence levels.

In the Spring of 2015 the first evaluation of the learning outcomes was scheduled. Two equivalent case-based authentic tasks were designed, with equal numbers of intrinsic elements, and based on interviews with employed alumni that graduated two to three years ago. Case A was about the migration of IT-related matters of a flourishing, fast growing company in the automotive industry; case B concerned a future-proof adaptation of the IT architecture of a large pension and insurance organization. Both cases were connected to the assignment to create a Project Initiation Document, which would be presented to the Director of the company. The worked out assignments of freshmen and sophomores were the data (input) for the evaluation of the new IT-curriculum.  

The central research question was “How do freshmen and sophomores of the broad ICT bachelor perform on authentic, professional tasks”? HBO-I intends that graduates perform at a HBO-I 3 level, which corresponds to a SOLO-B level.

It was hypothesized that in the new broad curriculum freshmen would perform on a SOLO D level, and sophomores on a SOLO C level. 

Method

Participants Thirty-eight freshmen and 25 sophomores participated in the study. Their mean age was 21, 24 years (SD 2.43) and 6.35% was female. Forty-nine per cent of the participants had an educational background in vocational education; the other participants were from generic secondary education or pre-university education. Participation was voluntary, and students received a gift voucher of € 10.- if they filled out the survey, and completed the assignment. Design The cross-sectional, mixed-method study was a combination of a quasi-experiment and a survey study, aimed at measuring the effectiveness of the new IT-curriculum in the first two years of the new program. Materials Case A and Case B were randomly assigned. Instruments Participants were asked to fill out a form with personal data such as demographic information (age; gender), prior education, current year of study, obtained European Credit Transfer System points (ECTS), and work experience. Furthermore, the questionnaire entailed five scales (Bijker, 2013; Raemdonck, 2006; Pintrich, Smith, Garcia, & McKeachie, 1993): an 18-item dichotomous scale Self-Directed Learning (SDL); a 9-item 3-point Likert Scale “Case Perceptions” (CP), specifically designed for the current study; two 5-item 5-point Likert scales Intrinsic Motivation (IM) and Task Self-Efficacy (TSE): and a 4-item 5-point Likert scale Education Self-Efficacy (ESE). The respective Cronbach’s alpha’s of these scales were .80 (SDL); .81 (CP); .83 (IM); .83 (TSE); and .81 (ESE). Finally, the survey also entailed three open questions about the perceived strengths and weaknesses of the curriculum. Procedures Potential participants were personally approached by the researchers. A classroom where participants could work on their personal laptops was reserved to conduct the study. Participants could choose a printed version of the case, a pdf-version on a USB-stick, or an aural version (MP-3). They were allowed to make notes and to use the Internet. After half an hour, all case versions were handed over to the researchers, after which the participants continued with another USB-stick. On this stick they saved their worked out assignment in a document that encompassed a macro in which the participant filled out his student ID number. Furthermore, the document provided the web address of the questionnaire that automatically saved the responses. The maximum time to work on the assignment and fill out the questionnaire was 2.5 hours. The questionnaires were downloaded from the internet, and uploaded and coded in Atlas-ti 7.5.7. Analysis Descriptive statistics, inter-rater reliability, correlation analyses, ANOVA’s and ANCOVA’s were executed in SPSS-22.

Expected Outcomes

Inter-rater Reliability Domain-specific codes for Case A and Case B are developed, based on worked out PID’s by IT-professionals. Cohen’s Kappa of the SOLO-codes = 0.82. Testing the hypotheses The expectation was that freshmen would achieve a SOLO-D (Unistructural) level, and sophomores a SOLO C (Multistructural) level. One-sample T-tests however demonstrate that students’ SOLO achievements are significantly lower than the expected SOLO levels in both study years. Correlations Case perceptions (CP) correlate significantly and positively with SOLO: r = .34, p = .01, and the same applies to Task Self-Efficacy (TSE): r = .31, p = .02. The other scales are not significantly related to the SOLO scores. Perceived strengths and weaknesses of the curriculum A top-10 of strengths and weaknesses is identified. Strengths are subcategorized into Teaching, Design, and Facilities; Weaknesses into Teaching, Design, and Assessments. The broadness of the new curriculum is Strength number 1, followed by the teaching and learning climate (nr 2), and the multiple opportunities to self-direct the learning processes. Weakness nr 1 is communication between teachers, nr 2 specific subjects, and nr 3 teacher’s domain-specific expertise. Recommendations The qualitative findings and recommendations to optimize the curriculum are presented and discussed during the conference.

References

Biggs, J. B., & Collis, K. F. (1982). Evaluating the quality of learning: the SOLO taxonomy. New York: Academic Press. Bijker, M. M. (2013). Understanding the gap between business school and the workplace: Overconfidence, maximizing benefits and the workplace. (Doctoral dissertation), Open University of the Netherlands, Heerlen, the Netherlands. Burnett, P. C. (1999). Assessing the structure of learning outcomes from counselling using the SOLO taxonomy: an exploratory study. British Journal of Guidance & Counselling, 27(4), 567-580. Campbell, K. J., Watson, J. M., & Collis, K. F. (1992). Volume measurement and intellectual development (pp.279-298). Tasmania: University of Tasmania. EU (2014). Towards a European Competence Framework. Retrieved from http://www.ecompetences.eu/ HBO-I Stichting (2014). Domeinbeschrijving Bachelor of ICT [Domain Description Bachelor of ICT]. Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1993). Reliability and predictive validity of the motivated strategies for learning questionnaire (MSLQ). Educational and Psychological Measurement, 53, 801-813. Raemdonck, I. (2006). Self-directedness in learning and career processes. A study in lower-qualified employees in Flanders. (Doctoral thesis), University of Gent, Belgium. Scholten, I., Keeves, J. P., & Lawson, M. J. (2002). Validation of a free response test of deep learning about the normal swallowing process. Higher Education(44), 233-255. Trigwell, K., & Prosser, M. (1991). Relating approaches to study and quality of learning outcomes at the course level. British Journal of Educational Psychology, 61, 265-275. Tynjälä, P. (1999). Towards expert knowledge? A comparison between a constructivist and a traditional learning environment in the university. International Journal of Educational Research and Evaluation(31), 357-442. Veldhuis-Diermanse, A. E. (2002). CSCLearning? Participation, learning activities and knowledge construction in computer supported collaborative learning in higher education. (Doctoral dissertation), Wageningen University, the Netherlands. Retrieved August 5, 2006, from http://edepot.wur.nl/121278 Watson, J. M., & Moritz, J. B. (2000). Development of understanding of sampling for statistical literacy. The Journal of Mathematical Behavior(19), 109-136. Zuyd (2014). Kern Strategie [Core Strategy].

Author Information

Monique Bijker (presenting / submitting)
Zuyd University of Applied Sciences
Mevrouw
Heerlen
Chris Kockelkoren (presenting)
Zuyd University of Applied Science
ICT Faculty
Heerlen

Update Modus of this Database

The current conference programme can be browsed in the conference management system (conftool) and, closer to the conference, in the conference app.
This database will be updated with the conference data after ECER. 

Search the ECER Programme

  • Search for keywords and phrases in "Text Search"
  • Restrict in which part of the abstracts to search in "Where to search"
  • Search for authors and in the respective field.
  • For planning your conference attendance, please use the conference app, which will be issued some weeks before the conference and the conference agenda provided in conftool.
  • If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.