A unique evaluation approach to examine and enhance effectiveness of a Leadership Management Training Programme (LaMP) for Medical Specialty Trainees

Author(s):

Linda Halley(presenting / submitting)Matt McGovern(presenting)

Conference:

ECER 2013

Network:

11. Educational Improvement and Quality Assurance

Format:

Paper

Session Information

11 SES 09 B, Functions of Educational Leadership at Schools

Paper Session

Time:

2013-09-12

11:00-12:30

Room:

D-311

Chair:

Irina Maslo

Contribution

Effective clinical leadership is the cornerstone of high quality patient care and the improvement of our services as recognised in the NHS Scotland Quality Strategy and embedded in curricula that support doctors through training. To provide high-quality service to patients, Doctors need to possess leadership and management competences¹.

NHS Education for Scotland (NES) and the Training Development Support Unit (TDSU) deliver a national Leadership and Management Training Programme (LaMP) of skills and knowledge to Specialty Trainees aiming to:

support medical professionals to become effective leaders and managers building capacity to take on senior roles
support delivery of generic competencies identified within Good Medical Practice² guidance, the PMETB standards³ and Royal College curricula⁴ that map to the Medical Leadership Competency Framework⁵

Research and evaluative approaches used to assess the effectiveness and impact of such training upon a doctors’ practice can be limited. Training programme evaluations could benefit from taking into account the influence of context (i.e. a doctors’ complex work environment) upon the transfer of learning from training which is key to identifying why a programme was successful/unsuccessful in achieving its’ outcomes.

New educational research work by TDSU is attempting to broaden and improve upon their existing evaluation framework for examining the effectiveness of LaMP training through developing an integrated approach combining process/formative evaluation and outcome/summative evaluation.

We currently combine the use of an enhanced version of the Kirkpatrick Model⁶ (to measure programme outcomes) with a Theory of Change approach⁷ i.e. a Logic Model to allow for a more explicit articulation of LaMPs’ implementation and programme theory (and outcome identification) illustrating the logical relationships with respect to how invested resources/programme activities lead to results. This unique approach will provide greater exploration into why and how the intended and unanticipated outcomes of the programme are achieved while considering the influence of environmental factors; a level of depth unattainable in the sole use of the Kirkpatrick approach.

The aim and objectives of the evaluation are based around the first three levels of the Kirkpatrick model (longer-terms impacts are considered in the Logic Model) and assess the effectiveness and impact of the LaMP programme upon clinicians who attend the courses through:

examining the general level of satisfaction (reaction Level 1) amongst clinicians with respect to structure, content and delivery to provide a continuous feedback mechanism to improve the content and delivery of courses i.e. TDSU Course Improvement Procedure
assessing the resulting modification in clinicians’ attitudes/perceptions and knowledge/skills (learning Level 2) relating to taught course content
assessing any changes in clinicians’ behaviour/practice in clinical settings as a result of attending LaMP training, intention to and actual transfer of learning (behaviour Level 3) from training context to the workplace
uncovering the theory underpinning the programme (using Logic Models) and examining how the outcomes generated were achieved and any unanticipated outcomes.
examining external/contaminating factors which may confound the attribution of outcomes to LaMP and any contextual factors which may be facilitating or inhibiting the delivery of the programme.

Method

Evaluators are employing a mixed methods approach to collecting and analysing data including a range of qualitative and quantitative tools and data sources: A comprehensive self-report pre/post-course questionnaire tests clinicians’ -pre-training attitudes and behaviours related to Leadership and Management, -post-training satisfaction with course content and delivery (L 1), -pre and post-course knowledge of course objectives (L 2), -post-training behavioural intentions for change and transfer of learning (L 3) An objective pre/post-course multiple choice test of knowledge examines learning acquired by delegates of course taught content (L 2) A detailed self-reported training impact survey and semi-structured qualitative interviews explore the impact of the training on knowledge, skills and attitudes and behaviour change/transfer of learning including follow-up post-training assessment of attitudinal and behavioural criteria related to leadership and management (L 2 & 3). A pre/post programme analysis of an independent source of medical workplace assessment multi-source feedback data measures changes in clinicians’ leadership and management attitudes/behaviours (L 3) Unintended outcomes, potential confounding and contextual factors are assessed through the aforementioned methods at various levels. Qualitative data is being transcribed and thematically analysed. Access, Excel, SPSS are being used to analyse quantitative data to produce relevant descriptive statistics.

Expected Outcomes

Evaluation is on-going and has captured a number of positive results and high satisfaction endorsing the quality and delivery of the LaMP programme with approximately 89% of courses being rated as Good/Excellent and 85% agreeing the programme is meeting learning needs. A continuous feedback mechanism for course improvement has been implemented within the TDSU and future analysis will determine whether this has been effective in increasing satisfaction ratings. Results from the tests of knowledge will be presented as well as the impact of the programme upon clinicians’ attitudes/behaviour and transfer of learning. Self-reported data will be triangulated with workplace assessment (MSF) data to strengthen findings. Leadership and Management training may have the potential to exert a positive impact upon Doctors. Improving quality of training could strengthen its’ impact and this can be more reliably assessed using an integrated evaluation approach that facilitates explicit articulation of programme theory and outcomes. The feasibility of incorporating objective observational measures of performance (i.e. workplace assessment data) will be further discussed. Implications and learning from the programme and evaluation are informing the wider national and international agenda of health service quality improvement and better patient care.

References

1. Generic Training: NHS Education for Scotland: http://www.nes.scot.nhs.uk/education-and-training/by-discipline/medicine/about-medical-training/generic-training/leadership-and-management-(lamp).aspx 2. Good Medical Practice: General Medical Council 2006: http://www.gmc-uk.org/guidance/good_medical_practice/index.asp 3. Generic Standards for Training: PMETB 2006: http://www.pmetb.org.uk/ 4. Generic Curriculum for the Medical Specialties: Federation of the Royal Colleges of Physicians UK 2006: http://www.jrcptb.org.uk/Pages/homepage.aspx 5. Medical Leadership Competency Framework: NHS Leadership Academy: http://www.leadershipacademy.nhs.uk/discover/leadership-framework/ 6. Kirkpatrick, D. L. Evaluating training programs: the four levels. San Francisco: Berrett-Koehler 1994 7. Weiss, Carol Hirschon. Evaluation Research: Methods of Assessing Program Effectiveness. Englewood Cliffs, NJ: Prentice-Hall 1972

Author Information

Linda Halley (presenting / submitting)

NHS Education for Scotland

Educational Research & Development

Edinburgh

Matt McGovern (presenting)

NHS Education for Scotland, United Kingdom

Search the ECER Programme

Search for keywords and phrases in "Text Search"
Restrict in which part of the abstracts to search in "Where to search"
Search for authors and in the respective field.
For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.