Student self-report measures of change are widely used in evaluation research to measure the impact and outcomes of an educational programme or intervention. Traditionally the design used to measure outcomes is the measurement and comparison of the student's pretest-posttest scores. Traditional pretest-posttest measures work on the assumption that the respondent's assessment of the measurement will not change from the pretest to the posttest. However, the respondent's perception of the constructs under evaluation may change as a result of the educational intervention leading to an underreporting by the respondent of any real change occurring between pretest and posttest, this change in perception is known as response shift. One method that has been suggested as reducing the confounding effect of this response-shift is the use of retrospective pretests. This paper discusses the use of the retrospective pretest as a means of controlling response shift bias in the evaluation of a Masters in Nursing programme. The retrospective pretest instrument was developed from the identification in the literature of the core outcomes of master's level education and was situated within the conceptual framework of Kirkpatrick's (1998) four-level evaluation model. The first level of Kirkpatrick's model measures reaction to the educational experience and usually concentrates on student's satisfaction with a course of study; the second level identifies knowledge gained during the experience; level three is concerned with behaviour change as a result of the programme and; level four measures outcomes achieved. Generally, in the majority of evaluations of nursing programmes levels one and two are assessed, with levels three and four being neglected. The retrospective pretest instrument, the Masters in Nursing Outcomes Evaluation Questionnaire (MNOEQ), was developed to measure student self-reports of change and the impact of the programme at levels three and four of Kirkpatrick's model. Outcomes and behaviours measured included the development of professional communication, the facilitation of the development of leadership and teaching roles, the development of critical thinking skills, research practice and utilisation, initiating positive change in the profession, and development of advanced specialist practice competencies. Data was also collected on students' demographic and professional profiles. The scales that comprise the MNOEQ were tested using factor analysis and classical test statistics (Cronbach's alpha and inter-item correlations). The process of undertaking the retrospective pretest consisted of firstly asking respondents to report their level of ability at present on each item following completion of the master's programme (called a posttest) and then asked to think back and rate themselves on each item before the programme commenced (called a thentest). The collection of thentest and posttest ratings at the same time led to a reduction of response shift bias due to the fact that the respondent was making the ratings at time 1 (thentest) and time 2 (posttest) from the same perspective. The results showed significant differences between the posttest and thentest on a number outcomes. The overall value of the retrospective pretest method was that respondents were less likely to overestimate their ability at the pretest phase of the study due to the fact that this evaluation was taken at the same time as the posttest phase. The theoretical assumption underlying the retrospective pretest method is that by asking the respondent to rate where there are now in terms of ability in relation to the construct under investigation and where they were prior to the educational intervention, they will be using the same internal frame of reference to rate the construct of interest. The retrospective pretest instrument, as an evaluation tool, was found to be reliable and valid in the evaluation of outcomes from a master's level programme. Furthermore, it enabled the measurement of the impact of master's education on the nurse's actual performance and professional practice. The retrospective instrument developed for this study may be of use to other researchers in the evaluation of postgraduate educational programmes in a variety of settings. Reference Kirkpatrick D. (1998) Evaluating Training Programmes: The Four Levels. Berrett-Koehler Publishers, San Francisco.