We run a randomized controlled trial (RCT) and employ regression analysis to identify the effect of the treatment. The first outcome variable is calculated as a sum score of three sub-scales from the CDMSE short form (Betz & Taylor 1983). The second outcome variable (degree specificity) is coded as dichotomous variable that indicates whether someone intends to choose a popular study programme or not. Standard socio-demographic control variables (gender, age, migration background, SES of the parents) are included in the analysis. Moreover, we additionally include a pre-treatment measure of the outcome variable in our analysis. To estimate the treatment effect, we perform instrumental variable (two-stage least squares) estimation with the actual treatment status as (endogenous) independent variable and the assignment to treatment as instrument to account for endogenous non-compliance. Moreover, we combine this estimation technique with a recent adaption of the random forest algorithm as suggested by Athey et al. (2017) to account for effect heterogeneity. This approach enables a non-parametric estimation of effect heterogeneity, i.e. it quantifies the heterogeneity with respect to a set of covariates without imposing a functional form on the heterogeneity. From a practical point of view, this is of great importance as previous research has sometimes erroneously concluded that effect heterogeneity plays a minor role once it could not be modelled with parametric techniques. While a parametric test would of course give a more direct hint on the effectiveness of the treatment for certain sub-groups, the non-parametric counterpart is more flexible and can quantify the degree to which treatment effects differ between persons or groups of persons, even if it is hard to spot with parametric techniques.

Our (very) preliminary results show that the treatment has a positive effect on both outcome variables. While the level of CDMSE increases slightly in the control group, the increase is steeper in the treatment group. Correspondingly, the two-stage least square estimations reveal significant treatment effects. Similarly, the treatment increases the likelihood that pupils want to study a non-standard study programme that does not belong to the most frequently chosen programmes. This result is robust to changes in the estimation technique, e.g. compared to semi-parametric IV estimation as suggested by Frölich (2007). Finally, our non-parametric estimations point to a non-negligible extent of effect heterogeneity, and comparisons between the IV estimator and naïve per-protocol analyses reveal strong differences between the estimated effects. These results support our theoretical argument that some pupils choose popular study programmes due to a lack of information and the difficulty in finding suitable information about more appropriate study programmes. Generally speaking, this encourages the view that intensive profiling can lead to a better fit between study choice, skills and interests. It is subject to future research how these differences in study intentions translate into differences in long-term outcomes, such as employment outcomes or job satisfaction. From a methodological point of view, the comparison of the different estimation techniques confirms the argument that future evaluations should provide a more rigorous assessment of non-compliance (since even the selection due to non-compliance substantially changes our results) as well as effect heterogeneity (since the random forest estimation point to substantial effect heterogeneity, even if this is hard to model with parametric techniques).

Athey, Susan; Tibshirani, Julie; Wager, Stefan (forthcoming): Generalized Random Forests. In: Annals for Statistics. Frölich, Markus: Nonparametric IV estimation of local average treatment effects with covariates. In: Journal of Econometrics 139 (1), 35-75. Isik, Erkan (2013): Effect of Interest Inventory Feedback on Career Decision Self-efficacy among Undergraduate Students. In: Procedia - Social and Behavioral Sciences 84, S. 1437–1440. Koen, Jessie; Klehe, Ute-Christine; van Vianen, Annelies E.M. (2012): Training career adaptability to facilitate a successful school-to-work transition. In: Journal of Vocational Behavior 81 (3), S. 395–408. Koivisito, Petri; Vinokur, Amiram D.; Vuori, Jukka (2011): Effects of Career Choice Intervention on Components of Career Preparation. In: The Career Development Quarterly 59, 345-366. Taylor, Karen M.; Betz, Nancy E. (1983): Applications of self-efficacy theory to the understanding and treatment of career indecision. In: Journal of Vocational Behavior 22 (1), S. 63–81. Whiston, Susan C.; Li, Yue; Goodrich Mitts, Nancy; Wright, Lauren (2017): Effectiveness of career choice interventions: A meta-analytic replication and extension. In: Journal of Vocational Behavior 100, S. 175–184.