Educational equity has been regarded as core value for a democratic society to achieve. One of the approaches that address inequity is to provide educational programs to develop students’ potentials regardless of their backgrounds (Rochex, 2012). Furthermore, evaluation can be used to describe and judge the process quality and actual effects of the program (Pearson et al., 2007). Notably, schools may need different resources for their students due to the unique characteristics of their community and students. This study argues that a school is an autonomous agency to address their students’ disadvantages, by integrating the various programs that properly target their students’ unique needs. Therefore, the purpose of this study is to develop essential criteria for evaluating disadvantaged assistance programs that are implemented by schools. The study used the Taiwanese case to provide topics for reflection on the policy and school practices related to the disadvantaged assistance programs.

This study included two strands of literature: equity and program theory evaluation. First of all, a lot of discussions have focused on the equity issues. Some have paid special attention on the development of equity indicators (European Group for Research on Equity in Educational Systems, 2005; Walberg, H., & Zhang, G., 1998). These results show that equity is a multi-faceted construct, comprising the context characteristics, as well as access, process, and outcome equity. 

The second strand of the literature that informs this study is the program theory evaluation, which covers the different terms in evaluation research, such as theory-based evaluation (Weiss, 1998) and theory driven evaluation (Chen, 2005). Program theory evaluation is used to develop a change mechanism behind the program. It is commonly used in a variety of program evaluations, including educational evaluation. For example, Dyson and Todd (2010) used the program theory to evaluate the full service extended schools initiative in UK. Their evaluation structure consisted of the situation, main strands of action (including community and parental involvement in schooling, services for young people, raising school performance/profile), and outcomes (including raising aspirations of community, achievement and attainment in school, removing the barriers to learning, and thriving schools). 

This study used the Fuzzy Delphi technique to develop consensus of opinions among a panel of experts through survey. This technique used the fuzzy calculation to reflect the potential ambiguity of responses to the survey (Chen & Hwang, 1992). The panelists included 9 education scholars and 10 principals who were recommended for their expertise with disadvantaged assistance programs in Taiwanese schools. The principals were invited from the schools of urban, periphery, remote areas, as well as the schools in between, in elementary, junior high, and senior high schools. The survey for this study had three sections, including the background of the study, rating scales of the criteria, and open-ended questions for comments. A Likert-type scale with a five-point continuum was used to judge the fit of evaluation criterion for disadvantaged assistance programs, ranging from 1(very low) to 5 (very high). The Defuzzification was used for data analysis, and α-level at .7 was chosen as the threshold to determine whether the evaluation criteria are maintained in the framework. The Fuzzy Delphi survey was developed based on two rounds of expert reviews. The first round of review was conducted with seven experts through a group interview to investigate the appropriateness of the criteria framework for disadvantaged assistance programs. According to the results of the interview, major revision of the criteria framework included: 1. Reorganizing the dimensions of criteria into context, program design, program implementation, external systems (government policy as well as family and community support), and program outcomes; 2. Adding a sub-dimension of administrative leadership; 3. Categorizing the program outcomes into primary outcomes and extended outcomes; and 4. Rephrasing the descriptions to increase the clarity of evaluation criteria. The second round of review was conducted with eight experts through a written form of three judgments including appropriate, appropriate after revision, and inappropriate. The experts were also invited to offer suggestions for revising the criteria. Based on the review, this study integrated the dimension of external systems into the context, in addition to merging the similar criteria and rephrasing the descriptions to increase its clarity.

The initial criteria framework included four dimensions, nine sub-dimensions, and fifty-one criteria. Four dimensions covered context, program design, program implementation, and program outcomes. Specifically, two sub-dimensions and 10 criteria underneath the context were school, family, and community, as well as policy network. One sub-dimension and four criteria belonged to the dimension of program design. Four sub-dimensions underneath the program implementation were administrative leadership, consulting and instruction, resources connection and utilization, along with connection with family and community. The program outcomes contained two sub-dimensions (primary outcomes for students and extended outcomes mainly for teachers, parents, schools, and community) and 15 criteria. Based on the results, this study addresses the research limitations, and provides implications for policy and school practices in helping disadvantaged students.

Chen, H. (2005). Practical program evaluation: Assessing and improving planning, implementation, and effectiveness. Thousand Oaks, CA: Sage. Chen, Shu-Jen., & Hwang, Ching-Lai. (1992). Fuzzy multiple attribute decision making: Methods and applications. Berlin, Germany: Springer-Verlag. Dyson, A., & Todd, L. (2010). Dealing with complexity: Theory of change evaluation and the full service extended schools initiative. International Journal of Research and Method in Education, 33 (2), 119-134. European Group for Research on Equity in Educational Systems (2005). Equity in European educational systems: A set of indicators. Retrieved from http://www.aspe.ulg.ac.be/equite/fichier/pdf/2005PDF_ENGLISH.pdf Pearson, D., Dyson, D., Muijs, R., Cummings, C., Tiplady, L., Todd, L., ... Crowther, D. (2007). Evaluation of the full service extended schools initiative: Final report (DfES). Nottingham: Department for Education and Skills. Rochex, J. (2012). General conclusion: Priority education policies in Europe, from one age and one country to another. In M. Demeuse, D. Frandji, D. Greger, & J. Rochex (Eds.), Educational Policies and Inequalities in Europe (pp. 288-317). England, UK: Palgrave. Walberg, H., & Zhang, G. (1998). Analyzing the OECD indicators model. Comparative Education, 34 (1), 55-70. Weiss, C. H. ( 1998 ). Evaluation: Methods for studying programs and policies (2nd ed. ). Upper Saddle River, NJ: Prentice Hall.