Evaluation and Accountability

The evaluation design for the North Carolina State Implementation Project purposively includes methods that encompass the goals, objectives, and outcomes of the project. To promote thoroughness and usefulness of the evaluation information gathered, a collaborative evaluation approach will be employed. The evaluation procedures also include monitoring the effectiveness of project activities in such a way that project implementers receive regular performance feedback as well as annual assessments of outcome achievements.

Thus, the evaluation will include a process orientation (to review implementation activities and their quality) as well as an outcome orientation (to progressively examine accomplishments and outcomes). The evaluation design will measure the project’s performance in attaining the objectives and outcomes identified in the proposal, while acknowledging that much of the evaluation information collected will require qualitative treatments.

In addition, the design calls for a collaborative evaluation process using both external evaluation consultants and internal staff to assist with evaluation. The Evaluation, Assessment, & Policy (EvAP) Center will act as external evaluators for the project.

The Center, located at the University of North Carolina, is an evaluation consulting group that conducts evaluations and provides training and technical assistance in evaluation, assessment, and strategic planning to educational, community, and service organizations across the United States.

return to top

Collaborative Evaluation
The term collaborative evaluation often is used interchangeably with participatory and/or empowerment evaluation (Fetterman, 1996). Cousins, along with colleagues (1992, 1995, 1996), has done considerable work in the area of collaborative and participatory evaluation. He defines collaborative evaluation as "any evaluation in which there is a significant degree of collaboration or cooperation between evaluators and stakeholders in planning and/or conducting the evaluation" (Cousins, Donohue, & Bloom, 1996, p.210).

To the extent that they are able, project members of the North Carolina State Improvement Partnership should be considered part of the evaluation team. This does not relieve the external evaluator of the overall responsibility for conducting the evaluation or producing evaluation results. Evaluators are engaged because of the expertise they bring to the endeavor, and leadership for the evaluation resides with that expertise. The collaborative approach used for this evaluation implies that people share responsibility and decision-making for the evaluation. This is very consistent with the collaborative nature of this project. Three factors swayed the decision to use a collaborative evaluation approach: (a) engagement of project implementers in the evaluation will greatly assist in the development of effective evaluation tools and data collection strategies; (b) including project personnel among the evaluation team members provides critical resources to guarantee a thorough evaluation; and (c) collaborative evaluation increases the usefulness of evaluation findings.

return to top

Evaluation Methods
To assist us with evaluation planning, we investigated the relationships between the programmatic inputs planned by the project and the desired outcomes for the project. This process of causal mapping helps to clarify the relationships between the project strategies and activities and the expected outcomes (Venezky, R.L., 1999). As can be seen in Figure 2, our assumption is that there is an array of intermediate goals which must be met before the expected outcomes will be attained. However, the relationships between the intermediate goals and the outcome goals are only informed assumptions. It is quite possible that the project will accomplish one or more of the intermediate goals in an effective and efficient manner without seeing the desired impact on the outcome goals. On the other hand, it is also possible that one or more of the intermediate goals may not be met but the outcome goal or goals associated with the intermediate goal is attained. To determine what state improvement project strategies and activities are important to continue, and to gain insight as to directions for change, it is critical that the evaluation process provides us with as much explicit information as possible about relations between program strategies, intermediate goals, and outcome goals.

return to top