Assessment of Organizational Evaluation Capacity

Research

Assessment of Organizational Evaluation Capacity

Organizational Evaluation Capacity (EC) can be generally defined as an organization’s visible, enacted evaluation practices and processes. In a nutshell, organizations with a high level of evaluation capacity conduct quality evaluation studies in a timely manner, use evaluation findings regularly as part of program and organizational decision-making, and adopt an evaluative culture founded on the principles of organizational learning.

The Organizational Evaluation Capacity Assessment Instrument is based on a framework of organizational evaluation capacity developed by Bourgeois and Cousins (2013). The instrument is meant to highlight organizational evaluation capacity in six main dimensions, which constitute the main sections of the instrument (divided into “capacity to do” and “capacity to use” evaluation)

One instrument should be completed per organization. In many instances, the person completing the instrument may need to draw on the knowledge and opinions of other organizational members. This can include (but is not limited to): senior executives, program managers, corporate staff, etc. These individuals can be consulted in a number of different ways, depending on the individual and the situation (e.g., formal or informal interviews, departmental evaluation committee meetings, written communications, etc.). We encourage you to complete it in a workshop format, where the group can discuss the ratings to be given to each item. We have often found that these discussions end up being as valuable as the completed instrument results.

To learn more about the conceptual framework on which the instrument is based as well as how it was developed and validated, please see these two papers:

The instrument has been used as part of several different research projects and publications. The following list provides basic details about each study as well as key references:

 

Measurement of organizational evaluation capacity in public health: This project, conducted between 2014 and 2018, sought to adapt the capacity assessment instrument in order to measure organizational evaluation capacity in Ontario public health units. This project also enabled us to develop and test various evaluation capacity building strategies in participating health units.

  • Bourgeois, I., Simmons, L. Hotte, N., & Osseni, R. (2016). Measuring organizational evaluation capacity in Ontario public health units: An empirical investigation. Canadian Journal of Program Evaluation, 31 (2), 165-183. https://evaluationcanada.ca/system/files/cjpe-entries/31-2-165.pdf
  • Bourgeois, I., Simmons, L., & Buetti, D. (2018). Building Evaluation Capacity in Ontario’s Public Health Units: Promising Practices and Strategies. Public Health, 159, 89-94. DOI: https://doi.org/10.1016/j.puhe.2018.01.031
  • Bourgeois, I., & Buetti, D. (2019). Using Action Research to Build Evaluation Capacity in Ontario: Lessons Learned. Journal of Multidisciplinary Evaluation, 33, 81-90.

Adaptation of the instrument to the community sector: Research conducted by David Buetti as part of his doctoral work focused on adapting the instrument to the community sector, in order to better reflect the context and realities of community-based organizations in the health and social services sectors located in Québec. The adapted instrument enabled a systematic assessment of organizational evaluation capacity in this sector.

  • Buetti, D., & Bourgeois, I. (2019). L’étude des capacités en évaluation chez les organismes communautaires du Québec : proposition d’un cadre conceptuel et d’une grille d’analyse organisationnelle. Intervention, 150, 25-46.
  • Buetti, D., Bourgeois, I., & Savard, S. (2019). Modélisation des capacités organisationnelles en évaluation dans le secteur communautaire et implications pour le contexte québécois. Revue canadienne de recherche sur les OSBL et l’économie sociale, 10 (2), 16-39.

Comparisons of organizational evaluation capacity: This project sought to compare and contrast, using the validated instrument, types of levels of evaluation capacity in various organizations.

  • Bourgeois, I. &Cousins, J.B. (2008). Informing evaluation capacity building through profiling organizational capacity for evaluation: An empirical examination of four Canadian federal government organizations. Canadian Journal of Program Evaluation, 23(3), 127-146.
  • Bourgeois, I., Whynot, J., & Thériault, É. (2015). Application of the Organizational Evaluation Capacity Self-Assessment Instrument in different organizations: Similarities and lessons learned. Evaluation and Program Planning, 50, p. 47-55. https://doi.org/10.1016/j.evalprogplan.2015.01.004