Building evaluation capacity in to a large public sector emergency service : an action research study

Koljonen, H

Abstract

This action research study investigated the feasibility of building evaluation capacity into the design process of Greater Manchester Fire and Rescue Service’s (GMFRS) community initiatives. GMFRS runs over 400 community initiatives every year and an appropriate evaluation framework is needed to provide evidence of their impact and effectiveness. Information that previous evaluation processes had failed to provide.

The feasibility of establishing an evaluation framework for use by non-specialists users was explored, the development of a new evaluation system for GMFRS was examined critically, and the processes and challenges involved in embedding evaluation within a large public sector emergency service investigated.

Mixed research methods -- document analyses, qualitative interviews, observations and focus groups -- were used in the three action research cycles conducted between April 2008 and April 2010. In cycle one, existing evaluation materials and key characteristics of the initiatives were assessed. GMFRS’s community initiatives were found to lack direction, and the existing evaluation tool lacked the detail required for use by personnel with no previous evaluation experience. Hence, new evaluation materials were developed and, in the second cycle, their use was observed, and interviews conducted to ascertain the barriers to evaluation practices. Lack of resources, organisational guidance, and support with evaluation activities were identified as barriers. In the third cycle focus groups were used to gain feedback on the usability of the new evaluation material and processes.

The principal conclusions of the research are that there were no theoretical models and/or guidance to assist Emergency Services to develop internal evaluation capacity. Insufficient attention has been paid to organisational support processes to nurture individuals’ evaluation skills and abilities, and to assist complex organisations utilise evaluations. For evaluation to become an embedded and systematic activity, it has to be supported by a project management methodology that underpins evaluation processes.

Citation

Koljonen, H. Building evaluation capacity in to a large public sector emergency service : an action research study. (Thesis). University of Salford

• Thesis Type: Thesis
• Deposit Date: Nov 8, 2013
• Publicly Available Date: Nov 8, 2013