Evaluating the effectives of IDLO's training activities
In 2009 IDLO launched an institutional approach to evaluate the effectiveness of its training activities. The approach is based on the model developed by Donald Kirkpatrick in the 1950s, and outlines four levels of results at which training is expected to produce change in its beneficiaries: “Level 1” – satisfaction with the training, “Level 2” – new knowledge acquired, “Level 3” – change in professional behavior and performance, and “Level 4” – change in institutional performance and capacity obtained through reaching a sufficiently critical mass of individuals.
The methodology to assess Levels 1 and 2 is standard in IDLO’s approach and is based on questionnaires and knowledge tests. Levels 3 and 4 depend on the specific objectives and target groups of each program. Traditionally, performance change has been assessed only upon completion of the programs, in the context of final evaluations. Changes in beneficiaries’ behaviors are often difficult to assess, and program evaluations can mainly rely on “self-assessments” of beneficiaries, whose opinions are often subjective and difficult to verify. In order to provide more objective measurement of training effectiveness, and to make its results available to the continuous program internal learning, JTTP developed an innovative approach to monitor Level 3 changes produced by training and follow-up coaching on participants’ performance at work.
Change in professional behaviour and performance (Level 3)
The approach is based on objectively verified standards established for different categories of justice professionals against specific functions that they are expected to carry out in the management of criminal proceedings.
For each of the five categories targeted by JTTP – judges, defense lawyers, trial and investigative prosecutors, and police officers – the team prepared checklists outlining expected behaviors that would ensure fair trial and justice to the accused in line with the Afghan Criminal Procedure Code, and then verified the extent to which individuals in each group aligned to identified best practices. The data has been used primarily for planning and management of the training and coaching activities: it identifies gaps in the capacity of the target legal professionals, which are considered when arranging future training, and helps identify coaching needs of training graduates.
The program team completed the checklists through interviews with justice professionals and their supervisors. The information collected was further cross-checked through file records, direct observation of proceedings in court and interviews with other stakeholders. The information was then synthesized and shared with the program team and target institutions.
Baseline information was gathered from professionals who had not been trained, and was later compared with data acquired from training graduates who also benefited from follow-up coaching. JTTP collected data from 619 justice sector professionals, which include 322 non trained professionals (pre training), and 297 graduates (post training). Women professionals represented 11% of the completed checklists. The training is considered more effective when justice professionals meet higher standards of compliance with the performance requirements established in the Criminal Procedure Code. In particular, JTTP set its standards at the highest level (80% or more of the practices listed in the checklists).
On average, 54% of the participants who were reached by JTTP training and coaching meet 80% or above of the required standard practices, against the 16% achieved by professionals who did not attend any training provided by JTTP. The most prominent change can be observed among trial prosecutors: data shows that 69% are practicing at the top level of performance, which is reached by only 11% of those who did not attend the training. Likewise, data for Crime Investigation Department (CID) officers also reflected a noteworthy change in their work- related behavior as a result of JTTP training and coaching. Overall, 48% of the CID officers interviewed meet the highest professional standards, against the 11% who do so before participating in JTTP training.
Other targeted professionals like investigative prosecutors and defense lawyers also demonstrate a positive difference in their behaviors. Analyses show that 63%of investigative prosecutors, and 31% of defense lawyers who were reached by the program are highly compliant. Finally, judges have attained 29% improvement in their behavior following JTTP training and coaching sessions.
Daniela Di Lorenzo, Program Manager, Afghanistan
Manuela Marin, Program Monitoring, Evaluation and Reporting Officer