Government of Canada

Foreign Affairs, Trade and Development Canada

www.international.gc.ca

ARCHIVED - Review of the Effectiveness of CIDA’s Multilateral Delivery Channel

Warning This Web page has been archived on the Web.

Archived Content

Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated after the date of archiving. Web pages that are archived on the Web are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats by contacting us.



List of Acronyms
Executive Summary
1.0 Objectives of the Review
2.0 The Multilateral Channel
2.1 Scope of the Review 3.0 Methodology Part One: Review of Evaluations 4.0 Methodology Part Two: Other Sources 5.0 Results Part One: Review of Evaluations 6.0 Managing for Effectiveness: Multilateral Organizations 7.0 Managing for Effectiveness: CIDA's Role 8.0 Conclusions and Recommendations 9.0 Management Response
Appendix 1: CIDA Disbursements by Funding Type (2001/02 - 2006/07)
Appendix 2: Evaluation Sample
Appendix 3: Evaluation Scoring Chart
Appendix 4: Detailed Results of Evaluation Review
Appendix 5: Multilateral Organization Review Results



List of Acronyms


AfDB
African Development Bank


CGIAR
Consultative Group on International Agricultural Research


CIDA
Canadian International Development Agency


COMPAS
Common Performance Assessment System


EBRD
European Bank for Reconstruction and Development


ECG
Evaluation Cooperation Group


FAO
Food and Agriculture Organization


GAVI
Global Alliance for Vaccines and Immunization


GEF
Global Environment Facility


GFATM
Global Fund to Fight AIDS, Tuberculosis and Malaria


GPS
Good Practice Standards


HIV/AIDS
Human Immunodeficiency Virus/Acquired immunodeficiency syndrome


IAWG
Inter-Agency Working Group


IBRD
International Bank for Reconstruction and Development


ICRC
International Commission of the Red Cross


IFAD
International Fund for Agricultural Development


IFC
International Finance Corporation


IFRC
International Federation of Red Cross and Red Crescent Societies


IFI
International Financial Institution


MDB
Multilateral Development Banks


MDGs
Millennium Development Goals


MERA
Multilateral Effectiveness and Relevance Assessment


MfDR
Managing for Development Results


MOPAN
Multilateral Performance Assessment Network


OAS
Organization of American States


OCHA
Office for the Coordination of Humanitarian Affairs


RBM
Results-based Management


RMAF
Results Management and Accountability Framework


SGPB
Sectors and Global Partnerships Branch


UN
United Nations


UNAIDS
United Nations Joint Program on HIV/AIDS


UNDP
United Nations Development Program


UNEG
United Nations Evaluation Group


UNFPA
United Nations Population Fund


UNHCR
United Nations High Commissioner for Refugees


UNICEF
United Nations Children's Fund


UNMAS
United Nations Mine Action Service


UNRWA
United Nations Relief and Works Agency for Palestine Refugees in the Near East


WFP
World Food Program


WHO
World Health Organization





Executive Summary

Objectives of the Review

The objectives of the Review of the Effectiveness of CIDA's Multilateral Delivery Channel (the Review) were to:
  • Profile CIDA's use of the multilateral channel;

  • Develop clear and defensible findings on the effectiveness of the multilateral channel;

  • Identify and assess efforts by CIDA's key partner multilateral institutions in improving effectiveness management (including reporting on effectiveness, results-based management and monitoring and evaluation) over the period under review;

  • Document CIDA's role in providing support and/or guidance to key multilateral organizations in order to improve the effectiveness of the channel;

  • Develop a brief set of recommendations aimed at increasing the effectiveness of the multilateral channel; and

  • Contribute to CIDA's response to the request from the Treasury Board Secretariat for a review on the multilateral channel.

Methodology

The Review employed three principal methodologies: a systematic review of published evaluations of multilateral organizations and their programs; interviews of staff of CIDA and over twenty key partner multilateral organizations; and a review of policy and reporting documents relating to managing for development and humanitarian effectiveness.

The Review of evaluations encompassed a representative sample of 117 evaluation reports published between April 2004 and April 2008 including 34 evaluations of International Financial Institutions (IFI) and their programs, 66 evaluations of United Nations (UN) organizations and their programs and 17 evaluations of the programs of other organizations including the Red Cross agencies.

Evaluations were assessed against five criteria: relevance, objectives achievement, cost effectiveness, sustainability and some aspects of internal management, namely results-based management (RBM), evaluation, effectiveness reporting and knowledge management. Evaluation findings under each criterion were classified as highly unsatisfactory, unsatisfactory, satisfactory or highly satisfactory. The Evaluation Scoring Chart is outlined in Appendix 3 to the Report.

The Review team undertook interviews with 21 multilateral organizations accounting for over 87% of the $7.6 billion which CIDA invested through the multilateral channel between 2001/02 and 2006/07. They supplemented those interviews with an analysis of each organization's documents on RBM, evaluation and effectiveness reporting. This resulted in the Review assigning a score to each agency relating to managing for effectiveness.

Interviews and document reviews were also used to document CIDA's role and contribution in influencing multilateral organizations to improve their effectiveness in development and humanitarian programming.

Limitations

It is important to note several significant limitations of the methodologies used by the Review. These limitations do not prevent the Review from meeting its objectives but they do mean care should be taken in how the results are interpreted.
  • The conclusions reported below on development and humanitarian effectiveness are based on a sample of published evaluations and are relevant in assessing the effectiveness of multilateral organizations as a group or in significant sub-groups such as International Financial Institutions or United Nations agencies. They are not valid and have not been used to address the effectiveness of individual organizations;

  • The review of evaluations did not impose its own definition of the five effectiveness criteria on the evaluations themselves but rather relied on definitions used by the institutions themselves. The Review was, however, able to develop and implement very clear guidelines on how evaluation findings were to be classified for each performance category (unsatisfactory, satisfactory, etc.):

    • Some internal management systems were assessed only in relation to systems directly involved in managing for development effectiveness: RBM, evaluation, effectiveness reporting and knowledge management;

    • The assessment of these systems by the Review team was based on a very small set of interviews for each organization supplemented by a review of agency documents. They should not be read as definitive for a given organization but are indicative of the overall trend towards strengthening managing for effectiveness among CIDA's key multilateral partners; and

    • The assessment of some aspects of internal management systems is not capable of identifying strong or weak performing multilateral organizations and was not intended to do so. Rather it provides a brief assessment of the current status of selected systems for managing for development effectiveness and, more importantly, the current trend to strengthening those systems within the multilateral channel as a whole.

Findings and Conclusions

Development and Humanitarian Effectiveness

Based on the contents of a sample of 117 evaluations, using an aggregate measure (an averaging of ratings for objectives achievement, relevance, cost effectiveness, and sustainability), 69 percent of the multilateral organization programs evaluated were either highly satisfactory (3%) or satisfactory (66%) in development and humanitarian effectiveness; 30 percent of the multilateral programs evaluated were unsatisfactory and 1 percent highly unsatisfactory.

Distribution of Evaluated Initiatives by Performance Category

Criteria Percent in Performance Category
N HU U S HS Total
Objectives Achievement 114 1 % 28 % 67,5 % 3,5 % 100 %
Relevance 113 0 % 12 % 83 % 5 % 100 %
Cost Effectiveness 67 0 % 43 % 54 % 3 % 100 %
Likelihood of Sustainability 106 3 % 39 % 59 % 0 % 100 %
Development and Humanitarian Effectiveness   1 % 30 % 66 % 3 % 100 %

N= Number of Observations, HU = Highly Unsatisfactory, U = Unsatisfactory, S=Satisfactory, HS=Highly Satisfactory.

For most of the criteria assessed by the Review, United Nations agencies and International Financial Institutions (IFI) were broadly similar in the portion of evaluations rating their programs satisfactory or better. For relevance this was true for 82 percent of IFI programs and 92 percent of UN agency programs. Turning to objectives achievement the figures are 70 percent satisfactory or better for IFIs and 67 percent for UN agencies. With regard to sustainability, results were more often satisfactory for UN agency programs (56%) than for IFIs (49%).

The small number of evaluations which attempted to assess cost effectiveness meant that only an overall rating for all agencies (57% rated satisfactory or highly satisfactory) was developed by the Review.

Managing for Effectiveness: Multilateral Organizations

The strength of internal management systems with a direct link to development effectiveness was assessed both through the review of published evaluation reports and by direct assessment by the Review team using interviews and a structured review of agency policy documents, reports and guidelines.

The evaluations reviewed rated multilateral programs satisfactory or better in relation to strategic planning and priority setting 48% of the time. Results were worse for results-based management (RBM) with 23% and evaluation services with 24% rated satisfactory or better. Knowledge management was rated satisfactory or better in a slightly more encouraging 31% of the evaluations reviewed.

These negative internal management findings may result from some evaluations in the sample covering programs and projects prior to 2004. They may also reflect the decentralized nature of evaluation operations in many multilateral organizations (with many evaluations initiated and managed at country office level) and the difficulties this creates for enforcing evaluation standards and applying quality assurance procedures.

When the Review conducted its own direct analysis of RBM, evaluation, and effectiveness reporting systems in the most important of CIDA's multilateral partner organizations, results were much more positive than those reported in the evaluations.

There are two main explanatory factors for the difference between published evaluation results and the direct assessment undertaken by the Review team.
  1. The fact that many of the published evaluations will have assessed the operation of RBM and evaluation systems and processes at a decentralized level while the Review team assessed them from a headquarters perspective; and,

  2. The fact that some of the evaluations reviewed were from earlier in the period under review (2004) and will have focused on systems and procedures in place even before then, up to five years before their publication date. They will not have captured the significant efforts to improve these systems in more recent years in many organizations.
Over half (11 of 21) of the multilateral organizations directly reviewed for the strength of effectiveness reporting, RBM and evaluation systems were rated strongly (2.0 out of a possible 3.0), while over three quarters (17 of 21) achieved an acceptable 1.5 scoring. Only 3 multilateral organizations reviewed were rated at 1.0 or less, which is clearly an unsatisfactory rating. In the rating system applied by the Review, IFI organizations were ranked highest, while political organizations were ranked at the lowest level.

While UN organizations tended to be in the middle set of scores for RBM, evaluation and results reporting systems, there is no structural impediment that stops them from being rated higher. This can be illustrated by the World Health Organization (WHO), which received the highest possible rating. Similarly, the United Nations Development Programme (UNDP) may be expected to move to such a rating when its new Results Management and Accountability Framework (RMAF) is fully implemented.

Most importantly, there is a strong trend among multilateral organizations to strengthen systems for RBM, evaluation and results reporting and to link them more closely to program design and implementation processes. Of the 21 organizations reviewed, 15 were classified as positive in terms of their effort to assign resources and to make progress in results definition, results reporting, independent and objective evaluation and, linking performance information to programs and budgets.

At a system wide level, UN agencies and IFIs have made significant efforts to strengthen evaluation and results reporting. The United Nations Evaluation Group (UNEG), for example, has made considerable progress since 2004 in:
  • promoting professionalism among UN evaluation staff;
  • developing and popularizing norms and standards for evaluation to protect and advance its independence and relevance; and,
  • ensuring that evaluation plays an important role in United Nations Reform.
The Evaluation Cooperation Group (ECG) performs similar functions for the IFIs. The ECG has developed Good Practice Standards (GPS) for evaluations carried out by its members in relation to different types of lending (public sector investment, policy based lending, and country assistance lending for example). It also regularly conducts benchmarking studies to assess the extent that member organizations are applying the GPS.

In the same time period, the IFIs have also cooperated in the development and operation of the Common Performance Assessment System (COMPAS) which focuses on reporting organizational effectiveness in managing for development effectiveness at country level.

The Review finds that multilateral agencies are making significant investments in strengthening systems for managing effectiveness, especially RBM, evaluation and results reporting systems. This is being done on both an individual agency (with some notable exceptions) and a system-wide basis. It will be essential for CIDA and other donors to find ways to monitor these developments and to see if they translate to better performance in development and humanitarian effectiveness.

Managing for Effectiveness: CIDA's Role

CIDA's Sectors and Global Partnerships Branch (SGPB) has made a significant effort to strengthen its ability to monitor the effectiveness of multilateral organizations1 This has included, for example, its own internal monitoring processes including the field survey of multilateral effectiveness and the Multilateral Effectiveness and Results Assessment (MERA) process.

In 2008, SGPB undertook an enhanced and more rigorous exercise under the heading of the Multilateral Program Review. This process is ongoing and was conducted with the specific purpose of providing the Minister with CIDA's most current evidence on the roles and comparative strengths and weaknesses of the organizations which make up the multilateral channel.

As well as implementing its own internal monitoring systems and processes, CIDA has consistently supported and made use of the Multilateral Organization Performance Assessment Network (MOPAN). Most recently, CIDA participated in the Working Group established by MOPAN to carry forward the work of implanting its strengthened Common Approach in 2009.

The Review has confirmed from interviews with multilateral organizations that CIDA has been a consistent and effective advocate for increased investment in systems for strengthening management for effectiveness, especially including RBM, evaluation, and effectiveness reporting at the outcome and impact level.

CIDA has also made important contributions to the drafting and acceptance of action-oriented decisions by executive boards of multilateral organizations: decisions requiring agencies to strengthen their systems for managing for effectiveness.

On the other hand, the staff of multilateral organizations interviewed by the Review team often reported that they were not aware of a specific CIDA plan or strategy relating to how their own organization should evolve over time. While they appreciated CIDA advocacy for improved systems for RBM, evaluation and effectiveness reporting, they were not able to translate that support into a CIDA (or a broader Canadian) strategy for the organization itself.

Recommendations

In light of its overall findings and conclusions, the Review of the Effectiveness of CIDA's Multilateral Delivery Channel recommends that CIDA:
  1. Develop a concise organizational strategy for each multilateral organization where significant funding is channeled, with a view to providing more clarity to the organization on CIDA's goals and expectations. As the lead within CIDA for relations with multilateral partners, Sectors and Global Partnerships Branch would be well placed to prepare these strategies. These strategies should also promote greater consistency across different CIDA branches and offices as they communicate their views on agency priorities.

  2. Explore with other bilateral donor agencies the feasibility of undertaking a similar review of evaluations for assessing the effectiveness of individual multilateral organizations. Any resulting proposed approach should build on work undertaken under the MOPAN Common Approach currently in its testing phase.

  3. Continue to underscore to multilateral institutions the importance of strengthening systems and processes for managing for effectiveness, including RBM, evaluation and effectiveness reporting.

  4. Encourage UN agencies to improve their capacity to provide quality assurance and technical assistance to strengthen the quality of decentralized evaluations at country office and regional level.

  5. The Review recognizes that CIDA's ability to influence the content and direction of programming by multilateral organizations varies depending, among other factors, on the form of support CIDA provides: core funding, initiative specific funding or multi-bilateral funding. While opportunities will be limited when providing core funding, it is recommended that CIDA place strong emphasis on programming factors which evaluations indicate are strongly linked to increased effectiveness when it engages in providing core funding, supporting specific initiatives or providing multi-bilateral support. These include the following:

    1. Emphasis on harmonization, national ownership and the principles of the Paris Declaration;

    2. Effective and sustained capacity development support to national institutions and to local communities in planning and priority setting and in the operation and maintenance of assets and services following program completion;

    3. Greater participation by both civil society organizations and the private sector in planning and priority setting and in program operations;

    4. Clear and consistent targeting of program services; and,

    5. Use of cost benefit and economic sustainability analysis in program design as well as local sourcing of inputs and more frequent use of private-public partnerships. In general there is a strong argument in the evaluations for greater emphasis on cost effectiveness in program design and in evaluation.

1.0 Objectives of the Review

The objectives of the Review of the Effectiveness of CIDA's Multilateral Delivery Channel (the Review) were to:
  • Profile CIDA's use of the multilateral channel;

  • Develop clear and defensible findings on the effectiveness of the multilateral channel;

  • Identify and assess efforts by CIDA's key partner multilateral institutions in improving effectiveness management (including reporting on effectiveness, results-based management and monitoring and evaluation) over the period under review;

  • Document CIDA's role in providing support and/or guidance to key multilateral organizations in order to improve the effectiveness of the channel;

  • Develop a brief set of recommendations aimed at increasing the effectiveness of the multilateral channel; and

  • Contribute to CIDA's response to the request from the Treasury Board Secretariat for a review on the multilateral channel.



2.0 The Multilateral Channel

2.1 Scope of the Review

The Review encompassed all of the different types of multilateral organizations through which CIDA invests resources as well as the different funding mechanisms used (core funding, initiative-specific funding and multi-bi funding). For the purpose of this exercise, initiative-specific funding refers to multilateral funding earmarked by CIDA to a particular initiative of a multilateral organization, while multi-bi funding refers to the use of bilateral funding to support activities of a multilateral organization in a specific developing country or region.

The Review focused on the six-year period from 2001/02 to 2006/07 in order to have access to the strongest possible information and evidence base on effectiveness.

2.2 CIDA Investments in the Multilateral Channel

All forms of CIDA funding through multilateral organizations accounted for $7.6 billion in the period 2001/02 to 2006/07 with Sectors and Global Partnerships Branch (SGPB—formerly Multilateral Programs Branch) accounting for $5.5 billion (72 percent) and multi-bi funding programmed through country and regional programs accounting for $2.1 billion (28 percent). Multi-bi funding is a rising portion of all CIDA multilateral funding, from 15.4 percent in 2001/02 to 40.7 percent in 2006/07.

SGPB funding has remained relatively stable across different programs and types of funding with fairly strong concentration of funding in a small set of organizations.

In conducting the Review, CIDA has targeted its efforts toward assessing the performance of a core set of multilateral organizations. By combing data on the most heavily-funded organizations, it was possible to identify a set of 23 multilateral organizations accounting for $6.8 billion in funding or 89.7 percent of all CIDA funding through the multilateral channel over the six- year period. Table 2.1 identifies the key organizations which were the focus of the Review and the total CIDA investment they received in the six years from 2001/02 to 2006/07.

Table 2.1: CIDA Disbursements to 23 Multilateral Organizations, 2001 to 2006-07

Rank Multilateral Organization Total Funding
1 World Bank (including IFC and IBRD) $974,977,333
2 World Food Programme $795,449,307
3 United Nations Development Program $694,850,945
4 UNICEF $606,626,431
5 Asian Development Bank $581,899,723
6 GFATM $528,528,249
7 African Development Bank $498,067,344
8 World Health Organization (including Stop TB Partnership) $479,884,338
9 Global Environmental Facility $302,401,001
10 United Nations High Commissioner for Refugees $195,130,663
11 GAVI - UNICEF $188,000,000
12 United Nations Population Fund $179,573,827
13 International Committee of the Red Cross $145,638,200
14 International Fund for Agricultural Development $124,255,749
15 UNRWA $111,484,246
16 Consultative Group on International Agricultural Research $80,620,000
17 Commonwealth Fund for Technical Cooperation $76,565,000
18 United Nations Mine Action Service $53,226,000
19 European Bank for Reconstruction and Development $52,470,726
20 Office for the Coordination of Humanitarian Affairs $44,092,000
21 International Federation of Red Cross and Red Crescent Societies $42,265,570
22 Organization of American States $40,842,468
23 Red Cross International Aid Trust of Canada $30,240,399
Total $6,827,089,519

For a profile of CIDA investments channeled through multilateral organizations from 2001/02 to 2006/07 by funding type, see Appendix 1.


3.0 Methodology Part One: Review of Evaluations

The methodology for the Review included a systematic review of completed and published evaluation reports on the field effectiveness of the 23 core multilateral agencies listed in Table 2.1. The evaluations to be surveyed included:
  • Evaluations carried out by and for United Nations Agencies among the 23 organizations;

  • Evaluations carried out by and for International Financial Institutions among the 23 organizations;

  • Institutional and global program evaluations carried out on behalf of more than one donor agency with CIDA taking an active role; and,

  • Evaluations of CIDA supported multi-bilateral programs and projects.

3.1 Sample Design

An on-line review of published evaluations as of May 2008 identified over 300 evaluation reports either publically available or accessible on request by the evaluation team. These included a large number of project evaluation reports which were too small in scope to be used as a measure of field effectiveness on anything but a very local scale. As a result, the list was reduced to just over 200 evaluation reports by listing evaluations which covered:
  • An entire multilateral institution (such as the Independent External Evaluation of the International Fund for Agricultural Development (IFAD);

  • A global or regional sector-specific program implemented by one of the 23 organizations; or,

  • An evaluation of a large scale country program of work implemented by one of the 23 organizations.
A final representative sample of 117 evaluation reports published between April 2004 and April 2008 was selected from the universe of available studies and reviewed by a team of four analysts under the direction of the team leader. The evaluations in the sample included:
  • 34 Evaluations of International Financial Institutions and their programs including the World Bank, Asian Development Bank, African Development Bank, the International Fund for Agricultural Development and the European Bank for Reconstruction and Development);

  • 66 Evaluations of United Nations Organizations and their programs including UNDP, UNICEF, GFATM, WHO, GAVI, FAO, UNFPA, UNHCR, CGIAR, UNMAS and OCHA; and

  • 17 evaluations of other organizations including large organizations such as the OAS and much smaller agencies and programs receiving multi-bilateral program funding including: the Canadian Foodgrains Bank, the Kofi Annan International Peacekeeping Training Centre, the National Solidarity Program in Afghanistan and the South Asia Enterprise Development Facility .
The large number of UN organization evaluations resulted from three key factors:
  • The fact that 13 of the 23 organizations are UN agencies;

  • The necessity of including a number of evaluations of multi-bilateral programming which is often channeled through UN agencies; and,

  • The desire to include a number of multi-donor and country led evaluations supported by CIDA which have often focused on UN organizations.
Of the 117 evaluations in the sample, 19 were joint evaluations in which CIDA was either a major supporter or took the lead and 14 were focused on CIDA's use of the multi-bilateral mechanism (undertaking part of its bilateral program in a country by using a multilateral institution as an implementing agency).

The full sample of evaluations is provided in Appendix 2 to the Report.

3.2 Criteria and Scoring

The findings in each evaluation were assessed in relation to five specific criteria and their components, as in Table 3.1:

Table 3.1: Evaluation Review Criteria

Main Criteria Components
1. Relevance 1.1 Agency development programs and projects are aligned with national and international development goals.
1.2 Agency humanitarian activities are coordinated with other agencies and align with national priorities.
1.3 Agency takes part in local planning and coordination bodies.
2. Aspects of Internal Management 2.1 Effective agency level strategic planning and priority setting.
2.2 Effective results-based management systems.
2.3 Reliable and useful program evaluation systems and processes.
2.4 Effective knowledge management systems and processes.
3. Objectives Achievement 3.1 Extent development and humanitarian program and project objectives are achieved.
4. Cost Effectiveness 4.1 Agency programs and projects are cost effective.
5. Sustainability 5.1 Benefits accrued to participants will continue after project or program closure.
5.2 Agency projects and programs address the capacity of local institutions to sustain results.
5.3 Agency projects and programs provide adequate human and other resources (including host government resources).

The evaluation findings of each reviewed evaluation as they related to these criteria were then classified on a four-point scale (highly unsatisfactory, unsatisfactory, satisfactory and highly satisfactory). In addition, each analyst provided comments gathered from each evaluation regarding factors which led to the classification and which may have contributed to a satisfactory or unsatisfactory rating. The Evaluation Scoring Chart is outlined in Appendix 3 to the Report.

The findings on internal management were then analyzed in conjunction with the interviews and document reviews carried out as described in Section 4.

3.3 Process and Quality Assurance

The Review team first developed a basic evaluation scoring chart (see Appendix  3). This was then accompanied by a detailed, paragraph long definition for each level of scoring to be provided under each of the evaluation sub-criteria. Using the scoring chart and accompanying scoring guide, the team members then proceeded to analyze a test evaluation and to make adjustments in their scoring. In addition, the team leader pulled evaluation scoring grids and compared them to the findings in the actual evaluation reports at different occasions throughout the process. Finally, the team leader reviewed all 117 completed scoring sheets for consistency of interpretation of the Review criteria and the attached commentary. In very few instances he made small adjustments in the scoring to ensure continued consistency of application across the analysts on the team.

The results of the meta-evaluation are presented in Sections 5.0 and 6.0. Detailed tables of results are presented in Appendix 4.

3.4 Limitations

While the Review was able to access a good sample of evaluations representing approximately half of the published evaluations meeting the criteria used for selection (published, showing evidence of independence, meeting quality scoring criteria, and covering programs at an institutional, global, regional or national level) there are still some important limitations which inevitably arise in this type of meta-review:
  • While the overall number of evaluations is reasonable, the number of cases becomes much smaller when the analysis compares results for different types of organizations (IFIs and UN agencies for example);

  • The sample size does not allow the review to estimate the performance of specific agencies or identify high performing and low performing agencies (and indeed the review was never intended to do so). Results are meaningful only for the group of agencies covered in the evaluation and for some classifications and some issues (for example IFIs and UN agencies);

  • Not all evaluations cover all of the criteria used by the Review. Cost effectiveness, for example, was covered in only 67 of the 117 evaluations in the sample. Furthermore, only some of the factors reported as affecting cost-effectiveness are under the control of the organizations;

  • As for any meta-evaluation, the precise definition of what constitutes a given criterion depends to a large extent on the definitions used in each evaluation. The review first assessed the overall quality of the evaluation report and then determined if a given criterion had been addressed before classifying the findings;

  • In order to ensure consistency in classifying the findings for each of the criteria and sub-criteria, the team used very precise definitions of the evidence required for each effectiveness rating (highly unsatisfactory, unsatisfactory, satisfactory and highly satisfactory) for each criterion.
The Review team has taken care to present the number of relevant evaluation cases supporting each of the findings reported. Where numbers (N) are so small as to call any findings into question this has been highlighted.


4.0 Methodology Part Two: Other Sources

4.1 Interviews

In order to supplement the meta-evaluation findings and to develop an understanding of the current state of management for development effectiveness in multilateral agencies, the Review team undertook in person interviews with 21 important multilateral organizations (including the Joint United Nations Program on HIV/AIDS (UNAIDS) and the Food and Agriculture Organization (FAO) which, while not on the list of 23 in Table 2.1, are still significant multilateral partners for CIDA. Of the 23 organizations listed in Table 2.1, the United Nations Relief and Works Agency for Palestine Refugees in the Near East (UNRWA), the Consultative Group on International Agricultural Research (CGIAR), the United Nations Mine Action Service (UNMAS) and the Red Cross International Aid Trust of Canada were not profiled.

In each case, the Review team met with staff responsible for managing for development and humanitarian results systems and staff responsible for monitoring and evaluation.

In addition, the Review team met on several occasions with CIDA personnel in headquarters and at missions abroad (Geneva, London, New York, Rome and Washington) responsible for managing cooperation with multilateral organizations to discuss how CIDA staff interact in the governance and support of these institutions.

In particular, interviews with CIDA staff focused on how CIDA's efforts are aimed at improving the effectiveness of key institutions that make up the multilateral channel. Their responses were cross-checked against the views of staff of the agencies concerned.

4.2 Document Reviews

In addition to the meta-evaluation described in Section 3.0, the Review team gathered and analyzed institutional documents relating directly to how multilateral organizations define and report on development effectiveness including annual development results reports, reports to their governing bodies, and policy documents and guidelines on systems and approaches for managing for development effectiveness. These documents were used to provide a cross-check to the results of the interviews and to provide concrete illustrations of some of the organizational advances and reforms reported in the interviews.

4.3 Limitations

Given the breadth of the multilateral channel in relation to the time and resources available to the Review team, neither the interviews nor the document reviews can be viewed as comprehensive. In each multilateral organization, the team conducted interviews with professional staff working either directly or in support of evaluation and RBM functions but only at headquarters level. In some organizations the meetings also included staff responsible for donor relations and staff responsible for strategic planning and relations with the governing body. However, in a small group of organizations, only one staff person (usually in evaluation or RBM) was able to meet with the team.

Similarly, the document review, while reasonably extensive, did not cover all available reports on development and humanitarian effectiveness available from all the organizations. The Review team necessarily chose the most recent key documents describing systems for managing for effectiveness in each organization along with key annual effectiveness reports.

The limitations of the interview and document review process are especially important when interpreting the results on managing for effectiveness by multilateral organizations as presented in Section 6.1.

In conducting the interviews and document review, the Review team focused on three key systems of internal management: results-based management (RBM), program and project evaluation, and effectiveness reporting. These systems were chosen for review because they correspond to those systems recognized for their direct link to managing for development effectiveness. The Review did not assess other internal management and governance systems such as project planning and approval, corporate governance, financial management or human resources management.


5.0 Results Part One: Review of Evaluations

5.1 Overall Effectiveness

The Review produced results across four criteria based on the representative sample of 117  evaluations, namely objectives achievement, relevance, cost effectiveness and the likelihood of sustainability, as illustrated in Table 5.1. For the purpose of this Review, overall development effectiveness is therefore defined as achieving expected results that are relevant and likely to be sustainable, at a reasonable cost.

Based on the contents of a sample of evaluations, using an aggregate measure (an averaging of ratings for objectives achievement, relevance, cost effectiveness, and sustainability), 69 percent of the multilateral organization programs evaluated were either highly satisfactory (3 percent) or satisfactory (66 percent) in development and humanitarian effectiveness; 30 percent of the multilateral programs evaluated were unsatisfactory and 1 percent highly unsatisfactory.

Table 5.1: Distribution of Evaluated Initiatives by Performance Category

Criteria Percent in Performance Category
N HU U S HS Total
Objectives Achievement 114 1 % 28 % 68 % 4 % 100 %
Relevance 113 0 % 12 % 83 % 5 % 100 %
Cost Effectiveness 67 0 % 43 % 54 % 3 % 100 %
Likelihood of Sustainability 106 3 % 39 % 59 % 0 % 100 %
Development and Humanitarian Effectiveness   1 % 30 % 66 % 3 % 100 %

N= Number of Observations, HU = Highly Unsatisfactory, U = Unsatisfactory, S=Satisfactory, HS=Highly Satisfactory.

5.2 Relevance

In all three sub-categories of relevance, multilateral organizations scored very highly in the evaluations reviewed.

For sub-criterion 1.1: alignment with national and international development goals, 88 percent of the evaluations which addressed this issue rated the agency concerned as satisfactory or highly satisfactory. Interestingly, the UN agencies ratings were somewhat higher than the International Financial Institutions, with the IFIs at 82 percent and the UN agencies at 92 percent.

Table 5.2 Programs Align with National and International Development Goals

Rating IFI UN Other Total N %
Highly Unsatisfactory 0.0% 0.0% 0.0% 0 0.0%
Unsatisfactory 17.6% 8.1% 11.8% 13 11.5%
Satisfactory 76.5% 87.1% 82.4% 94 83.2%
Highly Satisfactory 5.9% 4.8% 5.9% 6 5.3%
Total 100.0% 100.0% 100.0% 113 100.0%

Sub-criterion 1.2, referring to agency coordination of their humanitarian activities with other agencies and alignment with national priorities, was addressed in a smaller number of evaluations (83) but produced similarly positive results, 88 percent of the evaluations which covered this issue reported that agency programs and projects were either satisfactory or highly satisfactory.

Sub-criterion 1.3 (Table 5.3) addressed the extent agencies take part in local planning and coordination bodies such as development partner working groups which bring together multilateral and bilateral development organizations and government bodies in each country. Here the results were less positive than for the first two sub-criteria.

Only 56 percent of IFI program evaluations rated performance in this area satisfactory, while 65 percent of UN evaluations gave the same satisfactory scoring. When results for all the agencies are considered; 68 percent of the evaluations rated programs satisfactory while 32 percent rated them unsatisfactory.

Table 5.3 Agency Takes Part in Local Planning and Coordinating Bodies

Rating IFI UN Other Total N
%
Highly Unsatisfactory 0.0% 1.7% 0.0% 1 0.9%
Unsatisfactory 43.8% 33.3% 0.0% 34 31.5%
Satisfactory 56.3% 65.0% 100.0% 73 67.6%
Highly Satisfactory 0.0% 0.0% 0.0% 0 0.0%
Total 100.0% 100.0% 100.0% 108 100.0%

On balance and when viewed across all three sub-criteria, the evaluations reviewed indicate that multilateral agencies as a group score highly in area of program relevance. Their programs are largely aligned with national and international development goals and they score well with regard to coordinating humanitarian activities.

Important factors contributing to the relevance of programs and projects delivered by multilateral institutions were noted in most of the evaluations reviewed. Some of the most frequently cited factors which tend to strengthen relevance are:
  • The accepted importance of the Millennium Development Goals (MDGs) for developing country governments, bilateral donor agencies and multilateral organizations;

  • The continuing evolution of national poverty reduction strategies which provide a stronger framework for all agencies to align their programs with national priorities;

  • The influence of the Paris Declaration requirements on multilateral organizations as well as on host governments which puts a priority on national ownership and alignment with national goals;

  • The emergence of coordinating forums at country level such as development partner (donor) working groups at a sector level (health, education, HIV/AIDS, etc.) in many countries; and,

  • A greater interest on the part of multilateral institutions (and a greater acceptance by national governments in developing countries) to include civil society organizations in dialogue on program and sector priorities.
Where evaluations have reported less satisfactory performance in the area of program relevance, especially relating to coordination they have sometimes identified contributing factors. Relevance tends to diminish whenever:
  • There is a lack of integration of national and local government agencies in the host country during the planning and development of multilateral agency supported projects and programs;

  • There is insufficient capacity development work to support the capacity of national and local agencies to allow them to take an effective role in planning and development of programs;

  • Multilateral organizations sometimes are not able to respond well to the current trend in many countries to decentralized project and program authority to provincial and local government entities; and,

  • The most frequently cited negative factor tending to diminish program relevance and effective coordination is the failure to provide adequate access and to promote effective participation by civil society organizations in developing countries in planning and management of programs supported by multilateral institutions.
One could reasonably argue that the last point in this list, adequate participation by civil society, is the responsibility of governments receiving support from multilateral organizations since they control national development planning processes. Nonetheless, the evaluations illustrate the importance of securing this participation if support from multilateral organizations is to prove effective.

5.3 Achieving Development Objectives

Perhaps the most important criterion for assessing the effectiveness of the multilateral channel at CIDA concerns whether the programs and projects implemented or supported by multilateral organizations achieve their development and/or humanitarian objectives. As noted above, 71 percent of the 114 evaluations reporting on this criterion classified multilateral organization performance as either satisfactory or highly satisfactory versus 28 percent unsatisfactory and 1 percent highly unsatisfactory.

Interestingly, UN organizations and International Financial Institutions scored similar results with 71 percent of IFI evaluations reporting satisfactory or highly satisfactory results and 67 percent of UN evaluations reporting the same performance. For this criterion, the organizations which are neither UN or IFI (including, for example, the ICRC and IFRC) were rated satisfactory or highly satisfactory in 88 percent of the evaluations concerning those agencies. But with only 17 of those evaluations in the body of 114 reviewed, it is important to take care in interpreting this result.

Table 5.4 Achievement of Objectives

Rating IFI UN Other Total N Total %
Highly Unsatisfactory 3.0% 0.0% 0.0% 1 0.9%
Unsatisfactory 27.3% 32.8% 11.8% 32 28.1%
Satisfactory 69.7% 65.6% 70.6% 77 67.5%
Highly Satisfactory 0.0% 1.6% 17.6% 4 3.5%
Total 100.0% 100.0% 100.0% 114 100.0%

The evaluations reviewed did point to some important factors which are positively correlated with higher probability of objectives achievement. They include:
  • Flexibility and agility on the part of some multilateral organizations in allowing program priority setting to be partially decentralized so that they can reflect local development and humanitarian conditions;

  • A long term commitment to capacity building efforts so that national and local partners in developing countries can increase their effectiveness in designing and implementing programs supported by multilateral organizations;

  • Efforts by multilateral organizations to strengthen their own capacity at country office levels has helped some to improve the quality of project and program design and to take a more effective part in policy dialogue;

  • As under relevance, a number of evaluations indicated that effectiveness was increased when civil society organizations and the private, for profit sector were able to take part in program development and implementation; and,

  • Clear and consistent measures for targeting services (especially social services in health, education and food security) to vulnerable group members.
Indeed, effective targeting and an explicit commitment to long-term capacity development for developing country institutions were the two most frequently cited factors contributing to objectives achievement.

Factors which were noted in evaluations as contributing to lower levels of objectives achievement include:
  • Failure to integrate programs supported by multilateral organizations into the wider institutional structure of host developing country governments;

  • Poor participation by all stakeholders, including civil society organizations in program planning;

  • Weak linkages from project and program inputs to actual service providers in the field;

  • An inability for some organizations to define and concentrate on their comparative advantages as development or humanitarian agencies;

  • The absence of a clearly defined set of program beneficiaries combined with poor or insufficient efforts to target benefits to vulnerable groups; and

  • A lack of consistent focus on strengthening institutional capacity among developing country agencies.
Just as effective targeting and a commitment to capacity development are cited most often in support of findings of objectives achievement, failures to deal with these two issues are most often associated with unsatisfactory ratings.

5.4 Cost Effectiveness

Cost effectiveness was the criterion which received the lowest level of attention in the evaluations reviewed. Of 117 evaluations in the sample, only 67 addressed any aspect of cost effectiveness in a substantive way. Of those 67 evaluations, 57 percent rated the cost effectiveness of multilateral programs as either satisfactory (54 percent) or highly satisfactory (3 percent). A comparison of UN and IFI agency cost effectiveness was not considered valid given the small number of observations for each category.

At the broader level of the 67 evaluations dealing with this issue, there were two strong lessons reported in relation to strengthening program and project cost effectiveness;
  • The need to include cost/benefit and/or economic sustainability analysis during the program design and approval processes;

  • The need to source key inputs such as training specialists, materials and supplies in the country where the projects and programs supported by multilateral agencies are taking place.
There was a much longer list of factors which tended to reduce the cost effectiveness of programs and projects supported by multilateral organizations, only some of which are under the control of the organizations. If addressed, these negative factors represent possible positive strategies for improving cost effectiveness. Factors contributing negatively to cost effectiveness cited in the evaluations include:
  • Excessive reliance on external funding by the national and local agencies receiving support so that any disruptions in disbursements cannot be made up from local sources;

  • Tying of inputs flowing through multilateral agencies imposed by national governments providing them, in particular technical assistance, equipment and supplies;

  • Excessively long project and program preparation and approval times;

  • Imposition of overly complex and overlapping procurement practices by multilateral agencies and bilateral donors; and,

  • Untimely disbursements of funds provided through multilateral organizations to national and local institutions in developing countries, whether a result of internal problems in the multilateral organizations or disruptions and conflicting budget cycles among bilateral donor agencies providing support.
The last point was the most frequently and strongly cited factor tending to produce unsatisfactory findings for cost effectiveness. Delays and disruptions to disbursements cause parallel increases in program costs as staff are laid off and rehired and construction started and stopped. At the same time, these disruptions reduce the benefits of many projects and in some cases are critical factors in not achieving objectives.

5.5 Sustainability

The most critical and immediate aspect of sustainability is the question of whether or not program and project benefits will persist after the project is completed. For this criterion, over half the evaluations reviewed (58.5 percent) reported satisfactory results, while 41.5 percent were either unsatisfactory or highly unsatisfactory. In this area, the results for evaluations of UN projects and programs were better (56.4 percent satisfactory) than those for International Financial Institutions (48.6 percent satisfactory). The generally smaller organizations classified under the category of "other" were much more likely to achieve higher scores with 87.5 percent of evaluations of their programs rating sustainability as satisfactory.

Table 5.5 Benefits Continue After Program Closure

Rating IFI UN Other Total N Total %
Highly Unsatisfactory 5.7% 0.0% 6.3% 3 2.8%
Unsatisfactory 45.7% 43.6% 6.3% 41 38.7%
Satisfactory 48.6% 56.4% 87.5% 62 58.5%
Highly Satisfactory 0.0% 0.0% 0.0% 0 0.0%
Total 100.0% 100.0% 100.0% 106 100.0%

As already noted, evaluations frequently reported on the need for more consistent and focused attention to capacity development as a factor impeding objectives achievement. This makes it less surprising that 61 percent of evaluations rated programs unsatisfactory under the criterion of addressing the capacity of local institutions to sustain results. In this area, the International Financial Institutions had better results (44 percent satisfactory) than did the UN organizations (32 percent satisfactory).

One important factor contributing to lack of sustainability may be found in the perceived inadequacy of the human, financial and physical resources that are provided in support of programs and projects implemented through multilateral organizations. Of the 106 evaluations which dealt with this issue, 68 percent rated the results as unsatisfactory or highly unsatisfactory.

Overall, the most frequently cited positive criteria for higher performance relating to sustainability include:
  • High visibility for a given project or program with the developing country government leading to support with staff and finances both during and after the program;
  • Use of public-private partnership arrangements to sustain infrastructure and other key programming aspects after program completion;
  • Training and capacity development in maintenance and operation of newly created assets by local communities; and,
  • A general investment in capacity development for national and local institutions maintaining assets and delivering services after program completion.
Not surprisingly, the negative factors undercutting sustainability tend to be the mirror image of the positive ones cited above:
  • Lack of focus and underinvestment in capacity development;
  • Limited human resources available to national governments and multilateral organizations;
  • Weaknesses in multilateral organizations' ability to provide technical assistance;
  • Staff turnover in country-posted multilateral organization staff; and,
  • Insufficient time allotted to fully implement programs and to develop and test models for sustainable service delivery.

6.0 Managing for Effectiveness: Multilateral Organizations

6.1 Results of the Review of Evaluations

In keeping with recent practice in assessing the effectiveness of multilateral institutions and with recent developments in the work of the Multilateral Organization Performance Assessment Network (MOPAN), the Review of published evaluations added a fifth criterion on internal management. This additional criterion looked at the strength of multilateral agency systems aimed at improving development and humanitarian program effectiveness, including:
  • Strategic planning and priority setting which enables organizations to define expected results at an overall institutional level;

  • Results-based management which should contribute to better specification of expected results, better reporting of those results and, over time, improved effectiveness;

  • Program evaluation which should serve as an objective check on the output of performance monitoring and reporting systems and which should contribute to learning and improved effectiveness over time; and,

  • Knowledge management systems which should ensure that the lessons learned are disseminated internally and among key partner organizations to contribute to improved effectiveness over time.
Results for criteria relating to selected internal management systems were among the most negative encountered in the evaluations reviewed. As Table 6.1 indicates, when all four criteria are averaged, only 31 percent of the programs evaluated achieved a satisfactory or highly satisfactory rating. Systems for reliable and useful program evaluation are rated lower than the other three sub-criteria with only 23 percent of evaluations reporting satisfactory or highly satisfactory results.

Table 6.1: Percentage Scores by Performance Category: Internal Management Systems

Criteria Percent in Performance Category
N HU U S HS Total
Effective Strategic Planning and Priority Setting 115 3 % 50 % 46 % 2 % 100 %
Effective Results Based Management System 82 16 % 61 % 22 % 1 % 100 %
Reliable and Useful Program Evaluation 104 5 % 71 % 22 % 2 % 100 %
Effective Knowledge Management 92 3 % 67 % 28 % 1 % 100 %
Average Across Internal Management Criteria   7 % 62 % 30 % 1 % 100 %

N= Number of Observations, HU = Highly Unsatisfactory, U = Unsatisfactory, S=Satisfactory, HS=Highly Satisfactory.

Systems for effective strategic planning and priority setting were most often rated satisfactory or highly satisfactory in evaluations of International Financial Institutions (57 percent) and less often for UN Agencies (44 percent).

RBM systems were more comparable, with 22 percent of IFI evaluations rating these systems satisfactory versus 21 percent for UN agencies. Program evaluation systems were in a similar range with 22 percent of IFI evaluations rating them satisfactory or better and 26 percent of UN evaluations doing the same.

Finally, knowledge management was more often rated satisfactory in IFI evaluations (38 percent) compared to those for UN agency programs (24 percent).

These negative findings for criteria relating to management systems for improving development effectiveness were surprising to the members of the Review team who have considerable experience with recent efforts at improvement in the multilateral system: efforts aimed at precisely these systems.

Some explanation for the low frequency of satisfactory findings for the strength of internal management systems in the evaluations reviewed include:
  • The spread of reviewed evaluations across a period from 2003 to 2008 with many of the more recent evaluations reflecting recent efforts to improve RBM and evaluation systems;

  • Problems among multilateral institutions regarding the quality of decentralized evaluations carried out at country office levels without adequate technical support and quality assurance from headquarters;

  • Difficulties for multilateral organizations engaged in humanitarian relief operations to develop and implement effective procedures for real-time evaluation.
The evaluations themselves refer to a number of factors which have impeded efforts to strengthen systems for managing for development effectiveness. They include:
  • A poor record of utilization of evaluation and monitoring results by some organizations which undermines support for strengthening both RBM and evaluation systems;

  • Delays in implementing results monitoring and knowledge management systems in some organizations which weakens internal support;

  • Decentralization of the monitoring and evaluation function in many organizations with inadequate technical support and quality assurance provided from headquarters and regional offices to country offices;

  • Problems arising from shared accountability and the fact that multilateral organizations provide support to programs and projects which are "owned" by national governments;

  • The need for stronger national systems of data collection and analysis which can be used to assess program effectiveness over time;

  • The lack of capacity development investments and technical support for national data systems;

  • Complexity, overlap and duplication of monitoring and evaluation systems as required by bilateral donor agencies and multilateral organizations for the same national programs; and,

  • Resource constraints limiting the necessary investments in results based management, monitoring and evaluation and knowledge management.
CIDA's experience in accessing the multilateral channel as a means of achieving results in both development assistance and humanitarian relief is not only limited to providing financial support to projects. CIDA staff members interact on a regular basis with multilateral institutions during meetings of their governing bodies, during consultations on financing (including replenishment of funds for International Financial Institutions) and during discussions on project, program and institutional budgets and programs of work. CIDA staff also dialogue with multilateral institutions regarding the adequacy of their systems for managing for effectiveness including RBM, program monitoring and evaluation and the general issue of reporting on development effectiveness to governing bodies and donor agencies.

In the period from 2004 to 2008, which corresponds to the period of the published evaluations reviewed, CIDA staff and external researchers working for CIDA on evaluations have seen very significant efforts by multilateral institutions to strengthen their systems for managing for effectiveness.

This made it all the more interesting to note how frequently these very systems were deemed unsatisfactory in the evaluations reviewed. RBM and evaluation systems were found to be unsatisfactory in over 75 percent of the evaluations reviewed.

6.2 Direct Review of Internal Management Systems

In addition to the assessment of published evaluations the Review team undertook a direct review of systems for managing for development effectiveness. This was based on interviews with staff responsible for RBM and evaluation functions and sometimes with those engaged in strategic planning or external relations. This was supplemented with a review of evaluation and RBM policy documents and reports to governing bodies, annual effectiveness reports, and annual summaries of evaluation activities and results.

Based on these two sources, the Review team then used a simple scoring system which assigned zero points, one half point, or a full point to each of three key elements in systems used to manage for effectiveness (RBM, program evaluation, and overall effectiveness reporting)2. A very basic rule was used for assigning a given rating to each system under review: zero points where the system was essentially so weak as to be non-functional or seriously compromised, one half point where it was functional but lacking in some element(s) necessary for full effectiveness, and a full point where the system being reviewed was clearly producing reliable results and being used by internal and/or external stakeholders.. Thus any one multilateral institution could theoretically receive a rating from 0 to 3.0.

As well as assessing the overall state of systems devoted to managing for effectiveness, the Review team arrived at an assessment of the trend in the development of these systems. Where new systems were being established with considerable resources and senior management support or where existing systems were being strengthened through investments in quality assurance or linkages to program management, the team assigned a positive trend indicator.

Whenever existing systems for managing for effectiveness were being reasonably maintained and utilized, the team assigned a neutral rating. If, on the other hand, existing systems were receiving reduced attention and resources or being downgraded in terms of organizational management and governance processes, a negative trend indicator was assigned.

It is extremely important to emphasize that the ratings assigned by the Review team were those of the team members only. While they reflect input from staff in each organization they cannot and should not be attributed to any staff members of the organizations in question. It is also crucial to recognize that the purpose of the analysis was not to identify specific strengths and weaknesses in individual organizations but to assess the evolving strength of these systems in the group as a whole.

6.2.1 RBM, Evaluation and Effectiveness Reporting


As indicated in Table 6.2, the overview of organizational systems for RBM, evaluation and effectiveness reporting developed by the Review team stands in direct contrast to the findings on the strength of internal management systems as reported in the evaluations reviewed.

Of the 21 Multilateral Organizations reviewed, 11 scored a very strong 2.0 or higher in their combined scores while a further 7 managed a 1.5 combined score which the team designated as the minimal level for a satisfactory rating. Finally, three organizations were below this threshold with 1.0 or less.

Table 6.2: Organizational Scoring on RBM, Evaluation and Effectiveness Monitoring3

Multilateral Organization RBM Evaluation Effectiveness Reporting Combined Score Trend
European Bank for Reconstruction and Development 1.0 1.0 1.0 3.0 Positive
World Bank 1.0 1.0 1.0 3.0 Positive
World Health Organization 1.0 1.0 1.0 3.0 Positive
Asian Development Bank 0.5 1.0 1.0 2.5 Positive
International Fund for Agricultural Development 0.5 1.0 1.0 2.5 Positive
United Nations Development Program 0.5 1.0 1.0 2.5 Positive
Global Alliance for Vaccines and Immunization 0.5 1.0 0.5 2.0 Positive
Global Environmental Facility 0.5 1.0 0.5 2.0 Positive
Global Fund for Aids Tuberculosis and Malaria 0.5 1.0 0.5 2.0 Positive
United Nations Children's Fund 0.5 1.0 0.5 2.0 Neutral
United Nations Joint Program for HIV/AIDS 0.5 1.0 0.5 2.0 Positive
African Development Bank 0.5 0.5 0.5 1.5 Positive
Food and Agriculture Organization of the UN 0.5 0.5 0.5 1.5 Positive
International Federation of Red Cross and Red Crescent Societies 0.5 0.5 0.5 1.5 Positive
International Commission of the Red Cross 0.5 0.5 0.5 1.5 Positive
Office of the Coordinator for Humanitarian Affairs 0.5 0.5 0.5 1.5 Neutral
United Nations Fund for Population Activities 0.5 0.5 0.5 1.5 Positive
World Food Program 0.5 0.5 0.5 1.5 Negative
United Nations High Commissioner for Refugees 0.0 0.5 0.5 1.0 Negative
Commonwealth Secretariat 0.0 0.5 0.0 0.5 Negative
Organization of American States 0.5 0.0 0.0 0.5 Negative


International Financial Institutions

It is striking that the four of the five highest ranked organizations are International Financial Institutions (EBRD, World Bank, Asian Development Bank and IFAD). In part this reflects structural advantages of these multilateral organizations. IFIs tend to be more centralized than their United Nations counterparts and are able to provide more direct quality control over processes for setting objectives, defining results and monitoring and evaluating performance. It may also be a reflection of the fact that IFIs are able to concentrate their efforts in managing for performance on loan and investment projects which are reasonably discrete and lend themselves to measurement of results.

That being said, the advantages of the International Financial Institutions reflected in Table 6.2 go beyond those arising from their basic structure. As a group they have developed systems and processes for managing for development results which have the following important shared characteristics:
  • A clear delineation between program managers' responsibility for self-assessment of projects (usually exercised through end-of-project reports summarized annually in portfolio reviews) and the responsibility of an independent central unit to review and report on the reliability of these assessments;

  • Regular publication of any gap in the ratings assigned to projects by program managers and those of the central quality assurance unit;

  • Clear standards for portfolio coverage by end-of-project reports and portfolio reviews, including confirmation of those by the independent evaluation group;

  • Similar standards for coverage of organizational programs by independent evaluations which may be organized as project, country program, sector or thematic evaluations;

  • An evaluation group which is clearly independent of program management (including policy branches) and which reports to the agency governing body either directly or through the executive head of the organization;

  • A clear link from the process of results definition, monitoring and reporting back to agency processes for identifying, designing and implementing projects and programs;

  • Adequate financial and professional resources for managing complex evaluations and for accessing external consultants as required; and,
  • Recognition of the organizational role and professional standing of organization staff engaged in development and operation of results based management and evaluation systems.
It is notable that only one IFI in the group of organizations consulted by the Review team received a score of less than 2.0. The African Development Bank has invested in similar systems to those already described but has suffered from low coverage, particularly in end-of-project reporting and portfolio review reporting as a result, in part, of the emergency relocation of its headquarters to Tunis. There are some important efforts underway at AfDB to at least begin to re-instate reasonable coverage of the current loan portfolio and to strengthen the results orientation of reporting.

United Nations Agencies

There is generally a greater spread in the assigned rankings among the United Nations agencies than for the IFIs. The World Health Organization (WHO), for example, was among the highest ranked with a score of 3.0. This reflects a major initiative at WHO to implement a comprehensive system of strategic planning, results definition, results reporting and evaluation linked directly to program and project management under the heading of the Global Management System.

The WHO system has been built incrementally since 2001 and includes definition of results at the output, outcome and longer term objectives for 13 strategic objectives of the 2008-2013 Medium Term Strategic Plan. Reporting by individual offices against budgets and expected results is done electronically and sent directly to managers' desk top computers. An external and neutral quality assurance panel reviews the quality of results reporting by program.

The Evaluation Office of WHO contributes to this process through a biennial Performance Monitoring Report and by directly assessing elements of the program of work and budget and carrying out Country Performance Assessments. The 2008/09 annual performance report of WHO will be the first to fully reflect the operation of the system.

The United Nations Development Program is also ranked quite highly with a rating of 2.5 based on very strong systems for annual results reporting and for evaluation (Appendix 5). UNDP is also able to achieve good coverage of its program portfolio through its Annual Development Reports which cover its country programs and are carried out by the central evaluation unit.

UNDP can be expected to move to a full 3.0 rating as it continues to fully implement its recent Results Management and Accountability Framework (RMAF). The organization has in place a balanced scorecard approach integrating 18 different organizational sites and their performance measures with electronic access to results reports by all staff. In 2009, this will be linked to the results based budgeting system and to staff performance assessments in a comprehensive performance management system.

A significant number of UN organizations fall into the middle zone of Table 6.2, with ratings of 1.5 or 2.0 based on established systems of RBM, evaluation and effectiveness reporting (most often with annual effectiveness reports to their governing bodies) and a positive commitment to strengthening these systems.

Only one UN agency (the United Nations High Commissioner for Refugees) is among the three lowest ranked organizations. This ranking is based on continued results reporting on an activity rather than results basis, weak linkages between evaluation and results monitoring and the recent reduction in size of the evaluation unit; which may also be relocated organizationally within a program division.

The World Food Programme (WFP) assigned ranking has been influenced by recent decisions to reduce its staff complement for RBM from 13 officers to one RBM officer and one risk analyst. The rationale for this change is an effort to mainstream RBM. Nonetheless, it remains to be seen if the gains made in RBM at WFP over the past decade can be sustained without continued technical support from the centre.

It should be noted that UN agencies do face significant challenges in managing for development effectiveness through RBM, program evaluation and unified effectiveness reporting. These include:
  • Decentralized organizational structures, including decentralized responsibility for evaluation with country offices undertaking most evaluation work without adequate access to technical support and quality assurance from headquarters or regional offices;
  • The universal mandates of most UN agencies and programs requiring them to maintain activities in countries regardless of the development policy or security environment. This limits their ability to assign resources to countries or programs of high relative performance;
  • The fact that many UN agencies are involved in significant humanitarian relief operations responding to complex emergencies (including armed conflicts) and natural disasters. These operations are inherently difficult to pre-define in terms of expected outcomes and present special challenges for monitoring and evaluation.
UN agencies with a strong humanitarian relief role and mandate (OCHA, WFP and UNHCR) also tend to be clustered in the lower-middle part of the rankings in Table 6.2.

This may reflect their organizational cultures with a strong priority for taking action and emphasizing operations over systems for reflection and review. At the same time this pattern should not be over-emphasized. UNICEF, which also has a strong humanitarian relief role is among the stronger organizations rated in Table 6.2. There are also specific circumstances accounting for the rankings of both UNHCR and the WFP.

It is also clear that many UN agencies are making a major commitment to improving systems for defining results at all levels of their organization, for monitoring performance against those results, for evaluation and for reporting internally and externally.

Other Organizations

There are three major groups of organizations the Review team classed in this category:
  • Global programs often but not always associated with the United Nations but with their own management and evaluation functions (GAVI, GEF, GFATM, UNAIDS);
  • The Red Cross Family (the International Commission of the Red Cross and the International Federation of Red Cross and Red Crescent Societies); and,
  • Political organizations (the Commonwealth Secretariat and the Organization of American States).
From a managing for effectiveness perspective (at least as captured through the strength of RBM, evaluation and effectiveness reporting systems) global programs tend to have quite high ratings (2.0 or higher). This may reflect the fact that most are recently created and are subject to intense scrutiny from participating agencies and supporting government and private sources of funding. They also tend to be smaller organizations in staff terms with fairly clearly defined missions and a corporate culture focused on achieving results, especially at the output level.

The two Red Cross agencies have very different organizational structures with one (ICRC) highly centralized and the other (IFRC) decentralized and dependent on regional offices for quality assurance in managing for results. While they scored in the middle range of the team's analytical frame, both had a strong positive focus in strengthening RBM, evaluations and effectiveness reporting systems.

Finally, the two political bodies (the Commonwealth Secretariat and the OAS) received the lowest scores in the group of agencies. They generally do not systematically monitor the results of projects and programs and while they conduct some thematic evaluations, they either have very little evaluation capacity or no central evaluation authority. In neither case did the Review team find evidence of an intention to provide resources or senior management support to strengthening these functions. It should be noted, however, that the recent multi-year funding agreement between CIDA and the OAS included financial support to assist the OAS in strengthening and implementing evaluation and monitoring mechanisms and systems.

6.2.2 Positive Trends in Most Organizations

It is important to point out that the Review team found a strongly positive trend in terms of organizational efforts to significantly strengthen systems relating to managing for development effectiveness among CIDA key multilateral partners.

Of the 21 organizations reviewed, 15 were classified as positive in terms of their effort to assign resources and make progress in several of the following areas:
  • Developing clearer statements of organizational and program expected results;
  • Strengthening systems for reporting on results (including independent quality assurance);
  • Improving the quality, coverage and independence of evaluation functions;
  • Reporting internally and externally in relation to significant program outcomes; and,
  • In a few cases, better linking the information generated to program and project identification, design, implementation and budgeting and even to staff performance appraisal systems.
Only four of the organizations reviewed were found to be in a negative trend in terms of their commitment and support to RBM, evaluation and effectiveness reporting.

In light of this strong trend, it seems reasonable to expect that evaluations carried out from 2009 onward should reflect improvements in internal management systems with findings in the highly satisfactory or satisfactory range.

6.3 Other Initiatives

As already noted, many multilateral organizations have been investing in improving internal management systems for development effectiveness. These actions by individual organizations making up the multilateral channel available to CIDA have been supplemented by collective efforts to improve evaluation and effectiveness reporting. The most important of these have been the United Nations Evaluation Group (UNEG) among the UN agencies, the Evaluation Cooperation Group initiative for IFIs, and the Common Performance Assessment System (COMPAS) operated by the Multilateral Development Banks.

6.3.1 United Nations Evaluation Group

United Nations agencies had an Inter-Agency Working Group (IAWG) on evaluation from 1984 to 2002. Work began in 2003 on revitalizing and updating the work of the IAWG and in 2004, the United Nations Evaluation Group (UNEG) had its first annual general meeting in Bonn. It currently has 29 member organizations including all of the UN organizations listed in Table 2.1 as well as IFAD (which, while it has many characteristics of a development bank remains a UN agency) and the World Bank.

UNEG's mission statement as formally adopted at the April 2007 annual general meeting focuses explicitly on strengthening the evaluation function: UNEG's mission is to strengthen the objectivity, effectiveness and visibility of the evaluation function across the UN system and to advocate the importance of evaluation for learning, decision making and accountability4.

In order to carry out its mission, UNEG undertakes work in three key areas:
  • Professionalizing the evaluation function through training efforts and by developing a shared statement of core competencies for professional evaluators working in the UN system;
  • Directly supporting evaluation systems and processes among UN agencies by:
  • Publishing norms and standards for the evaluation function at the United Nations, including those for organizational independence of the function;
  • Developing guidelines on integrating human rights and equality between women and men into evaluations; and,
  • Supporting the evaluation function through peer review, guidelines for ethical evaluation and support to impact evaluations. Generally speaking UNEG work in this area is aimed at reinforcing and promoting the distinct role of evaluation in the UN system.
  • Ensuring that the evaluation function plays a key role in efforts at UN reform (specifically the evolution of the United Nations Development Framework and the One UN Pilot).
UNEG began its main contribution to UN Reform efforts by developing principles and a draft evaluation design for joint, country-led evaluation of UN programs at the country level. This was followed up in 2008 by UNEG support of the Joint South Africa country evaluation which assesses all UN activity and is being actively led by the Government of South Africa.

UNEG is also coordinating the evaluation of the UN Delivering as One Pilot countries and is currently assessing the role of individual UN agencies in the common evaluation of the United Nations Development Assistance Framework at country level.

The Review team noted that UNEG has played an important role in professionalizing and raising the profile of evaluation in a number of the organizations CIDA works most closely with in its use of the multilateral channel. Indeed, senior evaluation staff of a number of organizations pointed out that the UNEG norms and standards served as an effective means of convincing executive managers and governing bodies that evaluation offices should be independent of program management and have adequate human resources and budgets assigned.

6.3.2 Evaluation Cooperation Group

The Evaluation Cooperation Group (ECG) was established by the heads of evaluation in a group of multilateral development banks in 1996 in order to:
  • Strengthen the use of evaluation for greater effectiveness and accountability;
  • Share lessons from evaluations and contribute to their dissemination;
  • Harmonize performance indicators and evaluation methodologies and approaches;
  • Enhance evaluation professionalism within the multilateral development banks and to collaborate with the heads of evaluation units of bilateral and multilateral development organizations; and,
  • Facilitate the involvement of borrowing member countries in evaluation and build their evaluation capacity.
The ECG has developed Good Practice Standards (GPS) for evaluations carried out by its members in relation to different types of lending (public sector investment, policy based lending, and country assistance lending for example). It also regularly conducts benchmarking studies to assess the extent that member organizations are applying the GPS.

6.3.3 COMPAS

The Common Performance Assessment System (COMPAS) was developed in 2004 by the Multilateral Development Banks (MDB) working group on Managing for Development Results (MfDR) to provide a common basis for joint reporting on their own performance. The 2007 Report was issued jointly by:
  • The African Development Bank;
  • The Asian Development Bank;
  • The Inter-American Development Bank;
  • The Islamic Development Bank; and the
  • World Bank Group.
The International Fund for Agricultural Development will join the COMPAS system for the 2009 reporting cycle.

Under COMPAS the participating MDBs report on their ability and performance in adopting a managing-for-results approach in their operational processes. Information is gathered at the individual country level but not in relation to how national host governments perform. Instead they track the MDBs ability to support national governments in key areas of managing-for-development.

The most recent report examines key performance indicators in eight categories;
  1. Country capacity to manage for development results.
  2. Country strategies.
  3. Allocation of concessional resources.
  4. Projects.
  5. Institutional learning from operational experience.
  6. Results-focused human resources.
  7. Harmonization among development agencies.
  8. Private sector operations.
COMPAS reports are issued annually and each year there are some adjustments to the methodology and content. The 2007 report differed from the previous year in that it included modifications to some indicators, added indicators relating to private sector operations, expanded coverage to include the Islamic Development Bank and involved independent evaluation groups in the MDBs in providing essential data.

The 2007 COMPAS Report indicates that the participating MDBs improved their efforts to manage for results. Specifically they indicate: an increase in the number of countries where MDBs have provided capacity building for managing for results: improvements in the quality of MDB country strategies; improved quality of results frameworks; and, improved supervision of projects.

6.4 Summary

The Review team's direct findings on the state of RBM, evaluation and results reporting relating to the multilateral channel stand in direct contrast to the negative findings reported in Section 6.1 on some aspects of internal management as found in published evaluation reports. The published evaluations report the percentage of evaluated multilateral initiatives receiving satisfactory ratings as very low: 23 percent for RBM, 24 percent for evaluation and 29 percent for effective knowledge management.

In contrast, the Review team, through direct interviews and a review of policy documents and effectiveness reports, rated 17 of 21 CIDA's key multilateral partner organizations above a basic threshold satisfactory score, with more than half receiving a higher rating. More importantly perhaps, the team also assessed the trend in 15 of the 21 organizations as positive: an indication of increased resources, a higher organizational profile and greater executive management commitment to these elements of managing for development results.

This strongly positive trend, which is not universal of course, suggests that it will be important to repeat the review of published evaluations at some point in the future when recent efforts at strengthening managing for development results (especially, for example at WHO and the FAO) may be reflected in more positive results.

There are two main explanatory factors for the difference between published evaluation results and the direct assessment undertaken by the Review team.
  1. The fact that many of the published evaluations will have assessed the operation of RBM and evaluation systems and processes at a decentralized level while the Review team assessed them from a headquarters perspective; and
  2. The fact that some of the evaluations reviewed were from earlier in the period under review (2004) and will have focused on systems and procedures in place even before then, up to five years before their publication date. They will not have captured the significant efforts to improve these systems in more recent years in many organizations.
These two factors further reinforce the importance of repeating the analysis at some point in the future to see if the promised improvements in managing for development effectiveness will be verified by independent evaluations published 2008 forward.

Improvements in systems for evaluation and effectiveness monitoring and reporting are not limited to individual multilateral organizations. UN agencies and multilateral development banks have also invested in collective efforts to strengthen evaluation and report on effectiveness since 2004. At the same time, bilateral donor countries, including Canada, have undertaken measures to strengthen and refine their approach to joint monitoring of the effectiveness of multilateral organizations.


7.0 Managing for Effectiveness: CIDA's Role

7.1 Monitoring Organizational Effectiveness

7.1.1 Internal CIDA Systems

Field Survey of Multilateral Effectiveness

In recent years, Sectors and Global Partnerships Branch (SGPB) has conducted annual surveys of officers at CIDA posts abroad in an effort to gather an informed opinion on the performance of multilateral organizations. Each year a different set of country posts were requested to provide information on a different set of multilateral organizations. In each of the last three years 20 to 25 country posts have been surveyed with responses rates varying (14 responding in 2005, 22 in 2006 and 18 in 2007). The annual survey was summarized in a Field Survey Report and is provided to those officers at SGPB involved in programming for the agencies in question.

The field survey is intended to complement and provide input to SGPB's broader-based Multilateral Effectiveness and Results Assessment (MERA).

Multilateral Effectiveness and Results Assessment (MERA)

The MERA is an assessment tool developed in-house by CIDA's Sectors and Global Partnerships Branch. It was piloted-tested in 2006 in a cycle covering 22 multilateral organizations. Responsible SGPB desk officers undertake the MERA assessments drawing on available documents, including:
  • SGPB officer reports from organization Board and annual meetings;

  • Reports (including evaluation reports) from multilateral organizations;

  • Evaluations commissioned by CIDA or conducted jointly with other donors;

  • The field survey on multilateral organizations;

  • Inputs from other donor surveys; and,

  • MOPAN survey results.
Each assessment is reviewed and validated by the Branch Management Group. The 2006 MERA reports provided a score for each multilateral organization in three separate areas: relevance, results and management.

When the Review compared MERA assessments to the team's own rating of multilateral organizations, it revealed a generally consistent pattern across the two assessment systems. In both cases, the World Bank and ICRC were highly rated and UNHCR and the Commonwealth Secretariat ranked in the lower group. One anomaly concerned WFP which was more highly rated in the MERA exercise than in the Review.

One possible explanation for this difference is the relatively recent decision by WFP to mainstream RBM and to reduce staffing in the unit responsible for supporting it throughout the organization. This change, which was a key reason for the low rating assigned by the Review team, would have occurred after the MERA exercise.

As of 2008, individual reports prepared for the Multilateral Programming Review do not include numerical scores assigned under the four categories. They do, however, include a description of the opportunities for Canadian leadership present in each organization.

Multilateral Programming Review, 2008

Beginning in May 2008, the Sectors and Global Partnerships Branch at CIDA carried out a systematic review of multilateral programming as a prelude to possible decisions on program funding. The Review examined core funding to multilateral institutions and drew on: annual reports and publications of the institutions; independent evaluation and audit reports; MOPAN and COMPAS outputs; and surveys of CIDA headquarters and field officers (an enhanced and more rigorous MERA process).

The core funding CIDA provides to each organization was assessed against four criteria:
  • Relevance to Canadian priorities;
  • Institutional effectiveness;
  • Development results; and,
  • Adherence to aid effectiveness principles.
Each organization was rated high, medium, medium low or low across the four criteria. They were grouped for analysis purposes into four groupings: multilateral development institutions; international financial institutions; humanitarian organizations; and, Commonwealth/Francophonie.

The 2008 Multilateral Program Review is ongoing and was conducted with the specific purpose of providing the Minister with CIDA's most current evidence on the roles and comparative strengths and weaknesses of the organizations which make up the multilateral channel.

In summary, the Review points to a concerted effort by SGPB to provide evidence which can be used to link agency performance to resource allocation decisions in the future.

7.1.2 CIDA Support to the Multilateral Organization Performance Assessment Network (MOPAN)

MOPAN was created in 2002 as a network of donor countries with a common interest in sharing information and drawing on collective experience in monitoring the performance of multilateral organizations. Current members are: Australia, Austria, Canada, Denmark, Finland, France, Germany, Ireland, The Netherlands, Norway, South Korea, Spain, Sweden, Switzerland, and The United Kingdom.

MOPAN results are based on an annual survey of bilateral development agency staff which covers 3 agencies operations in 8 to 10 countries each year. The survey covers multilateral agency effectiveness in partnership with national stakeholders in each country (governments, civil society and the private sector).

In 2006, MOPAN members agreed that more information on multilateral organization performance was needed than could be provided in the annual survey. MOPAN member countries agreed to move toward a common approach for assessing the effectiveness of multilateral organizations based on a balanced scorecard of Key Performance Indicators.

In August 2008, MOPAN members launched the testing of a new performance assessment instrument called the Common Approach. Under the Common Approach, agency effectiveness is defined as the degree to which a multilateral organization is able to organize itself to produce and deliver expected results. This new approach expands MOPAN from a survey of the perceptions of bilateral agency staff to include documentary evidence of processes and systems. It involves gathering information on key performance indicators in four quadrants and 13 sub-quadrants.

1. Strategic management
  • corporate governance
  • corporate strategy
  • strategies (country, regional, thematic).
2. Operational management
  • use of performance information
  • financial resources management
  • human resources management
  • portfolio management.
3. Relationship management
  • ownership
  • alignment
  • harmonization.
4. Knowledge management
  • performance monitoring and evaluation
  • performance reporting
  • application of lessons learned.
The Common Assessment process is currently being tested and is expected to be fully implemented in 2009. The MOPAN secretariat rotates among participating countries and is currently held by Denmark. The secretariat will be supported in implementing the Common Approach by a dedicated team of consultants recently hired through an international competitive process.

In taking the work of the Common Approach forward, MOPAN established a Working Group with membership from Canada, Denmark, France, the Netherlands, Norway, Sweden and the United Kingdom. Canada is represented by CIDA SGPB in the Working Group and has been a strong supporter of MOPAN since its inception.

While still in its testing phase, the Common Approach represents an important joint effort by donor countries to further strengthen their common tool for regularly assessing the effectiveness of multilateral organizations.

7.2 Advocating and Supporting Effectiveness Management

Canada has many opportunities and occasions to influence and strengthen systems which help multilateral organizations to better manage for effectiveness. During the Review, the team met with many Canadian Government officials who interact on a regular basis with the key organizations CIDA supports in its use of the multilateral channel including:
  • Canadian Executive Directors of International Financial Institutions;
  • Canadian Ambassadors with responsibilities for United Nations agencies;
  • CIDA staff serving in Embassies and Permanent Missions to the UN;
  • CIDA staff representing Canada on agency governing bodies; and
  • CIDA staff from SGPB (and other CIDA branches) with responsibilities for managing the relationship between the agency and specific multilateral organizations.
All of these people were able to describe how they worked to advocate for stronger systems and procedures for managing for effectiveness within the specific organization of their responsibility. Most often this takes the form of working for more precise definition of agency results in programs of work and budgets and for better effectiveness reporting against those pre-determined results. It also includes arguing in meetings of the governing bodies for the strengthening of offices and systems for RBM and evaluation.

SGPB also provided the Review with example of decisions of executive boards of UN agencies where Canada has drafted, facilitated and/or negotiated wording calling on the agencies concerned to strengthen their systems for evaluation and effectiveness reporting. Examples include:
  • The Executive Board decision responding to the 2008 Annual Report on Evaluation of UNDP;
  • The Executive Board decision regarding follow-up to the Evaluation of Gender Policy Implementation at UNICEF (2009);
  • The Executive Board decision responding to the evaluation of UNDP in the net contributor countries of the Arab region (2008);
  • The Executive Board decision recognizing the revisions to UNDP's strategic plan for 2008-2011;
  • The Executive Board decision regarding strengthening of Internal Oversight and Audit at UNDP, UNFPA and UNOPS (United Nations Office for Project Services) (2008); and,
  • The Executive Board decision regarding the revised Evaluation Policy of UNICEF (2008).
SGPB was also able to provide the Review team with examples of Canadian effectiveness in negotiating board decisions seeking improvements to systems and processes relating to institutional cost effectiveness in such areas as program support costs, biennial budgeting processes and human resource management.

Most importantly, when Review team members met with staff members of multilateral organizations, they confirmed that CIDA presents a very consistent Canadian position, continuously advocating for a stronger commitment to managing for effectiveness. There was remarkable consistency among evaluation and RBM managers in the multilateral organizations that CIDA represented an important voice confirming to their own management just how important these systems are if continued donor support is to be secured.

The same managers report that CIDA is not alone in presenting this view. It often acts in concert with donor agencies representing the Nordic countries plus the Netherlands, Australia and the United States in advancing and defending the importance of investments in RBM, evaluation and results reporting. Nonetheless, they confirm that CIDA is one of the strongest and most consistent advocates of improved management for development and humanitarian effectiveness.

7.3 Governance and Programming: Organizational Strategies

It was more difficult for the staff of multilateral organizations interviewed by the Review team to describe or point to an overall strategic orientation on the part of CIDA as it relates to their organization as a whole. They could point to consistent CIDA development and humanitarian priorities including:
  • A focus on the Millennium Development Goals and the organization's contribution to meeting them;
  • A priority for poverty alleviation;
  • A focus on effective and responsible humanitarian relief;
  • An interest in promoting good governance in developing countries;
  • An interest in equality between women and men;
  • An interest in achieving the commitments made by all parties to the Paris Declaration regarding harmonization, alignment, and ownership; and,
  • An interest in supporting United Nations Reform.
Internal interviews within SGPB confirm that these are often advanced by CIDA staff in discussions relating to multilateral organizations and their program of work and budget.

The staff of the organizations themselves, however, often reported that they were not aware of a specific CIDA plan or strategy relating to how their own organization should evolve over time. While they appreciated that CIDA advocated consistently for better systems for RBM, evaluation and effectiveness reporting, they were not able to translate that support into a broader CIDA (or a broader Canadian) strategy for the organization itself.

SGPB's 2008 review of multilateral programming points out that where CIDA has identified a priority action and a strategy in relation to a specific multilateral organization it has been able to exert considerable influence as in the strengthening of oversight, accountability and results within the new structure of the Consultative Group on International Agricultural Research. It also notes that where CIDA has not developed a clear strategy or where it has had to rely on other government departments, it has been less successful.

There is potential for CIDA to develop a concise statement of its institutional strategy in dealing with multilateral organizations. Such a strategy would help to guide funding allocation decisions and to establish objectives in terms of the organizational change which could be monitored over time. It would also help to ensure coherent and consistent approaches to managing the institutional relationship across different CIDA branches and offices.

One strong example of how CIDA can be an important support for strategic change evolved directly out of the 2004 Independent External Evaluation of IFAD. As a consequence of that evaluation, IFAD committed to an Action Plan for significant organizational transformation which included: strategic planning; human resource management; program and project development, design, quality assurance and implementation; program supervision; and managing for development effectiveness. In 2008, CIDA and the Ministries of Foreign Affairs of Norway and the Netherlands collaborated in a review of progress under the Action Plan. The review endorsed progress to date and made important recommendations for consolidating and accelerating change.


8.0 Conclusions and Recommendations

8.1 Conclusions

8.1.1 Development and Humanitarian Effectiveness

Based on the contents of a sample of 117 evaluations, using an aggregate measure (an averaging of ratings for objectives achievement, relevance, cost effectiveness, and sustainability), 69 percent of the multilateral organization programs evaluated were either highly satisfactory (3 percent) or satisfactory (66 percent) in development and humanitarian effectiveness; 30 percent of the multilateral programs evaluated were unsatisfactory and 1 percent highly unsatisfactory.

Distribution of Evaluated Initiatives by Performance Category

Criteria Percent in Performance Category
N HU U S HS Total
Objectives Achievement 114 1 % 28 % 67,5 % 3,5 % 100 %
Relevance 113 0 % 12 % 83 % 5 % 100 %
Cost Effectiveness 67 0 % 43 % 54 % 3 % 100 %
Likelihood of Sustainability 106 3 % 39 % 59 % 0 % 100 %
Development and Humanitarian Effectiveness   1 % 30 % 66 % 3 % 100 %

N= Number of Observations, HU = Highly Unsatisfactory, U = Unsatisfactory, S=Satisfactory, HS=Highly Satisfactory.

For most of the criteria assessed by the Review, United Nations agencies and International Financial Institutions (IFI) were broadly similar in the portion of evaluations rating their programs satisfactory or better. For relevance this was true for 82 percent of IFI programs and 92 percent of UN agency programs. Turning to objectives achievement, the figures are 70 percent satisfactory or better for IFIs and 67 percent for UN agencies.

With regard to sustainability, results were more often satisfactory for UN agency programs (56%) than for IFIs.

Cost effectiveness was rated satisfactory or highly satisfactory in 57 percent of the evaluations reviewed. The small number of evaluations which addressed the question of cost effectiveness renders any comparison across types of agencies difficult.
Many, but not all, the evaluations reviewed pointed to factors which contributed to positive results for development and humanitarian effectiveness. The most important of these include:

Relevance:
  • Commitment by multilateral organizations to harmonization with national priorities and ownership of programs by developing country governments;
  • Participation by multilateral organizations in developing country level coordination bodies such as sector working groups;
  • Openness to including national civil society organizations in planning and priority setting, and
  • Effective capacity development work to allow national and local organizations to take effective part in planning and implementing programs.
Objectives achievement:
  • A long term commitment by multilateral organizations to capacity development support to national and local partner agencies;
  • Effective capacity at country level on the part of the multilateral organizations themselves;
  • Increased participation by civil society and the private sector; and,
  • Clear and consistent measures for targeting program services to vulnerable group members.
Cost effectiveness:
  • Use of cost/benefit and economic sustainability analysis in program design and approval processes;
  • Sourcing of key inputs in the country where the programs take place;
  • Consistent national government funding so that programs are not overly reliant on multilateral agency support;
  • Untying inputs provided by bilateral agencies which flow through multilateral programs;
  • Simplified procurement requirements relying on national systems to the maximum extent possible; and,
  • Timely disbursement of funds whether provided from the organizations core resources or earmarked by bilateral donors so that costly disruptions in program implementation can be avoided.
Sustainability:
  • Consistent support with both staff and finances by the host government during and after program completion;
  • Use of public-private partnership arrangements to sustain infrastructure after program completion;
  • Training and capacity development for community members in maintenance of new assets created with program support; and
  • A general investment in capacity development support for national and local institutions operating and maintaining assets and delivering services after the program is complete.
8.1.2 Managing for Effectiveness: Multilateral Organizations

Generally speaking, the meta-analysis of published evaluations reported quite negative results relating to the strength of internal management systems in multilateral organizations. For example, the evaluations reviewed rated multilateral programs satisfactory or better in relation to strategic planning and priority setting only 48 percent of the time. Results were even worse for RBM (with 23 percent) and Evaluation services (with 24 percent) rated satisfactory or better. Knowledge management was rated satisfactory or better in a slightly more encouraging 31 percent of the evaluations reviewed.

These negative findings may result from some evaluations in the sample covering programs and projects prior to 2004. They may also reflect the decentralized nature of evaluation operations in many multilateral organizations and the difficulties this creates for enforcing evaluation standards and applying quality assurance procedures.

When the Review conducted its own direct analysis of RBM, evaluation and effectiveness reporting systems in 21 of the most important of CIDA's multilateral partner organizations, results were very different from those reported in the evaluations.

Over half (11 of 21) of the organizations reviewed were rated strongly (2.0 out of a possible 3.0), while over three quarters (17 of 21) achieved an acceptable 1.5 scoring. Only 3 multilateral organizations reviewed were rated at 1.0 or less, which is clearly an unsatisfactory rating. In the rating system applied by the Review, IFI organizations were ranked highest, while political organizations were ranked at the lowest level.

While UN organizations tended to be in the middle set of scores for RBM, evaluation and results reporting systems, there is no structural impediment that stops them from being rated higher as WHO received the highest rating of 3.0 and UNDP may be expected to move to such a rating as its new Results Management and Accountability Framework (RMAF) is fully implemented.

Most importantly, there is a strong trend among multilateral organizations to strengthen systems for RBM, evaluation and results reporting and to link them more closely to program design and implementation processes. Of the 21 organizations reviewed, 15 were classified as positive in terms of their effort to assign resources and to make progress in results definition, results reporting, independent and objective evaluation and linking performance information to programs and budgets.

At a system wide level, UN agencies and IFIs have made significant efforts to strengthen evaluation and results reporting. The United Nations Evaluation Group (UNEG) has made considerable progress since 2004 in:
  • promoting professionalism among UN evaluation staff;
  • developing and popularizing norms and standards for evaluation to protect and advance its independence and relevance; and,
  • ensuring that evaluation plays an important role in United Nations Reform.
The Evaluation Cooperation Group (ECG) performs similar functions for the IFIs. The ECG has developed Good Practice Standards (GPS) for evaluations carried out by its members in relation to different types of lending (public sector investment, policy based lending, and country assistance lending for example). It also regularly conducts benchmarking studies to assess the extent that member organizations are applying the GPS.

In the same time period, the IFIs have also cooperated in the development and operation of the Common Performance Assessment System (COMPAS) which focuses on reporting organizational effectiveness in managing for development effectiveness at country level.

In summary, the Review finds that multilateral agencies are making significant investments in strengthening systems for managing effectiveness, especially RBM, evaluation and results reporting systems. This is being done on both an individual agency (with some notable exceptions) and a system-wide basis. It will be essential for CIDA and other donors to find ways to monitor these developments and to see if they translate to better performance in development and humanitarian effectiveness.

8.1.3 Managing for Effectiveness: CIDA's Role

CIDA's Sectors and Global Partnerships Branch (along with other Branches with responsibility for selected multilateral organizations) has made a significant effort in monitoring the effectiveness of multilateral organizations. In the first instance this has included its own internal monitoring processes including the field survey of multilateral effectiveness and the Multilateral Effectiveness and Results Assessment (MERA) process. Most recently, in an enhancement of the MERA process, SGPB has been conducting a thorough Multilateral Program Review undertaken for the Minister to provide information on the overall performance of the multilateral channel as well as a comparative rating of individual organizational performance.

As well as conducting its own internal monitoring systems and periodic reviews, CIDA has consistently supported and made use of the Multilateral Organization Performance Assessment Network (MOPAN). Most recently, it participated in the Working Group established by MOPAN to carry forward the work of implanting its strengthened Common Approach in 2009.

The Review has confirmed from interviews with multilateral organizations that CIDA has been a consistent and effective advocate for increased investment in systems for strengthening management for effectiveness, especially including RBM, evaluation, and effectiveness reporting at the outcome and impact level.

CIDA has also made important contributions to the drafting and acceptance of action oriented decisions by executive boards of multilateral organizations requiring agencies to strengthen their systems for managing for effectiveness.

On the other hand, interlocutors at the multilateral organizations indicate that CIDA does not seem to have a clear strategy at the organizational level (although it does support a consistent set of developmental and humanitarian priorities).

8.2 Recommendations

In light of its overall findings and conclusions, the Review of the Effectiveness of CIDA's Multilateral Delivery Channel recommends that CIDA:
  1. Develop a concise organizational strategy for each multilateral organization where significant funding is channeled, with a view to providing more clarity to the organization on CIDA's goals and expectations. As the lead within CIDA for relations with multilateral partners, Sectors and Global Partnerships Branch would be well placed to prepare these strategies. These strategies should also promote greater consistency across different CIDA branches and offices as they communicate their views on agency priorities.

  2. Explore with other bilateral donor agencies the feasibility of undertaking a similar review of evaluations for assessing the effectiveness of individual multilateral organizations. Any resulting proposed approach should build on work undertaken under the MOPAN Common Approach currently in its testing phase.

  3. Continue to underscore to multilateral institutions the importance of strengthening systems and processes for managing for effectiveness, including RBM, evaluation and effectiveness reporting.

  4. Encourage UN agencies to improve their capacity to provide quality assurance and technical assistance to strengthen the quality of decentralized evaluations at country office and regional level.

  5. The Review recognizes that CIDA's ability to influence the content and direction of programming by multilateral organizations varies depending, among other factors, on the form of support CIDA provides: core funding, initiative specific funding or multi-bilateral funding. While opportunities will be limited when providing core funding, it is recommended that CIDA place strong emphasis on programming factors which evaluations indicate are strongly linked to increased effectiveness when it engages in providing core funding, supporting specific initiatives or providing multi-bilateral support.
These include the following:
  1. Emphasis on harmonization, national ownership and the principles of the Paris Declaration;

  2. Effective and sustained capacity development support to national institutions and to local communities in planning and priority setting and in the operation and maintenance of assets and services following program completion;

  3. Greater participation by both civil society organizations and the private sector in planning and priority setting and in program operations;

  4. Clear and consistent targeting of program services; and,

  5. Use of cost benefit and economic sustainability analysis in program design as well as local sourcing of inputs and more frequent use of private-public partnerships. In general there is a strong argument in the evaluations for greater emphasis on cost effectiveness in program design and in evaluation.


9.0 Management Response

Many development issues require a global response. Multilateral organizations allow member states to pool resources, knowledge and expertise to address specific global problems. e.g. the Global Environment Facility for the environment, the Global Fund to fight AIDS, Tuberculosis and Malaria, the World Food Programme for food assistance and the International Financial Institutions for large infrastructure projects. A number of them, the United Nations Development Program for example, are equipped and mandated to operate in fragile states where they can facilitate harmonized and coordinated action. Organizations like the World Bank can provide advice in sensitive areas like economic reform programs where bilateral donors would be reluctant to act alone. Moreover, because multilateral organizations must use international competitive bidding for procurement, their aid is untied and they avoid some of the efficiency losses that have often been associated with the aid tying of bilateral aid programs.

Another attribute of multilateral aid is its global reach, that is, the fact that multilateral organizations are mandated to operate in all developing countries. As bilateral programs, including Canada's, become increasingly focused on fewer countries, multilateral aid will continue to flow to countries where bilateral donors are no longer as present. It is thus essential that multilateral organizations become more effective and that they deliver strong results at the country level.

There are a variety of ways in which Canada can influence the priorities and performance of multilateral organizations. Because Canada provides core funds to multilateral development organizations, it has a role to play on each of these organizations' governing bodies. For this reason multilateral agencies should not be regarded as purely independent third parties to which we provide aid monies but rather as organizations in which Canada is a member and for which it has a shared accountability with other nations for the organization's management and ultimately for the results it achieves. In fact, it is chiefly through our role on the governing bodies of these organizations, that CIDA ensures accountability of our core funding to these institutions. CIDA's main task on the governing bodies is to ensure that the organization is effectively empowered to fulfil its development or humanitarian mandate and deliver results on the ground. CIDA uses its voice on multilateral governing bodies to influence the strategic direction of the organization and works with other donors to strengthen results. CIDA also uses the results of the organizational performance assessments it conducts to identify areas where the organization needs strengthening. Over the years CIDA has been a particularly strong advocate of results-based management reforms, equality between women and men policies and an improved audit and evaluation function within multilateral organizations.

This Review of the Effectiveness of CIDA's Multilateral Delivery Channel raises some important concerns about the performance of multilateral organizations and the systems they have in place to assess their own performance. It thus provides a useful foundation for understanding the efforts and initiatives put in place in recent years by multilateral organizations themselves and by CIDA to monitor and capture multilateral effectiveness and to identify areas where improvements may be required. For example, CIDA has been playing a lead role working with other bilateral donors in developing a common approach toward assessing the effectiveness of multilateral organizations within the Multilateral Organizations Performance Assessment Network or MOPAN. MOPAN is an informal grouping of 15 donor countries. CIDA is one of six countries within MOPAN participating on the Working Group that has the lead in developing and implementing the Common Approach.

The objective of the Common Approach is to reduce duplication of performance measurement activity by different donors with the same multilateral organizations and increasing the amount and scope of information available on the effectiveness of those institutions. The MOPAN Common Approach is based on a "Balanced Scorecard" which assesses organizations across four categories: Strategic Management, Operational Management, Relationship Management and Knowledge Management. Each of these four categories will be assessed in turn through a set Key Performance Indicators. The Common approach was tested with two multilateral organizations in 2008 and beginning in 2009 will be used to assess 6 multilateral organizations per year.

The Review of the Effectiveness of CIDA's Multilateral Delivery Channnel identified the strong positive trend among multilateral organizations in managing for development results and highlighted the significant investments made by those organizations in evaluation and reporting systems. The review showed satisfactory findings with regard to the achievement of multilateral development objectives and the relevance of multilateral programming (including aligning with national priorities and participating in local planning and coordinating bodies). However, the Review reported weaker performance in terms of the overall cost effectiveness and longer-term sustainability of activities funded through multilateral organizations.

Management notes the limitations identified in the Review and the caution needed in interpreting the findings as a result. As the Review notes: "The conclusions … are based on a sample of published evaluations and are relevant in assessing the effectiveness of multilateral organizations as a group or in significant subgroups…. They are not valid and have not been used to address the effectiveness of individual organizations." Similarly, in addressing the state of results-based management systems in multilateral organizations, it notes that "some of the evaluations reviewed were from the earlier period under review (2004) …. (and) will not have captured the significant efforts to improve these systems in more recent years in many organizations."

Recommendations

Overall, Management agrees with the Review's recommendations. They are consistent with efforts to enhance the overall effectiveness of Canada's program of development assistance.

The table below illustrates management's commitment. Management particularly supports the recommendation to develop organizational strategies that will be used to guide our interaction and our level of engagement with multilateral partners.

Recommendations Commitments and Action Responsibility Centre Planned Completion Date Progress
1. Develop a concise organizational strategy for each multilateral organization where significant funding is channeled, with a view to providing more clarity to the organization on CIDA's goals and expectations. As the lead within CIDA for relations with multilateral partners, Sectors and Global Partnerships Branch would be well placed to prepare these strategies. These strategies should also promote greater consistency across different CIDA branches and offices as they communicate their views on agency priorities. Management agrees with this recommendation.
Sectors and Global Partnerships Branch will develop institutional strategies for each of its 15 main institutional partners. These strategies will identify short and medium term objectives for CIDA's interaction with each institution and will help ensure a coherent CIDA approach to working with multilateral organizations.
The strategies will also review the areas of strength and areas needing improvement in each institution. They will identify concrete actions CIDA will take to help an organization improve its performance or measures to be taken if there is no commitment to improvement.
Director General—Multilateral Development Institutions Directorate Fall 2009  
2. Explore with other bilateral donor agencies the feasibility of undertaking a similar review of evaluations for assessing the effectiveness of individual multilateral organization. Any resulting proposed approach should build on work undertaken under the MOPAN Common Approach currently in its testing phase. Management agrees with this recommendation.
Discussions on the possibility of undertaking a similar review of evaluations have been initiated through the DAC Working Party on Aid Evaluation. Management agrees that such reviews should complement the work done by MOPAN by focusing on relevance, objective achievement, cost effectiveness, sustainability and development results attained by individual multilateral organizations; i.e. the areas of evaluation MOPAN does not examine. The MOPAN Common Approach looks at how the MOs are organized from the point of view of management and leadership, use of resources, systems and processes - including RBM and evaluation, and partner relations to deliver better results.
Director -
Evaluation Division,
Strategic Policy and Performance Branch (SPPB) in collaboration with Director - Strategic Planning and Analysis Division (SGPB)
March 2010  
3. Continue to underscore to multilateral institutions the importance of strengthening systems and processes for managing for effectiveness, including RBM, evaluation and effectiveness reporting. Management agrees with this recommendation.
SGPB will continue to emphasize the importance of strengthening systems to manage for effectiveness by: 1) incorporating this as a key priority in the institutional strategies currently under development, 2) continuing to raise these issues at the governing bodies of each institution and through ongoing discussions with individual institutions, and 3) in collaboration with other donors, via continued active involvement in the MOPAN. In addition to the annual survey, the MOPAN Common Approach will include an assessment of the results-based management practices of one organization per year (organizations showing weakness in this area in the Common Approach findings).
Director General - Multilateral Development Institutions Directorate &
Director - Strategic Planning and Analysis Division (SGPB)
Ongoing  
4. Encourage UN agencies to improve their capacity to provide quality assurance and technical assistance to strengthen the quality of decentralized evaluation functions at country office and regional level. Management agrees with this recommendation.
SGPB will continue to press for strengthened evaluation functions of UN agencies at the country and regional levels by: 1) incorporating this as a key priority in the institutional strategies currently under development, 2) continuing to emphasize this issue at the Boards and in discussions with individual UN organizations, and 3) by working closely with CIDA field staff to pursue this issue with UN agencies in the field, particularly in countries where CIDA is supporting the One UN pilots.
Director - UN & Commonwealth Division, Multilateral Development Institutions Directorate Ongoing  
5. Place strong emphasis on programming factors which evaluations indicate are strongly linked to increased effectiveness when it engages in providing core funding, supporting specific initiatives or providing multi-bilateral support. These include the following:
1. emphasis on harmonization, national ownership and the principles of the Paris Declaration;
2. effective and sustained capacity development support to national institutions and to local communities;
3. greater participation by both civil society organizations and the private sector in planning and priority setting in program operations;
4. clear and consistent targeting of program services;
5. use of cost benefit and economic sustainability analysis in program design as well as local sourcing of inputs and more frequent use of public-private partnerships.
Management agrees with this recommendation.
The actions to be taken will vary across the three types of funding arrangements CIDA has with multilateral institutions (core funding, initiative-specific funding and multi-bi funding), since certain programming factors are more relevant to different funding types.
Core funding: Through its representatives on governing bodies and various board committees, SGPB will strengthen its emphasis on the follow-up to evaluations within multilateral organizations to ensure that lessons learned are incorporated in future programming. In terms of the programming factors listed in the recommendation, SGPB will give priority to the Paris principles on aid effectiveness and to enhanced participation of civil society and the private sector in priority setting.
Initiative-specific funding: In discussions on possible funding for a specific initiative at the thematic or sectoral level, SGPB will emphasize aid effectiveness concerns including capacity development and the use of cost benefit analysis. Where CIDA is funding the second phase of any activity it will ensure that evaluation findings are incorporated in program design.
Multi-bi funding: Since multi-bi funding is focused on a single country or group of countries, the Geographic Programs Branch will emphasize those effectiveness factors that are most relevant at the country level. These include the Paris Declaration principles, local capacity development and clear targeting of program services.
SGPB in collaboration with Geographic Programming Branches






Ongoing  



Appendix 1: CIDA Disbursements by Funding Type (2001/02 - 2006/07)

Rank Multilateral Organization Total Core Funding Initiative-Specific Multi-Bi
1 World Bank (including IFC and IBRD) $974,977,333 - $211,373,845 $763,603,488
2 World Food Programme $795,449,307 $144,672,785 $627,506,347 $23,270,174
3 United Nations Development Program $694,850,945 $270,000,000 $74,881,736 $349,969,209
4 UNICEF $606,626,431 $85,500,000 $328,840,025 $192,286,406
5 Asian Development Bank $581,899,723 $521,344,610 $7,986,563 $52,568,550
6 GFATM $528,528,249 $528,528,249 - -
7 African Development Bank $498,067,344 $463,578,477 $5,350,000 $29,138,867
8 World Health Organization (including Stop TB Partnership) $479,884,338 - $411,838,401 $68,045,937
9 Global Environmental Facility $302,401,001 $296,401,001 $6,000,000 -
10 United Nations High Commissioner for Refugees $195,130,663 $70,039,778 $112,905,885 $12,185,000
11 GAVI - UNICEF $188,000,000 $188,000,000 - -
12 United Nations Population Fund $179,573,827 $89,350,000 $2,500,000 $87,723,827
13 International Committee of the Red Cross $145,638,200 $ $28,000,000 $117,638,200 -
14 International Fund for Agricultural Development $124,255,749 $119,019,770 $2,800,000 $2,435,979
15 UNRWA $111,484,246 - $53,625,000 $57,859,246
16 Consultative Group on International Agricultural Research $80,620,000 $78,420,000 $2,200,000 -
17 Commonwealth Fund for Technical Cooperation $76,565,000 $76,060,000 $50,000 $455,000
18 United Nations Mine Action Service $53,226,000 $2,000,000 $11,976,000 $39,250,000
19 European Bank for Reconstruction and Development $52,470,726 - - $52,470,726
20 Office for the Coordination of Humanitarian Affairs $44,092,000 $6,100,000 $35,677,000 $2,315,000
21 International Federation of Red Cross and Red Crescent Societies $42,265,570 - $42,265,570 -
22 Organization of American States $40,842,468 - $625,000 $40,217,468
23 Red Cross International Aid Trust of Canada $30,240,399 - $30,240,399 -


Appendix 2: Evaluation Sample

  Evaluation Title Type of Evaluation Agency Type
1 Canadian Foodgrains Bank Evaluation CIDA-Led or Joint Other
2 United Nations Mine Actions Service: Summary of 6 Country Evaluations CIDA-Led or Joint Other
3 UNICEF: Peer Review of Evaluation Function CIDA-Led or Joint Other
4 Evaluation of the Canadian Consultant Management Fund CIDA-Led or Joint Other
5 Évaluation de la participation canadienne à la Francophonie pour la période 2000 à 2006 CIDA-Led or Joint Other
6 Third Overall Performance Study of the Global Environment Facility CIDA-led or Joint Other
7 Evaluation of the International Trade Centre UNCTAD/WTO CIDA-led or Joint Other
8 Local Solutions to Global Challenges: Towards Effective Partnership in Basic Education CIDA-led or Joint Other
9 Joint Evaluation of Effectiveness and Impact of the Enabling Development Policy of the World Food Program (WFP) CIDA-led or Joint Other
10 OCHA Humanitarian Response Review CIDA-Led or Joint UN
11 Peace and Security IMPACS evaluation CIDA-Led or Joint UN
12 OCHA Common Humanitarian Fund CIDA-Led or Joint UN
13 Meta-Evaluation of CIDA's Grant Program to the United Nations Development Program (UNDP) 2004-2007 CIDA-Led or Joint UN
14 Global Health Research Initiative Review CIDA-Led or Joint Other
15 AFDB: REVIEW OF BANK ASSISTANCE EFFECTIVENESS TO THE HEALTH SECTOR IFI 104 IFI
16 ASSESSING THE DEVELOPMENT EFFECTIVENESS OF THE AFRICAN DEVELOPMENT FUND (ADF) IFI 147 IFI
17 Review of Bank Group Assistance to the Agriculture and Rural Development Sector in Uganda IFI 148 IFI
18 EBRD: Country Strategy Evaluation for Mongolia IFI 156 IFI
19 EBRD: Extractive Industry Review IFI 165 IFI
20 Joint evaluation of the GEF Activity Cycle and Modalities IFI 187 IFI
21 Evaluation of IFAD's Regional Strategy in Asia and the Pacific - (EVEREST) IFI 214 IFI
22 ADB: 2007 Annual Evaluation Review: The Challenge of Capacity Development IFI 22 IFI
23 Direct Supervision Pilot Programme Corporate-Level Evaluation IFI 223 IFI
24 IFAD's performance and impact in decentralizing environments: experiences from Ethiopia, Tanzania and Uganda… IFI 224 IFI
25 Development Results in Middle-Income Countries, An Evaluation of the World Bank's Support IFI 241 IFI
26 A Decade of Action in Transport: An Evaluation of World Bank Assistance to the Transport Sector, 1995-2005 IFI 244 IFI
27 WB From Schooling Access to Learning Outcomes: An Unfinished Agenda IFI 249 IFI
28 Effectiveness of World Bank Support for Community-Based and Driven Development IFI 272 IFI
29 Special Evaluation on the Involvement of Civil Society Organizations in ADB Operations IFI 32 IFI
30 Special Evaluation on ADB's Fisheries Policy IFI 33 IFI
31 World Bank: Environmental Sustainability An Evaluation of World Bank Group Support IFI 348 IFI
32 ADB: SPECIAL EVALUATION STUDY ON URBAN SECTOR STRATEGY AND OPERATIONS IFI 35 IFI
33 IFC: Financing Micro, Small, and Medium Enterprises An Independent Evaluation of IFC's Experience with Financial Intermediaries in Frontier Countries IFI 351 IFI
34 Commonwealth Fund: EVALUATION OF THE COMMONWEALTH MEDIA DEVELOPMENT FUND IFI 391 IFI
35 ADB:Selected ADB Interventions on Nutrition and Food Fortification IFI 63 IFI
36 Ethiopia: Review of Bank Group Assistance to the Agricultural and Rural Development Sector IFI 98 IFI
37 Evaluating Bank's Assistance for Capacity Strengthening of Urban Water Supply and Sanitation Entities in Regional Member Countries IFI137 IFI
38 Evaluation of GEF Support for Biosafety IFI185 IFI
39 GEF: Biodiversity Program Study 2004 IFI194 IFI
40 WB Assistance to Agriculture in Sub-Saharan Africa, an IEG Review IFI243 IFI
41 Hazards of Nature, Risks to Development An IEG Evaluation of WB Assistance for Natural Disasters IFI250 IFI
42 Improving the WB's Development Effectiveness IFI282 IFI
43 Capacity Building in Africa, An OED Evaluation of World Bank Support IFI284 IFI
44 Extractive Industries and Sustainable Development, an Evaluation of World Bank Group Experience (IFC) IFI362 IFI
45 Evaluation of the Commonwealth Secretariat's Strategy for Gender Equality and Gender Mainstreaming IFI388 IFI
46 Special Evaluation Study on ADB policy for the Health Sector IFI39 (replaced IFI13) IFI
47 IFC: IFC Assistance to Magadan SEMS Multi bi 33 IFI
48 BRAC Educational Program Multi- Bi 55 Other
49 KOFl ANNAN INTERNATIONAL PEACEKEEPING TRAINING CENTRE - SUPPORT FOR CAPACITY DEVELOPMENT … Multi-bi 12 Other
50 ÉVALUATION PROJET PNUD CHAÎNE PÉNALE Multi-bi 25 UN
51 Government of Egypt and UNICEF Program of Cooperation 2002-2006 Multi-bi 27 UN
52 WB-Mid-Term Review of the National Solidarity Program (NSP) Afghanistan (could not find report desired, this is a substitute) Multi-bi 3 Other
53 Multi-bi: Official Development Assistance in Central Europe Program Multi-bi 35 Other
54 UNFPA and WB: Bangladesh Health and Population Reform Programme: Historical Review and Lessons Learned 2003 Multi-bi 37 UN
55 Mid-Term Review of the Microfinance Sector and MISFA in Afghanistan Multi-Bi 4 Other
56 Retrospective and Prospective Evaluation of Canada/ Cuba Cooperation in the Field of Modernization of the Economy Multi-bi 50 Other
57 Multi-bi: IUCN-ROSA ZAMBEZI BASIN WETLANDS PROJECT PHASE II Multi-bi 56 Other
58 SEAL Project Evaluation Multi-Bi 7 Other
59 Evaluacion del Programa Nacional de Salud Sexual y Reproductiva en el Ministerio de Salud Multi-Bi 8 Other
60 Mid-Term Review of the South Asia Enterprise Development Facility (SEDF) Multi-Bi19 Other
61 Central Emergency Response Fund: Review of First Year of Operations, Final report CIDA-Led or Joint UN
62 Multilateral response to the Indian Ocean Tsunami CIDA-Led or Joint UN
63 Report of the Independent External Evaluation of the Food and Agriculture Organization of the United Nations (FAO) CIDA-Led or Joint UN
64 Micronutrient Initiative Interim Report CIDA-Led or Joint Other
65 IFAD An Independent External Evaluation CIDA-Led or Joint IFI
66 UNICEF: Evaluation of United Nations-supported pilot projects for the prevention of mother-to-child transmission of HIV UN 110 UN
67 Thematic Evaluation of the WFP School Feeding in Emergency Situations UN 123 UN
68 WFP: Full Report of the Thematic Review of Targeting in WFP Relief Operations UN 138 UN
69 Thematic Review of WFP Food Aid for Nutrition: Mother and Child Nutrition UN 139 UN
70 Thematic Evaluation of the Protracted Relief and Recovery Operation (PRRO) Category (WFP) UN 148 UN
71 UNHCR: Real time evaluation of UNHCR's IDP operation in the Democratic Republic of Congo UN 162 UN
72 Evaluation of the Department of International Protection's Protection Information Section (PIS) UN 171 UN
73 UNHCR: The road to health and the road to Afghanistan UN 174 UN
74 UNHCR Evaluation of the UNHCR Medical Service UN 186 UN
75 GAVI: An evaluation of GAVI's Immunisation Services Support UN 2 UN
76 Evaluation of the ICRC Emergency Operation in Lebanon, 2006 UN 20 UN
77 WHO: Report on the Evaluation of the WHO Multi-country Family Health Nurse Pilot UN 200 UN
78 External Review of the African AIDS Vaccine Program UN 208 UN
79 Evaluation of the DFID-WHO Partnership UN 211 UN
80 ICRC: Evaluation of the ICRC Action in Favour of the Missing and Their Families in the Balkans UN 22 UN
81 WHO's Work with Collaborating Centres UN 235 UN
82 Selected Aspects of the Public Health and Environment Department UN 236 UN
83 An Evaluation of UNFPA Support for Preventing the Spread of HIV/AIDS UN 244 UN
84 UNMAS; Evaluation of the Framework for Mine Action Planning and Rapid Response UN 246 UN
85 Evaluation of the Mine Action Programme in Sudan UN 249 UN
86 UNDP Support to Conflict-Affected Countries UN 269 UN
87 Evaluation of UNDP's Role in the PRSP Process UN 277 UN
88 UNICEF: Assessment of Development Results: Colombia UN 286 UN
89 Assessment of Development Results Evaluation of UNDP's Contribution: Ethiopia UN 288 UN
90 Country Evaluation: Assessment of Development Results: Bangladesh UN 293 UN
91 Evaluation of Common/Pooled Humanitarian Funds in DRC and Sudan UN 32 UN
92 OCHA: Inter-agency Real-Time Evaluation of the Humanitarian Response to the Darfur Crisis UN 39 UN
93 GFATM: The Five Year Evaluation of the Global Fund: STUDY AREA 1 - GLOBAL FUND ORGANIZATONAL EFFICIENCY… UN 4 UN
94 Evaluation of the Response to the 2002-03 Emergency in Ethiopia, October 2004, by Peter Simkin, Ato Teferi Bekele, Ato Daba Gabissa, Angela Raven-Roberts, John Graham and Lyle Bastin UN 42 UN
95 IFRC: EVALUATION OF PADRU PAN AMERICAN DISASTER RESPONSE UNIT UN 47 UN
96 Evaluation of the HIV/AIDS Global Program 2002-2005 UN 52 UN
97 2008 Global: Evaluation of Gender Policy Implementation in UNICEF UN 59 UN
98 2007 Global: Review of UNICEF's Partnerships with Civil Society Organizations UN 61 UN
99 UNICEF: 2006 Global: The 2004 Indian Ocean Tsunami Disaster: Evaluation of UNICEF's Response (Synthesis Report) UN 62 UN
100 Country Programme Evaluation Government of the Transitional Islamic State of Afghanistan UN 66 UN
101 UNHCR Evaluation of the Department of International Protection's Protection Information Section (PIS) UN 67 UN
102 Evaluation of the Innocenti Research Centre UN 75 UN
103 Changing Lives of Girls: Evaluation of the African Girls' Education Initiative (UNICEF) UN 86 UN
104 2003 Global: Education as a Preventive Strategy Against Child Labour: Evaluation of UNICEF Global Child Labour Programme UN109 UN
105 Evaluation of WFP's Capacity Development Policy and Operations UN116 UN
106 Evaluation of WFP's Assistance to China (1979-2005) UN137 UN
107 UNHCR's age and gender mainstreaming pilot project 2004 UN176 UN
108 Independent External Evaluation of the Global Stop TB Partnership UN195 UN
109 Evaluation of WHO's Contribution to "3 by 5" UN204 UN
110 Addressing the Reproductive Health Needs and Rights of Young People since ICPD, The Contribution of UNFPA and IPPF UN237 UN
111 Evaluation of UNICEF's Support to Mine Action UN265 UN
112 The CGIAR at 31: An Independent Meta-Evaluation of the Consultative Group on International Agricultural Research UN266 UN
113 Evaluation of Results-Based Management at UNDP: Achieving Results UN267 UN
114 Evaluation of Gender Mainstreaming in UNDP UN273 UN
115 Real time evaluation of the drought response in the Horn of Africa UN37 UN
116 Evaluation of ICRC's Sub-Regional 'Unaccompanied Children (UAC) Tracing Programme" in West Africa UN15 UN
117 Mid-term Evaluation of the India Country Programme (2003-2007) UN127 UN



Appendix 3: Evaluation Scoring Chart

Appendix 4: Detailed Results of Evaluation Review

1.1 Agency programs and projects align with National and International Development Goals.

Rating Total %
Highly Unsatisfactory 0 0.0%
Unsatisfactory 13 11.5%
Satisfactory 94 83.2%
Highly Satisfactory 6 5.3%
Total 113 100.0%

1.2 Agency activities coordinate with other agencies and align with national priorities

Rating Total %
Highly Unsatisfactory 1 1.2%
Unsatisfactory 17 20.5%
Satisfactory 64 77.1%
Highly Satisfactory 1 1.2%
Total 83 100.0%

1.3 Agency takes part in local planning and coordinating bodies.

Rating Total% Total %
Highly Unsatisfactory 1 0.9%
Unsatisfactory 34 31.5%
Satisfactory 73 67.6%
Highly Satisfactory 0 0.0%
Total 108 100.0%

2.1 Effective agency level strategic planning and priority setting.

Rating Total% Total %
Highly Unsatisfactory 3 2.6%
Unsatisfactory 57 49.6%
Satisfactory 53 46.1%
Highly Satisfactory 2 1.7%
Total 115 100.0%

2.2 Effective Results Based Management system.

Rating Total% Total %
Highly Unsatisfactory 13 15.9%
Unsatisfactory 50 61.0%
Satisfactory 18 22.0%
Highly Satisfactory 1 1.2%
Total 82 100.0%

2.3 Reliable and useful program evaluation.

Rating Total %
Highly Unsatisfactory 5 4.8%
Unsatisfactory 74 71.2%
Satisfactory 23 22.1%
Highly Satisfactory 2 1.9%
Total 104 100.0%

2.4 Effective Knowledge Management.

Rating Total %
Highly Unsatisfactory 3 3.3%
Unsatisfactory 62 67.4%
Satisfactory 26 28.3%
Highly Satisfactory 1 1.1%
Total 92 100.0%

3.1 Achievement of objectives.

Rating Total %
Highly Unsatisfactory 1 0.9%
Unsatisfactory 32 28.1%
Satisfactory 77 67.5%
Highly Satisfactory 4 3.5%
Total 114 100.0%

4.1 Agency Program and Projects are Cost Effective

Rating Total %
Highly Unsatisfactory 0 0.0%
Unsatisfactory 29 43.3%
Satisfactory 36 53.7%
Highly Satisfactory 2 3.0%
Total 67 100.0%

5.1 The stream of benefits accrued will continue after project or program closure

Rating Total %
Highly Unsatisfactory 3 2.8%
Unsatisfactory 41 38.7%
Satisfactory 62 58.5%
Highly Satisfactory 0 0.0%
Total 106 100.0%

5.2 Agency programs and projects address the capacity of local institutions needed to sustain results

Rating Total %
Highly Unsatisfactory 2 1.9%
Unsatisfactory 61 59.2%
Satisfactory 40 38.8%
Highly Satisfactory 0 0.0%
Total 103 100.0%

5.3 Agency programs and projects provide adequate human, financial and physical resources (Including host country resources)

Rating Total %
Highly Unsatisfactory 1 0.9%
Unsatisfactory 71 67.0%
Satisfactory 34 32.1%
Highly Satisfactory 0 0.0%
Total 106 100.0%

1.1 Agency programs and projects align with National and International Development Goals.

Rating+ IFI % UN % Other % Total %
Highly Unsatisfactory 0 0.0% 0 0.0% 0 0.0% 0 0.0%
Unsatisfactory 6 17.6% 5 8.1% 2 11.8% 13 11.5%
Satisfactory 26 76.5% 54 87.1% 14 82.4% 94 83.2%
Highly Satisfactory 2 5.9% 3 4.8% 1 5.9% 6 5.3%
Total 34 100.0% 62 100.0% 17 100.0% 113 100.0%

1.2 Agency humanitarian activities coordinate with other agencies and align with national priorities

Rating+ IFI % UN % Other % Total %
Highly Unsatisfactory 0 0.0% 1 2.0% 0 0.0% 1 1.2%
Unsatisfactory 5 23.8% 9 18.4% 3 23.1% 17 20.5%
Satisfactory 16 76.2% 38 77.6% 10 76.9% 64 77.1%
Highly Satisfactory 0 0.0% 1 2.0% 0 0.0% 1 1.2%
Total 21 100.0% 49 100.0% 13 100.0% 83 100.0%

1.3 Agency takes part in local planning and coordinating bodies.

Rating+ IFI % UN % Other % Total %
Highly Unsatisfactory 0 0.0% 1 1.7% 0 0.0% 1 0.9%
Unsatisfactory 14 43.8% 20 33.3% 0 0.0% 34 31.5%
Satisfactory 18 56.3% 39 65.0% 16 100.0% 73 67.6%
Highly Satisfactory 0 0.0% 0 0.0% 0 0.0% 0 0.0%
Total 32 100.0% 60 100.0% 16 100.0% 108 100.0%

2.1 Effective agency level strategic planning and priority setting.

Rating+ IFI % UN % Other % Total %
Highly Unsatisfactory 1 3.0% 2 3.0% 0 0.0% 1 2.6%
Unsatisfactory 13 39.4% 35 53.0% 9 56.3% 57 49.6%
Satisfactory 18 54.5% 29 43.9% 6 37.5% 53 46.1%
Highly Satisfactory 1 3.0% 0 0.0% 1 6.3% 2 1.7%
Total 33 100.0% 66 100.0% 16 100.0% 115 100.0%

2.2 Effective Results Based Management system.

Rating IFI % UN % Other % Total %
Highly Unsatisfactory 5 21.7% 6 13.6% 2 13.3% 13 15.9%
Unsatisfactory 13 56.5% 29 65.9% 8 53.3% 50 61.0%
Satisfactory 5 21.7% 9 20.5% 4 26.7% 18 22.0%
Highly Satisfactory 0 0.0% 0 0.0% 1 6.7% 1 1.2%
Total 23 100.0% 44 100.0% 15 100.0% 82 100.0%

2.3 Reliable and useful program evaluation.

Rating IFI % UN % Other % Total %
Highly Unsatisfactory 1 3.1% 3 5.3% 1 6.7% 5 4.8%
Unsatisfactory 24 75.0% 39 68.4% 11 73.3% 74 71.2%
Satisfactory 6 18.8% 14 24.6% 3 20.0% 23 22.1%
Highly Satisfactory 1 3.1% 1 1.8% 0 0.0% 2 1.9%
Total 32 100.0% 57 100.0% 15 100.0% 104 100.0%

2.4 Effective Knowledge Management.

Rating IFI % UN % Other % Total %
Highly Unsatisfactory 1 3.4% 1 2.0% 1 8.3% 3 3.3%
Unsatisfactory 17 58.6% 38 74.5% 7 58.3% 62 67.4%
Satisfactory 11 37.9% 11 21.6% 4 33.3% 26 28.3%
Highly Satisfactory 0 0.0% 1 2.0% 0 0.0% 1 1.1%
Total 29 100.0% 51 100.0% 12 100.0% 92 100.0%

3.1 Achievement of objectives.

Rating IFI % UN % Other % Total %
Highly Unsatisfactory 1 3.0% 0 0.0% 0 0.0% 1 0.9%
Unsatisfactory 9 27.3% 21 32.8% 2 11.8% 32 28.1%
Satisfactory 23 69.7% 42 65.6% 12 70.6% 77 67.5%
Highly Satisfactory 0 0.0% 1 1.6% 3 17.6% 4 3.5%
Total 33 100.0% 64 100.0% 17 100.0% 114 100.0%

4.1 Agency Program and Projects are Cost Effective

Rating IFI % UN % Other % Total %
Highly Unsatisfactory 0 0.0% 0 0.0% 0 0.0% 0 0.0%
Unsatisfactory 9 36.0% 17 60.7% 3 21.4% 29 43.3%
Satisfactory 16 64.0% 9 32.1% 11 78.6% 36 53.7%
Highly Satisfactory 0 0.0% 2 7.1% 0 0.0% 2 3.0%
Total 25 100.0% 28 100.0% 14 100.0% 67 100.0%

5.1 Stream of Benefits Will Continue After Program Closure

Rating IFI % UN % Other % Total %
Highly Unsatisfactory 2 5.7% 0 0.0% 1 6.3% 3 2.8%
Unsatisfactory 16 45.7% 24 43.6% 1 6.3% 41 38.7%
Satisfactory 17 48.6% 31 56.4% 14 87.5% 62 58.5%
Highly Satisfactory 0 0.0% 0 0.0% 0 0.0% 0 0.0%
Total 35 100.0% 55 100.0 16 100.0% 106 100.0%

5.2 Agency programs and projects address the capacity of local institutions needed to sustain results.

Rating IFI % UN % Other % Total %
Highly Unsatisfactory 2 6.3% 0 0.0% 0 0.0% 2 1.9%
Unsatisfactory 16 50.0% 37 68.5% 8 47.1% 61 59.2%
Satisfactory 14 43.8% 17 31.5% 9 52.9% 40 38.8%
Highly Satisfactory 0 0.0% 0 0.0% 0 0.0% 0 0.0%
Total 32 100.0% 54 100.0% 17 100.0% 103 100.0%

5.3 Agency programs and projects provide adequate human, financial and physical resources (Including host country resources)

Rating IFI % UN % Other % Total %
Highly Unsatisfactory 1 3.2% 0 0.0% 0 0.0% 1 0.9%
Unsatisfactory 22 71.0% 42 72.4% 7 41.2% 71 67.0%
Satisfactory 8 25.8% 16 27.6% 10 58.8% 34 32.1%
Highly Satisfactory 0 0.0% 0 0.0% 0 0.0% 0 0.0%
Total 31 100.0% 58 100.0% 17 100.0% 106 100.0%



Alternate Formats

Note: If you cannot access the documents that are provided in an alternate format, refer to the Help page.

Review of the Effectiveness of CIDA's Multilateral Delivery Channel (419 KB, 86 pages)

Appendix 3: Evaluation Scoring Chart (62.58 KB, 3 pages)
Appendix 5: Multilateral Organization Review Results (88.19 KB, 8 pages)


Notes

1. SGPB, is the most important CIDA Branch in terms of the institutional relationship with multilateral organizations, but this role is not exclusive. The Europe, Middle East and Maghreb Branch (EMM) for example, takes the lead with the European Bank for Reconstruction and Development (EBRD) and the United Nations Relief and Works Agency for Palestine Refugees in the Near East (UNRWA).

2. A very basic rule was used for assigning a given rating to each system under review: zero points where the system was essentially so weak as to be non-functional or seriously compromised, one half point where it was functional but lacking in some element(s) necessary for full effectiveness, and a full point where the system being reviewed was clearly producing reliable results and being used by internal and/or external stakeholders.

3. As noted under limitations, the Review team assigned these ratings based on a small number of interviews and limited review of documents. They are not as useful for comparing individual organizations as they are for assessing trends and directions among multilateral organizations as a group.

4. UNEG Principles of Working Together. P.1.United Nations Evaluation Group. 2007