Executive Summary
Introduction
(1) This report describes the results of the first pilot test of the draft guidance on evaluating conflict prevention and peace building (CPPB) activities produced by the Organisation for Economic Cooperation and Development – Development Assistance Committee (OECD-DAC). OECD-DAC Members recognise that there is a need to improve understanding of how aid contributes to ending or sometimes sustaining conflicts, and by extension how to evaluate such development assistance for lesson learning and improved aid effectiveness. The purpose of the test in Sri Lanka was to improve guidance, but also to provide useful lessons and recommendations for development actors in Sri Lanka as they develop new strategies for future engagement, or in some cases as they withdraw and take their experience elsewhere.
Methodology
(2) The evaluation concentrated on three tasks: (i) an assessment of donor strategies to examine how they approached working in and on conflict in Sri Lanka and their adherence to DAC evaluation criteria; (ii) a meta-evaluation of project evaluations to derive lessons on what worked and what didn’t work, to examine how evaluations were done and draw lessons around process; (iii) a review of how donors have coordinated peace building efforts so as to draw lessons for improving joint programming, funding, implementation and monitoring. Certain forms of CPPB activities were excluded as either being too sensitive in the current context in Sri Lanka, or beyond the scope of the team, including: Track 1 activities, diplomatic and political engagement and security reform.
(3) Instead of conducting a separate conflict analysis, as would be required by a full evaluation, the study used the existing comprehensive strategic conflict assessments (SCAs) conducted in 2001 and 2005 as its point of reference. The deepening nature of the conflict since 2005 is recognised, but our understanding as to the root causes of conflict remains as delineated in these earlier studies.
(4) The evaluation concentrated on three thematic areas of relevance to development practitioners in Sri Lanka: (i) peace building, (ii) governance and human rights, and (iii) conflict-sensitive socio-economic development. The study assesses which theories of change are used in CPPB work in these three areas, and how well they have been applied.
(5) Ten donors provided a sample of 17 published strategies, covering both the post-ceasefire period and the return to conflict. For the meta-evaluation, 28 evaluations were made obtained from 13 donors including examples of country, sectoral and project evaluations; with an even mix of peace building, governance, human rights and development interventions.
(6) In sum, this pilot exercise aims to fit with the current political space to evaluate CPPB work in Sri Lanka, but at the same time to add value to existing analysis, test selected parts of the Guidance, and improve donors’ understanding of their work as they plan for the future.
Context
(7) Sri Lanka’s multiple conflicts are the product of the formation of the Sri Lankan nation-state, particularly under British colonial policy, and of the reliance of the country’s political system on patronage. State formation encompassed the evolution of policies that did not protect numerical minorities or prevent discrimination. Militant responses emerged in the 1970s through Tamil youth, which eventually formed the basis for the Liberation Tigers of Tamil Eelam (LTTE), but armed insurgency has not been limited to the LTTE in the North and the East, and in fact started in the South led by Sinhala youth.
(8) For this study, three phases of the political and conflict setting in Sri Lanka are important: (i) the Pre-Cease Fire Agreement (CFA) period, (ii) 2002-2005, and (iii) 2005 onwards. In the first period, the country suffered volatility following three elections over 1999-2001, with deep political, economic and military crises. With the CFA in February 2002, the second phase started and the future looked promising – at least on the surface. The peace process generated much support but within a year the LTTE withdrew, despite international mediation efforts and donor conferences that led to $4.5 billion in aid pledges. The third and current phase since 2005 is one where the peace process has been abandoned in favour of a military solution to the conflict. The space for peace building initiatives has reduced and with the abrogation of the CFA in January 2008, peace building efforts at Track 1 and 2 stopped.
(9) In terms of development status, Sri Lanka has reached lower-middle income status and while its health and education MDGs are on track, consumption poverty reduction has been modest and uneven. The December 2004 tsunami worsened poverty levels in the affected areas, and poverty distribution remains highly uneven with the wealthiest 20% accounting for 54% of total income. The situation in the North and the East is much worse than the rest of the country: the ongoing war has destroyed businesses, roads and other structures, while security issues, constant displacement and limited resources has left the area more devastated than the rest of the country.
(10) While Sri Lanka is not regarded as aid dependent, since overall growth has outstripped aid growth, aid volumes in the past five years show a strong rise. The two development banks (the Asian Development Bank (ADB) and the World Bank) and Japan together account for 60% of total aid flows (2002-07) but have no mandate to work on political issues and are reluctant to work on governance beyond public sector reform and decentralisation. Bilaterals are either exiting or reducing their programmes as they see Sri Lanka as a middle income country and a government with a less reform-minded development policy and uninterested in negotiated peace. A newer group of Asian development partners have emerged with a pro-government stance, particularly China (the largest lender), India, Iran and Pakistan.
(11) While from 2002-04, the agendas of the donors and the GoSL converged, since 2005 their interest have diverged. Over the period, donors have recognised and better understood the significance of conflict, yet the space for engaging on conflict issues has reduced with the intensified war. Increased emphasis on global security and counter terrorism led to a change of attitude among some bilateral donors towards the LTTE and since 2006, the government has used this to muster support for its “eradicating terrorism” agenda. European donors have attempted to use aid to promote peace building and human rights, but tackling sensitive issues is more difficult with little financial leverage, and donors either seek to maintain good relationships and avoid difficult issues, or seek to withdraw completely from a bilateral aid partnership
(12) Aid coordination though challenging in the face of huge inflows around the peace process and then with the post-tsunami humanitarian effort, moved to a higher level of donor-government coordination in 2002-04. Rising numbers of ministries and NGOs complicated coordination work. New donors like India and China generally do not have state development agencies and are not actively engaged in formulating more coherent joint donor positions, and are not with the exception of Japan, OECD-DAC members.
Relevance of Strategies
(13) There has been an evolution of strategies over the 2002-08 period. In the post-CFA period, the rationale behind most strategies was couched in terms of protecting development assets and rehabilitation of war-damaged infrastructure, in the hope that such rapid and broad-based development would sustain and deepen peace.
(14) The SCAs have judged that the Sri Lankan conflict is at root configured around state power sharing, but this has not been addressed directly by most donors. Likewise, less attention was paid to other politically-sensitive dynamics such as the nature of the political system and longstanding problems of injustice and impunity. Instead, most strategies implicitly view under-development - or ethnically inequitable development - as a major contributory factor. Rather than focusing on the root causes of the conflict, they detail the “costs” – both social and economic – particularly on the North and East. We assess many of the interventions identified within strategies as not addressing the root causes but instead focussing on the consequences of conflict, on preferred government choices or on relief.
(15) Protagonists from different sides in the conflict were highly suspicious of the motives and consequences of large inflows of foreign assistance. As a result, they were not prepared to make high-risk political concessions in exchange for the uncertain promise of ‘development’. Peace had to be based on a political settlement, yet the international community found it hard to forego their assumption that offering a peace ‘dividend’ built on an expanding economy and improving livelihoods would resolve political differences and encourage such a settlement.
(16) In most of the strategies reviewed, peace building is addressed indirectly, implicitly or not at all. While power and governance issues are identified as central in the 2005 SCA, these issues are given less emphasis in strategies compared to socio-economic development and service delivery. Only a small number of donors provided direct support for the “peace process” (defined as Track 1 or Track 1½ processes). From 2005, some strategies reflect a more cautious approach, under a context of growing conflict, with plans to exit or at least avoid new spending.
(17) This prompted the use of scenarios in strategies as one tool to permit flexibility in uncertain conflict situations, however the mechanisms to track context to trigger adoption of different scenarios were left unclear. The devastating 2004 tsunami knocked many of the strategies off-track, and diverted them from long-term goals; but surprisingly strategies written subsequently do not acknowledge or mitigate the risk of such shocks. After 2005, donor strategies became increasingly unaligned and unharmonised reflecting the lack of a cementing peace process, the collapse of the government’s poverty reduction programme and donor imperatives (often headquarters-driven) to either disengage or to remain engaged.
(18) In nearly all strategies, there is weak explicit identification of a causal logic or theory of change between proposed actions and the achievement of peace. However, several strategies imply such logic especially those founded on building good governance (through stronger democratic institutions), public opinion (through media support), reconciliation (through community peace building), individual behaviour (through training, dialogue) and community reintegration (through resettlement and housing). There is thus the basis for a causal logic in many strategies but few fully test their logic against an analysis of the conflict or extrapolate them sufficiently to explore their peace linkages.
(19) Few donors appear to have commissioned explicit conflict analysis to inform their strategies. Notable exceptions are the USAID 2003-07 and SIDA 2009-10 strategies. While the jointly-commissioned 2005 SCA is widely referred to, it is not used to form the basis for positioning strategies to tackle conflict more appropriately.
(20) While most strategies were strongly aligned to government, especially in 2002-04, there was little recognition of the political risks of (a) delivering aid through a ‘state’ that is a party to the conflict and (b) supporting the economic and political agenda of a government that represented only a portion of the political spectrum and was vulnerable to electoral defeat.
(21) In general there was a weak approach to conflict sensitivity in early strategies (the 2002-05 period), but the trend was to consider this aspect more explicitly in the later strategies (2006 onwards), including introduction of a conflict lens by the World Bank and a set of rules to ensure sensitivity by the ADB.
(22) Efforts to peace build or transform conflict over-emphasised the extent to which civil society and citizens could bring about transformation and peace building. Some strategies are based on unrealistic aims and timeframes.
(23) Strategies are generally not strongly results-based and strategic achievements are not thoroughly evaluated, although some good examples do exist.
Approach of CPPB programmes and projects
(24) Most development and governance projects treat conflict as an external factor or risk to achieving the intervention aims, and in the immediate post-CFA period, adopted a post-conflict mind-set that saw them engage in reconstruction work under the assumption that the improved social and economic outcomes would support the transition to peace. From 2005, socio-economic development projects increasingly accepted the need for conflict sensitivity and “do no harm” principles, and dropped the notion of a “peace dividend” – as there was no peace process to underpin. Human rights and governance projects typically were conceived of as either contributing to peace or mitigating the political abuses generated by the conflict, but most did not address the fundamental political and governance issues upon which the success of the peace process depended.
(25) For peace building work, during the recent period of reduced space, there has been a concentration on supporting local initiatives through development approaches rather than more directly such as on human rights and at the “Track 1” level. Some donors saw development projects as providing the means by which they could explore doing peace building work in a politically sensitive environment.
(26) Several projects claimed to focus on conflict transformation through inter-ethnic initiatives and community peace building, but there is little evidence of how they explicitly addressed the driving factors of the conflict. Very few programmes also worked directly at trying to address the “Sinhala south” or to build a constituency for peace within the southern polity.
(27) After the tsunami, for many addressing the root causes of conflict was forgotten in favour of using humanitarian aid to achieve a peace dividend. Subsequently the understanding of the conflict as ideology-based appears to have been forgotten and largely replaced by the view that the conflict is an “ethnic one”, leading to a proliferation of inter-ethnic/co-existence projects.
Findings and use of CPPB evaluations
(28) Many CPPB evaluations tend to focus on results rather than outcomes, are based on partial evidence and are beset by a shifting context where project designs are changed as circumstances alter. The result is that evaluations are often premature and impacts are not given time to emerge. Findings are often sensitive in an ongoing conflict setting and this can limit their sharing and subsequent lesson learning. Evaluations have often been more concerned with lessons for future programmes than about the actual impact of the programme being evaluated. Some peace building evaluations have been too conceptual and the findings hard to apply. Some have focussed more on organisational aspects than on the impact of the initiatives.
(29) Despite the above, some evaluations have generated a number of useful findings around the effective delivery of benefits especially at the grassroots level, as well as other lessons on how conflict affects project performance. However the centralised nature of Sri Lankan politics creates a challenge for local level initiatives to have any real impact on peace processes.
(30) Governance and human rights projects generally have been more successful at addressing individual and/or highly localized needs than at promoting broader group-based or systemic changes.
(31) While there is a consensus among peace building activities of the need for a “no peace without justice” perspective, the majority of peace building activities take place where inequalities and oppression prevail and where conflict and violence have reinforced inequalities. Often it is hard to address justice, and when doing so the risks of repercussions are enormous. The dilemma of peace building / conflict transformation work generally is the relevance of a peace project when injustice and inequality are not addressed.
(32) Community based programmes aimed at building “capacities for peace” were more successful at addressing conflicts at the community level than in making the linkages from the local to the national. The most effective and sustainable results for peace building at the local level have been achieved through (i) village level empowerment by fostering community based organizations; and (ii) local business strengthening by supporting chambers of commerce, farmers and youth employment. The impact of so-called “co-existence” projects is patchy.
Process
(33) Donors’ evaluation work in Sri Lanka has limitations even without conflict issues – in terms of ability to conduct evaluations and to learn from them. Some of the largest donors do little independent evaluation of their portfolios. The climate of mistrust in Sri Lanka also means that information sharing is reduced and the willingness to discuss results and engage in joint government-donor-civil society efforts to learn lessons is limited.
(34) Most TORs prepared for socio-economic development evaluations don’t call for CPPB aspects to be addressed. Those evaluations that did examine peace building interventions mainly focused on relevance and efficiency questions and did not address impact.
(35) Few evaluations conducted their own conflict analysis or were able to draw on a baseline against which to gauge impact, and there are only a few examples where there is an explicit use of theories of change.
(36) Most evaluations were largely donor-managed exercises with some but limited consultation with the government. Opportunities have been overlooked by donors to conduct more joint evaluations, even where joint funding is in place.
(37) A shortage of consultants with the right evaluation and conflict skills, and shortage of institutional guidance on conflict sensitive evaluations, has impeded the quality of evaluations.
Coordination and coherence
(38) Coordination has declined from the relatively strong period around the ceasefire to a more polarized situation as the GoSL and LTTE moved back to a war footing. In general the level of coordination between donors and the GoSL has become increasingly difficult - and for some pointless. Regular coordination events between donors appear to have reduced apart from those related to humanitarian action.
(39) In terms of peace building coordination, the Donor Working Group (DWG) was the leading mechanism, though in representation terms the most junior of the three main coordination bodies. The DWG aimed to demonstrate donor commitment to the peace process and help ensure appropriate action, but in 2005 the follow-on Donor Peace Support Group (DPSG) covered different topics including monitoring peace and conflict dynamics, advising on how to contribute to peace and identifying joint initiatives. The idea of collaboration between donors and the GoSL was no longer mentioned.
(40) The DPSG has sub-committees on various themes but the design has received mixed reaction. Some favour the opportunity to pursue specific themes where common interest and expertise exists, while others regard the structure as over-elaborate and even irrelevant in a context where there is little or no space for peace building endeavours. There is also a leadership vacuum in the DPSG as the full-time facilitator does not have the mandate to take decisions, while the rotating chairs have full-time responsibilities elsewhere.
(41) In February 2008, more modest aims were introduced for the coordination arrangements: reducing the frequency of meetings and focusing on sectors. During a period when donors have been under increasing criticism from the government, there is a need for stronger coordination, yet the DPSG appears to have become weaker. The Trust Fund set up in 2004 to support the DPSG has produced some important analysis, but has been under-spent and not used productively in the recent past.
(42) Some international actors have jointly supported the capacities of civil society, especially as the opportunity to collaborate with GoSL proved more difficult. There are good examples of donors pooling resources to reach beyond their individual limits to enhance coordination with local actors. There is limited evidence of gender being taken into account in terms of coordination.
Conclusions
(43) An important overall conclusion to emerge is that explicit peace building measures that emphasise security and dialogue are not necessarily more effective in mitigating conflict than long-term socio-economic investments, in say education or rural development. From the evidence studied, and under the conditions where one or both of the parties to the conflict see the continuation of war as being preferable to a negotiated political settlement, peace building programmes seem to have had modest, if any, impact.
(44) Although the study scope has been narrowed, this in itself is a useful lesson in piloting guidelines of this nature in a context of open conflict, where both government and development partners are sensitive to external assessment, and where much of the information is confidential. Moreover, despite the restrictions placed on the work, the material reviewed and interviews conducted still form a substantial evidence base. A separate report will be prepared to capture the lessons learned from conducting this Sri Lanka evaluation.
Lessons and Recommendations
(45) The key lessons and recommendations are set out below, while a more detailed set is given in the main report.
Recommendations on Strategies for Development Partners in Sri Lanka
1. More rigorous use of conflict and political-economy analysis (individual or joint) will inform strategic choices for donor engagement in conflict settings. Joint analysis is preferable as it ensures greater ownership and wider understanding.
2. For both strategic and programmatic reasons it is important to be clear exactly which aspects of CPPB are to be addressed and what theories underpin how interventions are expected to make a difference.
3. Donors seeking to address the conflict need to look for strategic ways to address the fundamental issues driving the conflict, which in our view are the competing visions of the Sri Lankan state and disagreement over the distribution of power between the main political groups.
4. More careful consideration is needed of what can and cannot be achieved by offering or supporting a “peace dividend”. Past experience in Sri Lanka indicates that centre-based political fractures were a key cause of conflict that local development improvements could not address.
5. More use of scenarios / flexibility helps strategies to be responsive and to manage risk. When scenarios are used, then better means of tracking context and achievements in relation to the chosen scenarios is required.
6. Individual donor strategies will be better grounded when they recognise and declare institutional capacity and comparative advantage to work on conflict prevention and peace building.
7. Improve the indicators to measure strategic outcomes on conflict, and specify how they will be measured and what resources will be available to collect the data.
Lessons / Recommendations Relevant to Projects / Programmes
Flexibility and coverage
1. Short-term programmes on CPPB of 1-3 years can have positive effects, provided they have a narrow focus, specific objectives and a clear strategy for withdrawal.
2. Flexibility in choice of partners, in types of peace building support, and in funding channel (such as a grant facility) has proved effective in working on peace building in a volatile conflict setting.
3. Programme strategies need to be rethought in response to major shifts in the political environment. Donor programmes often either carry on as normal or shift a little – few take a step backwards and rethink strategy and implementation.
4. There is need to address horizontal inequalities both between ethnic groups and between geographic regions more effectively. More conflict transformation coverage is for example needed to address dynamics within the Sinhalese population in the South.
Coordination and partnerships
5. There is a need to build strategic co-ordination across work at different levels (i.e. Track 1 to Track 2 to Track 3, and linking national and local initiatives) for any future peace work.
6. It should not be assumed that civil society can be a major force in support of conflict transformation in Sri Lanka. Indeed, the role of civil society in this regard has been overemphasised.
7. Delivering through community-based organisations rather than NGOs can nevertheless prove effective in terms of grassroots empowerment and conflict mitigation, since they may be more independent of local politics (though not always), can mitigate insecurity and may be more sustainable.
Organisational issues
8. Conflict resolution and transformation organisations need to invest in building a common identity within the organisation. It is important to work on internal staff dynamics and cohesion in organisations and recognise this is also a part of conflict resolution/transformation work.
9. Conflict resolution/transformation organisations need to be especially rigorous in who they hire. Anyone with links – however tenuous – to a political party or with political connections – may open the NGO to perceptions of political bias.
Gender
10. Gender aspects have often been weakly addressed in many CPPB strategies and projects, even though gender-based discrimination has been cited as a cause for women’s recruitment into the military. Where grassroots’ interventions have targeted women, the results have often been positive.
Media
11. Media strategies for conflict resolution/transformation organisations should be developed earlier and in a more proactive way. Too often they are poorly developed and only done in response to negative publicity.
12. There is a need to expand programmes focussing on more analytical peace building content in the mainstream media that reach a large audience, such as radio programmes and print media. More direct interventions in media economics will help address falling revenues due to the effects of conflict.
Lessons / Recommendations on Donor Coordination
1. Coordinated action and sharing of responsibilities can help donors reach beyond their limits.
2. However setting up a DPSG to improve joint understanding and encourage joint action can achieve limited success where donors have strongly polarised positions with regard to a government that has little interest in a negotiated peace process.
3. Lessons learnt can be more useful and acceptable if developed through joint work than by single agencies (e.g. SCA1 is not mentioned by many as it is a single country-led assessment but the multi-donor sponsored SCA2 is appreciated by many).
4. There is potential for more joint evaluation, for example on how partners have provided support to NGOs, and on analysing appropriate methods for evaluating conflict sensitive development in Sri Lanka.
5. For coordinated donor policies to have a real impact on the ground, both newer and the larger donors need to be convinced to engage with the others fully. This will require finding areas of mutual interest around do no harm principles, and may preclude wider discussion on more sensitive issues.
Emerging Lessons and Recommendations for OECD DAC Guidance
The pilot exercise in Sri Lanka produced lessons to improve best practice in conducting CPPB evaluations:
1. Recent practice in designing evaluations of CPPB efforts in Sri Lanka have suffered from a number of limitations. These include how the TORs are written, the lack of conflict analysis, the limited use of joint evaluations and a weak focus on impact. The DAC Guidance already discusses these points but needs to be improved to help to address these gaps better.
2. The draft OECD guidance offers no guidance on evaluating support for formal peace processes in Sri Lanka (or elsewhere) (Track 1): though perhaps it is too politically sensitive an area to be evaluated using normal DAC criteria.
3. Most evaluations fall into the formative rather than summative category. That is they are mainly conducted with the aim of adjusting the ongoing programme or for the next phase, rather than for drawing out evidence of impact. They are held sometimes after only 2- 3 years of implementation, and focus more on early results rather than on impact. This indicates a need to improve the means of measuring outcomes of strategy on conflict and to find better indicators at this level.
4. Evaluation guidance should be tailored to the different types of programmes undertaken by donors. For example, questions that may be relevant for evaluating national level peace processes may not be relevant for local level conflict resolution activities. The differences are even greater between programmes that are actively supporting CPPB and those that are ‘conflict sensitive’ or working around the conflict.
5. Evaluation TORs should outline, or request the development of, more explicit theories of change to explain how assistance will actually deliver intended outcomes. The Sri Lankan evaluation suggests others including: 1) Faster and more equitable socio-economic development will reduce the grievances that cause or fuel conflict; 2) Local government can be more conflict sensitive because it is ‘closer to the people’ and therefore is more sensitive and responsive to local dynamics; 3) protection of human rights and the improved provision of justice will reduce the causes of conflict and contribute to peace.
6. In terms of obtaining and using evaluators:
i. The shortage of consultants with a suitable mix of evaluation and peace building / conflict resolution experience means that evaluation commissioners need to plan in advance and be flexible in timing to ensure the best results.
ii. Since conflict “experts” often bring their own understanding of conflict, when hiring consultants, donors need to be clear about the ‘school’ of conflict expertise they need.
iii. There is a potential to moderate the biases that evaluators bring their to the conflict setting, so as to mitigate how evaluators are perceived and how findings are collected and interpret.
7. Because of the additional difficulties of conducting evaluation fieldwork during an ongoing conflict, there is a need to allow additional time for preparation and to expect delays if more reliable and representative evidence is to be obtained.
