Introduction
Conservationists are striving to improve their monitoring and evaluation practices. This is important for two reasons. Firstly, conservationists must be more accountable for failures and successes, both to the intended beneficiaries of the work and to those who fund it (Cleary, Reference Cleary2006; Ferraro & Pattanayak, Reference Ferraro and Pattanayak2006; Higgins et al., Reference Higgins, Touval, Unnasch, Reichle, Oren, Waldman and Hoekstra2006). Secondly, good project management is integrally linked to good monitoring and evaluation systems (Stem et al., Reference Stem, Margoluis, Salafsky and Brown2005). Yet, despite these efforts, there is uncertainty about how to approach the task of measuring conservation success, from determining the right level of investment (Salzer & Salafsky, Reference Salzer and Salafsky2003) to choosing the correct tools (Salafsky & Margoluis, Reference Salafsky and Margoluis1999). Moving beyond progress monitoring of activities and outputs to results-based monitoring is a particular challenge.
Conservationists are trained to monitor trends in biodiversity (Balmford et al., Reference Balmford, Green and Jenkins2003) but conservation success cannot be determined solely through biological monitoring. With socio-economic and political forces at the root of most conservation problems (Kleiman et al., Reference Kleiman, Reading, Miller, Clark, Scott and Robinson2000; Robinson, Reference Robinson2006) the task of conserving biodiversity is increasingly approached through community-based interventions (Walpole & Wilder, Reference Walpole and Wilder2008), and the success of these interventions must be tracked (Margoluis & Salafsky, Reference Margoluis and Salafsky1998).
To monitor socio-economic outcomes and impacts, natural scientists generally err towards indicator-based, quantitative techniques: questionnaires, household surveys and material wealth rankings (Bernard, Reference Bernard2006). These techniques are replicable across sites, produce data that are amenable to statistical analysis, and demonstrate progress within a conceptual model or logical framework. But they are time-consuming, expensive, and unsuitable for providing in depth understanding of an issue (Marsland et al., Reference Marsland, Wilson, Abeyasekera and Kleih2001). The data lack the contextual information that helps clarify the causal link between an observed change and the project activities, overlook unanticipated changes, and fail to unearth any flaws in a conceptual model or logical framework (Whitehouse, Reference Whitehouse and Cummings2005).
In a search for more complex data some conservationists also employ qualitative, participatory techniques such as semi-structured interviews and focus groups, which produce rich narrative data, demanding different analytical skills. Yet even these are not sufficient to track changes that, though subtle, are critical for the success of people-centred conservation, such as changes in community awareness, attitudes or behaviour, improvements in social cohesion or well-being, or increased empowerment.
This type of information is often found in anecdotes told by project staff or the intended beneficiaries themselves. Because such anecdotal evidence is rarely collected systematically or recorded formally in reports, it is often overlooked or discarded as insufficiently rigorous. Dismissing these data not only fails to report important project successes but means there is no system for learning from experiences and managing adaptively. As empirical evidence (Sutherland et al., Reference Sutherland, Pullin, Dolman and Knight2004; Pullin & Stewart, Reference Pullin and Stewart2006) may not always be available, particularly for social changes, efforts must be made to use systematically the information that is available, because enhancing the ability to learn from experience will have a significant influence on the effectiveness of conservation outcomes (Fazey et al., Reference Fazey, Fazey, Salisbury, Lindenmayer and Dovers2006).
In an effort to improve learning from experience in conservation, Fauna & Flora International has been investigating the use of one monitoring and evaluation method that, when used in conjunction with other conventional methods, helps to overcome some of the challenges. This paper explains the strengths and weaknesses of the method, describes our experiences with it, and makes recommendations for its use in conservation.
The Most Significant Change method
The Most Significant Change (MSC) method is one of a number of non indicator-based monitoring methods used in the socio-economic/development sector for assessing outcomes and impacts (Table 1). MSC was pioneered in a participatory rural development programme in Bangladesh (Davies & Dart, Reference Davies and Dart2005). It is a way of systematically collecting the anecdotal evidence of change that is missed by conventional monitoring techniques. The method provides evidence for the impact of a project as a whole, and promotes organizational learning within the project team, thereby contributing to evaluation and adaptive management (Dart et al., Reference Dart, Dysdale, Cole and Saddington2000; Dart & Davies, Reference Dart and Davies2003). In various forms MSC is now used worldwide by the development sector, and in environmental sectors in Australia (A. McGregor, pers. comm., 2007; J. Dart, pers. comm., 2007), but has only recently been used in the conservation sector. It is employed as an evaluation tool for coalition NGOs in Ghana (R. Dottey, pers. comm., 2007), for social impact assessment of conservation projects in Vietnam (Dawkins, Reference Dawkins2007), and for participatory film making in the Mekong Delta (Dubois, Reference Dubois2007).
Table 1 The relationship of the Most Significant Change (MSC) method to other non indicator-based monitoring and evaluation methods.
MSC is participatory, with project staff and stakeholders contributing to the documentation of evidence of project outcomes and impacts in the form of ‘significant stories of change’. These are not fictional stories but factual descriptions of observations or experiences backed by evidence. This qualitative information is gathered throughout the project cycle and discussed by project staff at regular meetings. The most significant stories are systematically selected and passed between the layers of an organization and feedback is provided to project stakeholders, so enabling both upward and downward accountability (Fig. 1).
Fig. 1 Example of a basic Most Significant Change system, illustrating the flow of stories between various groups within a project management system.
Unlike many other forms of monitoring MSC does not use narrow, predefined indicators. Rather, staff and stakeholders collect stories of change within broad categories, or domains, which relate to the project objectives but are not so restrictive that unexpected outcomes and impacts are overlooked. Although there is no strict format for stories, they are recorded on a specially designed data sheet to ensure a level of consistency (Appendix 1).
The system is highly malleable and is adapted to fit the local situation. It involves participation from as wide a range of stakeholders as is appropriate for the project situation. In some circumstances project beneficiaries themselves record stories of change, whereas in other situations stories originate from the observations of field staff.
The process
The Most Significant Change process is described briefly below (it is described in more detail by Davies & Dart, Reference Davies and Dart2005).
1 Assessing the situation for suitability and adapting the system to meet local needs and context. Domains relating to the programme's objectives are selected, and a data sheet for story collection developed.
2 Training teams and identifying so-called champions who can maintain motivation and keep the system on track. Ongoing training of teams is needed to improve observation and story recording.
3 Collecting stories of change. Field teams collect stories (including evidence for the change) as they go about their work, and record these on the data sheets.
4 Discussing and selecting the most significant stories. At regular meetings (their frequency depending on individual situations), each field team discusses the stories they have gathered. They consider the reasons for the change, its implications for the programme, and whether any response from the team is required. The group then decides which are the most significant stories, and why. Through a process of Summary by Selection, a smaller selection of stories is passed to the next organisational layer, and considered alongside stories from other field teams.
5 Feeding back the results of the process to other organisational layers.
6 Verification of the most significant stories through triangulation where possible. This is particularly important for stories passed to higher organizational layers where those assessing the stories are further removed from their origin.
7 Quantification and secondary analysis. Stories can be collated for meta-analysis to identify trends, and provide information about how well the system is working.
8 Dissemination to other parties. The data can be used in reports and proposals where appropriate, or shared with peers.
9 Fine-tuning the system where necessary, after several rounds of data collection.
Strengths
When MSC is used in conjunction with other monitoring methods it has various strengths. The system focuses on observed, demonstrable change, i.e. outcomes and impacts, rather than activities and outputs. It also enables documentation of change that would be missed by conventional indicator-based monitoring. For example, qualitative changes (such as change in community attitudes or change in social cohesion), and intangible changes (such as community well-being or improvements in project-community relations and trust), which are essential to the success of some projects, cannot be quantified easily without losing context. Unlike the data produced by simple indicator-based monitoring, the stories of change provide contextual evidence, including discussion of why the change happened, to understand causal links better. Likewise, MSC can capture changes that were unanticipated at the outset of the project and thus were not included in an indicator-based regime. The system also generates negative stories, which are essential for adaptive management, allowing managers to determine the success or failure of project activities, rather than simply whether they have been implemented.
Providing feedback to project stakeholders from the discussions and analysis of the stories is an integral part of the system, enhancing downward accountability and building the trusting relationships upon which successful community interventions depend. The documentation and the feedback systems also promote organizational learning and a culture of openness and transparency. Furthermore, the data gathered reflect what the stakeholders view to be the most important changes caused by the project, thereby helping managers understand whether the project is meeting local needs and objectives, and promoting examination of higher programme goals. This is a monitoring tool, gathering information throughout the project cycle, but because it focuses on outcomes and impacts the data can also contribute to more in depth evaluations of project achievements. Finally, MSC is not proscriptive: the system is intended to be adapted to fit the specific context of each project.
Addressing potential weaknesses
Because the system functions without indicators based on project objectives, some fear that it cannot track progress towards these objectives. However, so-called domains are selected based on the objectives so that the majority of the stories produced will relate to the objectives in some way. In addition, MSC is designed to be used in conjunction with indicator-based methods that track project progress towards project goals and objectives. The information produced by MSC can complement and augment that produced by traditional systems, help inform choice of indicators, and inform decisions about future project direction (Marsland et al., Reference Marsland, Wilson, Abeyasekera and Kleih2001).
The data are not collected on the basis of a random sample but this system of gathering data on exceptional circumstances (significant successes or failures) rather than the average condition of beneficiaries, known as purposeful sampling, is a legitimate form of sampling in qualitative research. One can learn more from unusual, information-rich cases than from ‘statistical depictions of the average case’ (Patton, Reference Patton1990). A combination of MSC with a conventional random sample technique that can make generalizations about the experience of all participants will yield the most comprehensive picture.
The reliability of data sources could be questionable, yet verification of stories is an important step in the process, carried out using methods that include informal verification by field staff and more formal triangulation using complementary data. The process of selecting the most significant stories is of course subject to the values and perspectives of those doing the selection but subjectivity can be reduced by developing criteria for selected stories according to project objectives. MSC emphasizes transparency by ensuring that the reasons for selection are recorded on the data form. Some may also consider the system liable to positive bias through the deliberate selection of stories of success. However, experience from the development sector (and our own experience) shows that as many negative as positive stories are produced and selected (Davies & Dart, Reference Davies and Dart2005).
FFI's experiences with MSC
Fauna & Flora International's (FFI, 2008) Biodiversity and Human Needs programme (Walpole & Wilder, Reference Walpole and Wilder2008) has been exploring alternative ways of measuring the outcomes and impacts of its livelihoods-focused interventions. We began piloting the MSC system in 2006 in three well-established projects (one each in Cambodia, Tanzania and Vietnam). Pilot projects were self-selected, and the technique was implemented with varying levels of intensity, according to local circumstances. We have invested most effort in the pilot in Cambodia, and therefore experiences from this project form our principal case study here.
The Cambodian Elephant Conservation Group (CECG) is a collaboration between FFI and the government departments responsible for forest management and the environment, with the goal of ensuring ‘elephant conservation stakeholders act strategically and proactively to stabilize and/or increase wild populations' (Heffernan, Reference Heffernan2006). Forest area is diminishing in Cambodia through small-scale encroachment by poor farmers and large-scale landscape development. With its natural habitat decreasing, the wild elephant population is increasingly coming into contact with poor farming communities living on the forest periphery, and destroying crops, damaging property and occasionally harming people. Such human-elephant conflict can ruin a poor farmer's livelihood and lead to retribution killings of elephants (Sitati, Reference Sitati2007; Walpole & Linkie, Reference Walpole and Linkie2007).
The majority of CECG's activities focus on reducing human-elephant conflict, with the aim of increasing farmers’ tolerance towards elephants and helping them understand the importance of conserving these animals and their habitat. CECG works with farmers to implement low tech deterrent and mitigation methods such as establishing simple alarm systems, growing less elephant-palatable crops and encouraging the formation of guarding groups. CECG also works at the national level, engaging with authorities and decision makers to improve conservation in the country as a whole. CECG works in two field sites, one each in the north-east and south, each of which has a field team working with afflicted farmers. The teams are coordinated by two team leaders based in the national office in Phnom Penh with the CECG technical advisor. Field staff are closely linked to the beneficiary communities, either actually living in the community or originating from it. In areas where an ethnic language is spoken the field teams contain at least one member from the ethnic group.
CECG keeps an extensive quantitative database of all human-elephant conflict to determine changing patterns of conflict, including the number of incidents, amount of damage, seasonality, determinants of susceptibility, and uptake of mitigation methods. However, to determine the effectiveness of their methods CECG required information on the following: (1) Are communities’ attitudes towards, and awareness of the importance of, elephant conservation changing? (2) Is this reflected in a change in the communities’ behaviour? (3) Are mitigation methods working or not? (4) Are livelihoods changing, for better or worse? (5) Are these changes a result of CECG's interventions or of some other force? (6) Are CECG activities having any unexpected outcomes? CECG felt their conventional quantitative monitoring systems, with narrow, predefined indicators, were failing to capture the rich contextual information necessary to answer these six questions properly. Their search for other, complementary methods led them to become FFI's first MSC pilot site. The aim of the pilot process was to understand whether the MSC system could be used in this community-based conservation context, and the challenges, benefits and whether there were any unanticipated outcomes of using the system.
Implementation in Cambodia
The MSC system was initiated in March 2006. The CECG team were familiarized with the concept, practicalities, and strengths and limitations of MSC. Together, we developed the domains based on the CECG objectives and adapted the generic MSC system to fit the structure and needs of the Group (Appendix 2). Stories of change were recorded by the field teams, who either recorded their own observations or elicited stories directly from community members on an informal basis during their regular interactions. Detailed, written instructions for MSC implementation were developed and, through an iterative process of trial and error over several months (through project visits and remote mentoring), the system was refined, the domains clarified and the data sheet adapted (Appendix 1). Additional training was provided in story-recording and the use of the data sheet. The following are summarized examples of stories, three each of positive and negative or unexpected change.
Stories of positive change (1) A simple alarm system constructed by CECG to warn farmers when elephants are approaching means that they do not have to keep watch throughout the night. Well-being has improved and, consequently, attitudes (crop yield has also increased due to improved guarding techniques, which can be verified through the human-elephant conflict incident database). Whereas previously villagers were angry with the elephants, they are now worried about the disappearance of a familiar elephant calf. Villagers say that this change in attitude is because they no longer raid crops. (2) Advised by CECG, a farmer planted crops that are unpalatable to elephants (radishes and peanuts instead of rice). The elephants did not destroy these crops, and she took them to market. CECG supported a farmer to start growing cassava, which is not susceptible to raiding by elephants. Because successful cassava cultivation has improved his income he no longer needs to make charcoal (and thus destroy elephant habitat) to make a living, and he has destroyed his charcoal kiln. CECG helped a community to form a guarding group that uses drums and torches to scare the elephants, and crop yields have subsequently increased. (3) The alternative crops advised by CECG must be taken to market. In previous years CECG helped villagers find suitable markets and transport crops but in 2006 they were sufficiently confident to take the crops to market independently.
Stories of negative or unexpected change (1) A farmer whose crops were destroyed is planning to construct an elephant trap. (2) A dam system implemented by CECG to improve farming productivity for afflicted farmers is causing conflict over water supply downstream. (3) CECG advised an afflicted farmer to plant peanuts rather than rice; elephants do not eat the peanuts but trample them.
Field trials in Cambodia
The MSC system was reviewed one year after implementation. We spent time with the Team Leaders and, with the help of a translator, with each field team, attending bimonthly meetings in the provinces and discussing progress in implementing the system. We also assessed the value and limitations of MSC as a monitoring tool above and beyond the existing CECG monitoring framework.
Experience implementing MSC After one year we found that the understanding of the concept of MSC amongst the CECG Team Leaders had improved, and the level of our discussions had risen. The Team Leaders themselves reported that their own capacity for analytical thought had increased and attributed this change to MSC. Both of the Cambodia field teams are diligently implementing the system and producing bimonthly stories. Team Leaders provide ongoing training to their field teams on story collection, the quality of which increased considerably. All stories now focus on specific events and individuals rather than generalized issues, as had previously been the case. In the bimonthly meetings the teams engaged in lively, participative discussions about each story, debating its verity and value and whether it could be considered a significant change.
MSC as a monitoring tool To be a valuable addition to project monitoring, MSC must provide useful, relevant and verifiable evidence of change that is additional to that provided by existing monitoring efforts, and that is simple and cost-effective to collect. Experience from Cambodia suggests that these criteria are at least partially met. The CECG teams report that MSC helps them record changes that have taken place as a result of their interventions, whereas before (using only human-elephant conflict monitoring forms) they just recorded information about elephant damage (crops targeted, extent of damage, meteorological conditions, state of the moon, size of the elephant's footprint, types of deterrent used, and the success or otherwise of the deterrent). Whilst this can be used to monitor changing patterns of conflict and make inferences about the role of mitigation methods, it requires time and analytical expertise to utilize and interpret the data fully. In contrast, MSC stories provide information on a wider suite of changes such as changes in attitudes, behaviour, well-being and livelihoods (in the broadest sense of the term) of the intended beneficiaries, and in a format that is easily accessible. Moreover, the broad domains also allow the CECG teams to record and respond to unexpected changes, obstacles and problems, as well as the expected changes. Although some of these data could potentially be detected by conventional interview techniques, the more intangible and unexpected changes, and contextual information about causal links, would be missed. The Team Leaders find the system particularly beneficial because it helps them to monitor change systematically and continuously, and adjust their work plans accordingly. It also allows them to report on CECG's impacts. All three pilot projects have used significant stories in donor reports or proposals, and these have been received positively by donors. The project meetings enable the two field teams to exchange methods and share knowledge. The system also provides an opportunity to learn from both mistakes and successes. Others in the wider national CECG team commented that the MSC discussions provide useful information for sharing CECG's lessons with other NGOs operating in similar fields.
Unexpected benefits for project management Many stories are not about changes arising from the project but of issues that required a reaction from the field team. These stories receive the most attention during the meetings, and by providing a forum for discussing these stories, there is a systematic way of ensuring that such issues are dealt with. At the end of each meeting a simple list of actions is made, with clear responsibilities, and checked at the following meeting. Thus, MSC also provides a way of strengthening adaptive management, which was deemed by the CECG team to be one of the most important functions of the system. MSC promotes feedback to the communities on issues they have raised with the field teams, thus building communities’ trust and improving cooperation, which are vital for CECG success. The Team Leaders also noticed that their field teams have increased in confidence and capacity as a result of MSC. Being involved in discussions and critical reflection about project outcomes, analysing stories and evaluating which changes are the most significant and why, has improved awareness and understanding of the outcomes and impacts of their work. The field teams now analyse situations and issues and decide how to respond to them without constant guidance from Team Leaders, working together better as a team.
Limitations of the system
MSC requires considerable investment of time and effort, not only in establishing the system and tailoring it to a project's needs but also in the continuous collection of stories, meeting for discussion, and feeding back to the field. The field teams found it logistically difficult to bring the whole team together. The teams agreed, however, that this time and effort was worthwhile. It should not be underestimated, however, how difficult the concept of MSC can be for project staff accustomed to quantitative monitoring. Areas of particular difficulty were the idea of monitoring without indicators, the meaning of domains, the information required when recording a story, and how the most significant stories should be selected.
Staff changes led to the quality of data from one team being substantially lower than the other, in which the Team Leader had been inducted in the method, worked closely with us, and fully understood the potential of the system. This demonstrates the importance of champions for a fledgling MSC system. It is essential that some team members can promote the value of dedicating time and effort to observing changes, recording stories, meeting regularly and reflecting analytically. This is reinforced by our experiences with the other pilot studies.
Field teams found some parts of the form difficult to complete. Some of the questions requiring analytical thought, in particular the question ‘Why is this story significant?’, were rarely answered well. Although the complexity of the method can be tailored to suit the capacity of the team, recording the reasons for selection of a story is essential for transparency and an integral part of MSC (Dart, Reference Dart2005). It is therefore necessary to strike a balance between oversimplifying the system and risking devaluing the data, and being overambitious and failing to capture any data at all.
Verification is an important issue. Field staff are sufficiently familiar with the project to be able to identify false claims, and most verification is currently informal through discussion amongst the team and, where necessary, further discussion with the subjects of the story. However, this system is not infallible; field staff inevitably have their own ties and biases within the communities. Formal, traceable verification, although it is a considerable burden, is essential, particularly for the significant change stories passed to higher levels within the organization that are used to examine project performance. Formal verification of stories about reduced crop raiding could be achieved through triangulation using the human-elephant conflict incident data. In the case of the most significant stories, to be used for external reporting, a follow-up evaluation mission may be useful to achieve more in depth verification and determine the sustainability of the changes that are claimed.
In the CECG all stories were initially written in the Khmer language, and only the most significant (those selected for the national meetings) were translated into English, with the preliminary discussion and verification of stories being carried out within the field teams in Khmer. Translation is resource-demanding and introduces an additional layer of potential bias. The need for translation may also hinder subsequent rechecking and verification of stories by others. The system produces copious hard copy data, and therefore a simple electronic database has been established for reference and for producing information for donor reports.
Although the system functions effectively for gathering information on progress towards the objectives of improving livelihoods and community attitudes, it proved less feasible to collect information in this way on changes in the wider political and institutional environment in which CECG operates. The scope for collecting candid stories of change may be limited by political sensitivities, either at the national institutional level or, in some project situations, at the project level.
Experience using MSC in other projects
In Tanzania FFI's partner, the Mpingo Conservation Project, was already using a survey-based socio-economic monitoring system to track the impact of its community forestry work. Seeking a deeper understanding of the communities than these surveys allowed, and a more participatory method, we developed an MSC system that involves stories told by community members at village meetings. The most significant of these are taken to the head office where they are discussed alongside stories from other stakeholders, such as NGOs, local government officials, and project staff. The results of the discussions are used to inform project decisions and are fed back to communities, and the significant stories stored in a database for reference.
In Vietnam FFI managed a pilot community-based conservation project with many stakeholder groups. MSC was introduced in the closing stages to garner evidence of the project's impact on non-financial livelihood capital and well-being, and to learn lessons for post-pilot activities. Stakeholders discussed their stories at an open meeting, and one story from each meeting was recorded by project staff and passed to the central office. This was the simplest version of the system we used, intended to reduce burden on already overstretched project staff.
During the Vietnam pilot process, staff increased their focus on the purpose and effect of their work. However, the system was introduced late in the project and floundered because the necessary financial and human resources were not built at the outset and other pressures on the team were too great (Swan, Reference Swan2006). Conversely, in Tanzania, where capacity and motivation are higher and resources budgeted, the system is progressing more smoothly (S. Ball, pers. comm., 2008).
In both Tanzania and Vietnam the system collects stories through community meetings, which introduces a significant risk of elite bias (Madey, Reference Madey1982; Sandelowski, Reference Sandelowski1986), whereby only the voices of the strongest in the community are heard, and the data gathered are thus not representative of the whole community. Despite this potential bias, MSC provided an opportunity for the teams to rethink the impact of their work, and this has been beneficial for project implementation (Ball, Reference Ball2006, pers. comm., 2008; Swan, Reference Swan2006, pers. comm., 2008).
Discussion
There is a recognized need to overcome the barriers between natural and social science disciplines in conservation (Campbell, Reference Campbell2005; Balmford & Cowling, Reference Balmford and Cowling2006; Fox et al., Reference Fox, Christian, Nordby, Pergams, Peterson and Pyke2006). Socio-economic monitoring and evaluation is an area where increased interdisciplinary collaboration would be particularly beneficial. A recent study of the methods used by six international development NGOs (Urveeja Bose, Reference Urveeja Bose2007) revealed that social scientists and the development community have extensive experience of measuring complex changes and employ a wide range of tools for capturing and using qualitative and narrative data. Conservationists can learn from these organizations, both from the methods they employ and from their institutional attitudes towards monitoring and evaluation and organizational learning.
This description of an attempt to employ a qualitative, non indicator-based monitoring method from the development sector in a conservation setting shows promising results. In our Cambodian example the Most Significant Change method provided a simple means of expanding the range of relevant information on project outcomes from quantitative data on crop damage to a much broader array of social changes, providing information in a usable form that has benefited project management and internal and external project communication. The fact that project staff have embraced the approach, when monitoring and evaluation is so often neglected, provides one indication that it has some value.
MSC does, however, face difficulties. It can be difficult to convince people of the value of collecting unfamiliar forms of data and it is time consuming, and therefore costly, to establish and implement. Parts of the process, including the gathering, translation and verification of stories, and the process of choosing the most significant stories, need careful attention. Translators, and even field teams, can act as brokers of meaning, potentially influencing stories according to what they perceive to be required by the project, thus producing a social construct in which the values of the field team and translators shape what is heard and how it is recorded for others (Lewis & Mosse, Reference Lewis, Mosse, Lewis and Mosse2006). In the Cambodian context, where the culture is one of respecting social hierarchy, keeping one's place within it, and never losing face, this is an important issue (O’ Leary & Nee, Reference O'Leary and Nee2001). Furthermore, although we rely on field teams being close to the community, the community is not homogeneous, and the field teams many not detect or report stories from some sectors of society, typically the most poor and marginalized and those involved in illegal activities. Development organizations in Cambodia are constantly challenged by these issues, and field staff and translators who appreciate the subtleties of data gathering and translation are highly valued (S. Milne, pers. comm., 2008).
However, our experience demonstrates that the system can produce benefits, both unexpected and profound. In Cambodia the MSC meetings at both provincial and national levels provide a forum for the teams to debate issues that are rarely discussed during day-to-day implementation of field activities, and in a more reflective and critical manner. As well as having benefits for organizational learning, this has led to an improved understanding of the outcomes and impacts of the teams' work and to considerable, and unexpected, improvement in team capacity. This has been found by users of MSC in non-conservation contexts (Dart, Reference Dart2000; Sisgaard, Reference Sigsgaard2002; Winterford, Reference Winterford2003). MSC has provided many stories illustrating project success but in the CECG situation the adaptive management element is proving at least as valuable.
The level of effort to implement the system, and the quality of data it will produce, depends on the initial capacity of the team. As with any monitoring system, success is more likely if human and financial resources are budgeted, and the team are motivated to participate fully, at the outset of a project. The benefits we have discussed are valuable but without the considerable, enduring effort of internal and external champions, the system would not have been successful. External mentoring and refining of the CECG MSC system will be required for some time before it is self-sustaining.
MSC may not be appropriate in all contexts. It may not prove cost effective in situations where the capacity of the team is limited, although, it could be argued that it is precisely in such situations that the most profound benefits, such as critical reflection and increased adaptive management, are felt. With sufficient support and the right conditions, MSC can be sustainable within a project context, and institutionalized in the longer term. Continued feedback to the community level will propagate people's incentive to provide stories of change. The CECG system is still functioning usefully after 2 years, without regular technical support during the second year.
It is critical to note that MSC is complementary to, not a replacement for, conventional indicator-based monitoring. The system is not universally suitable, and is certainly not a panacea for the difficulties of socio-economic monitoring in conservation. For instance, MSC will not provide the empirical data that some strive for (Wilkie et al., Reference Wilkie, Morelli, Demmer, Starkey, Telfer and Steil2006). When presented at a recent workshop on measuring the impacts of livelihoods initiatives in the conservation context (Walpole et al., Reference Walpole, Elliott, Granziera, Thomas and Wilder2007), the concept was met with a mixture of intrigue and scepticism. Yet, after considerable discussion, most participants recognised its value and were interested to learn more.
We therefore argue that as long as the challenges and constraints are understood, the system, with its ability to demonstrate various forms of socio-economic change, can be a valuable addition for monitoring and improving the impact and accountability of conservation. The brief trials of the method we have described need expanding upon in other conservation contexts but, based on our experiences, we can make some practical recommendations on the use of MSC in conservation projects:
(1) A prerequisite for attempting MSC is an organizational culture of learning, open to new systems, and discussing failures and challenges as well as successes.
(2) It is essential to allow substantial time and resources for training staff, developing the system, and mentoring thereafter. The initial input required will vary with team capacity (in Cambodia this process took 2 weeks of intensive input at the outset, and remote mentoring throughout the first year). Identifying a champion within the team, who can encourage others, train new staff, and be the main point of contact for remote mentoring, is helpful. It may be beneficial to recruit staff with a background in social science, or enlist them as local mentors.
(3) Using MSC as a tool to capture stories from individuals in the field, rather than during village meetings, reduces the risk of elite capture. Translation of stories is, in many cases, an unavoidable burden. Translators should be chosen carefully, to reduce the risk of bias or interpretation. It is important to build in a robust system of verification, ideally through triangulation with other sources of data, particularly for stories that are to be used as evidence of change. The process of choosing these stories should be carefully documented, and criteria for choosing can be used to lessen the subjectivity of this choice.
(4) If possible, the MSC system should be embedded at the outset of a project, and the meetings held regularly. A simple database should be established to manage the data, ideally alongside the other complementary monitoring data.
The system is not suited to every situation. Being a considerable burden to establish and maintain, it is important to consider carefully whether MSC is appropriate before using it. In general, MSC is most suited to projects that are complex, with divergent outcomes, have many sites and organizational layers, are participatory and focused on social change, and have regular contact between field teams and communities.
In adopting techniques that are traditionally the terrain of the development sector, and embracing their approaches, we are taking an important step towards the multidisciplinary approach that is required in modern conservation (Adams, Reference Adams2007). As we continue to explore the use of this method we welcome suggestions from others grappling with the issue of monitoring social change, and echo the call for continued and increased collaboration between the conservation and development sectors.
Acknowledgements
We would like to thank the Dutch Ministry of Foreign Affairs (DGIS) and British Airways for their financial support. We would also like to thank Rick Davies for his advice, the project teams (Cambodian Elephant Conservation Group, in particular, Joe Heffernan, Tuy ‘Vathana’ Sereivathana and Kannitha Lim; the Mpingo Conservation Project, in particular Steve Ball; the Hoang Lien Son Project, in particular Steve Swan), and Sarah Milne of the University of Cambridge Geography Department.
Appendices
The appendices for this article are available online at http://journals.cambridge.org
Biographical sketches
Lizzie Wilder worked with the IUCN Sustainable Use Specialist Group before joining FFI's Biodiversity and Human Needs Programme in 2004. She focuses on promoting understanding and better practice around the issues of conservation and poverty alleviation, with a particular interest in socio-economic monitoring and evaluation. Matt Walpole specializes in interdisciplinary approaches to conservation research and practice in the developing world, having worked on human-wildlife conflict mitigation and the role of ecotourism as a conservation and development tool. From 2004 he directed FFI's Biodiversity and Human Needs programme, exploring wider links between poverty and conservation and developing staff and organizational capacity to deliver multiple benefits from FFI's field programmes. In 2008 he joined the Ecosystem Assessment Programme at the UNEP-World Conservation Monitoring Centre.
