Voices of Verity
Anindo Banerjee
| This write-up is a tribute to the spirit of the community representatives involved in terminal evaluation of Community Based Pro-Poor Initiatives (CBPPI) programme in India (supported by United Nations Development Programme), who have been a great source of strength and support to their families and communities by virtue of their relentless struggle for a life of dignity. The write-up primarily seeks to reflect upon the processes and sidelights of a community-led evaluation exercise, and doesn’t aspire to report the findings per se. The experience reaffirms the capability of the poor (the team of evaluators being predominantly a representation of the poor, including tribal people, socially disadvantaged castes and women) to undertake evaluations of such scale that were hitherto bastions of ‘experts’. For many of us in Praxis (the facilitating agency), the experience emphasizes the criticality of community leadership, and shows the way forward. |
When Chitri Mali – the outspoken community leader hailing from a remote tribal hamlet of western Orissa alighted from behind the smoky windows of her train, accompanied by her equally flamboyant colleagues bedecked in shining bracelets and characteristic tribal attire, the hosts from a local NGO could hardly imagine the visitors to be the long-awaited emissaries from UNDP commissioned to carry out term-end evaluation of the NGO’s interventions over the past five years. During the week that followed, the visitors embarked upon an unprecedented trail of investigations, undeterred by a barrage of sneering remarks. The functionaries of the host NGO soon came to realize how erroneous were they in their attempts to size up the capabilities of their guests. A few days later, when the group presented their observations and findings before an elite panel of experts, the NGO representatives learnt many a valuable lesson – not merely regarding the shortfalls of their work, but more importantly, relating to the blunder of underestimating the capabilities of their under-rated visitors.
The run-up to the community-led evaluations
Although the idea of deploying community representatives to evaluate the achievements and impact of the Community Based Pro-Poor Initiatives (CBPPI) programme came about initially on an experimental note (one of Praxis' training alumni - Ms. Shashi Sudhir - drove the idea within UNDP), it met with tremendous responses of support from nearly all the highly-placed people who witnessed the presentations made by illiterate women and men at the UNDP (United Nations Development Programme) office in New Delhi after various teams of community representatives returned from their month-long visits to selected projects located in various states of India. As many as 13 major projects located across 11 states of India were evaluated in the process, by a team of 26 community representatives who were divided into four teams for operational convenience.
Each team of community representatives was accompanied by a carefully selected facilitator, whose main responsibility was to provide secretarial and logistical assistance to the respective teams for executing the evaluations and for writing the reports. The facilitators were selected and provided by Praxis (Institute for Participatory Practices headquartered in India and entrusted with overall management of the community-led evaluation exercise). Key traits looked for in the facilitators included a friendly attitude and the ability to keep the process community-led. They were also expected to refrain from imposing their own opinions or judgments about the work of the organizations under evaluation.
For ensuring neutrality in the observations of the evaluation teams, cross-visits were planned in such a way that no community representative visited the projects delivered for her own community, but instead visited a different project located in some other state. In practical terms, it also meant that the accompanying facilitators had to be fluent in the languages spoken by members of the community team, as well those spoken in the areas visited by them. As many as nine different languages were spoken amongst the twenty six members of the group, even if one were to overlook the numerous dialectical differences amongst their diverse mother tongues. Needless to say, identification of competent facilitators for each team turned out to be no less challenging than the arduous demands of the evaluation itself.
Gearing up
Before embarking upon the evaluation exercise, all the community representatives and their accompanying facilitators assembled at a workshop held at Jodhpur, Rajasthan, to arrive at a common understanding of the task at hand. The key objective of the workshop facilitated by resource persons from Praxis was to arrive at a list of key indicators, which the community representatives could use in the subsequent evaluations, and to identify a set of appropriate methods that could help in reaching out to different sections of communities benefited by the programmes. Given that all the community representatives participating in the evaluation were already familiar with the design of the CBPPI programme, having seen its implementation in their respective places of origin, they were well placed to suggest appropriate indicators that could be used for the evaluations.
The process of identifying indicators had its own hilarious highs. To start discussions on the subject, the concept of evaluation was introduced with a simple analogy – the act of buying potatoes. There wasn’t a single soul in the team of community representatives who had never been involved in purchase of potatoes, and they were asked about possible criteria that could be used to distinguish good potatoes from bad ones. Several interesting responses emerged. Someone said that a good potato is white in colour, while others thought that a good potato is tight in composition, doesn’t have wrinkles on its surface, doesn’t smell bad, or carry rotten cavities. The analogy was extended to the possibility of distinguishing a good programme from a bad one, at which people had a good laugh. This was followed by a group-exercise to generate criteria to distinguish a satisfactory programme from a not-so-satisfactory one. Needless to say that the exercise resulted in a long list of as many as 77 interesting indicators of a ‘good’ programme, which were subsequently classified by a smaller team of 4 community representatives into 7 thematic categories (using a colour-coding method aimed at identifying similar-sounding indicators listed on charts) with some help from the support facilitators. The categories were related to:
- Inclusion of the poorest
- Enhancement in the status of women
- Quality of the collectives (of poor) formed under the programme
- Quality of work done / services delivered under the programme
- Engagement of the project with pressing local problems and issues
- Quality of linkages formed with govt. agencies, institutions of local self-governance and other local institutions
- Sustainability (prospects of continuation of the processes after withdrawal by the implementing organization)
It would be worthwhile in this context to mention a few indicators that were unusual in their nature and scope. For instance, some community representatives wanted to examine whether the programme sought to directly address social problems like child marriages, dowry and out-of-school children. Some of them wanted to examine the prevalence of ‘casteism’ and discrimination within project processes as an indicator of success, while some others wanted to explore the role of the programme in encouraging women to air their views in village level meetings. One of the groups emphasized on the need to assess the efforts of the programme in seeking out the poorest of poor families for delivering benefits and chose to examine the criteria of selecting beneficiaries. To gather necessary insights in a systematic fashion, a range of PRA methods (ref. Box 1) were suggested by the facilitators from Praxis, which were improvised by the community representatives during preparations at the workshop, and ingenuously used while carrying out the evaluations subsequently.
Following the workshop, the community representatives broke into smaller groups and visited a few villages of the nearest organization implementing the CBPPI programme, to pilot test their approach to the evaluation exercise. This helped in sharpening their grasp of some of the methods earlier discussed during the workshop. When Zubeida Bibi, one of the active members of the team, returned after conducting a well-being analysis in one of the villages covered by the programme, she burst out with anger on failing to locate a single household from the poorest category amongst members of the local thrift group.
Box 1: Participatory Methods used by the community representatives in carrying out the evaluations(1) Facilitating ‘Social maps’ to identify the poorest households (based on people’s criteria) and to assess whether the physical assets generated by the project for the communities are located appropriately (i.e. ensuring convenience of the poorest in accessing the same). (2) Conducting ‘Well being assessments’ to see whether the poorest households have been included in the SHGs or not, and whether they have received project benefits or not. (3) Conducting ‘Cobweb analyses’ to assess (a) the quality of groups / collectives formed under the project (b) the degree of empowerment of women (c) the quality of structures / services available through the project (4) Open-ended ‘Before and Now Analyses’ on multiple parameters to assess changes brought about by the project on multiple parameters (5) Case-studies to understand the project’s impact on the lives of individual beneficiaries, particularly with respect to its ability to take people out of poverty (6) Identifying key problems in the community, using the ‘Card Sorting technique’ (7) ‘Gender roles and decisions matrix’ to assess the project’s impact on the lives of individual beneficiaries (8) Project performance rating on key parameters, e.g. inclusion of the poorest, enhancement in the status of women etc. |
Hitting the field
Over the next one-month, the 26 community representatives collectively traveled over 75,000 km, visiting different organizations implementing the programme located in different corners of India. For most of them, it was a mixed experience, replete with rare opportunities of traveling to faraway places, earning accolades for their capabilities and also occasional bouts of ridicule and contempt. Staying away from families for one long month had its own challenges for many of them. There were times when some of them had to grapple with bad news received from home, during which they got tremendous support from their teammates. Two of the community representatives lost their relatives while in transit. Another two lost some money in transit, but managed to tide over the crises with the support of their teammates. Fortunately, some of the organizations were sensitive to the conditions of the community representatives, and one of them even arranged warm clothes for them to face the extreme cold weather of the hills.
Making a statement
The highlight of the community-led evaluations was the analytical quality of the observations of the groups. Being beneficiaries of the CBPPI programme in their respective places of residence, all the team members had a ready frame of reference to compare their observations with. They looked at each intervention from a perspective that could easily detect the shortcomings and understand the operational dynamics. For instance, while extolling the good work done by one of the agencies in harvesting rain water, the group did not stop short of pointing out how in several villages the marginal land holdings of the poorest had found no favour from such interventions, due to being located at a distance. Similarly, the adverse consequences of conditioned loaning in villages of another organization, resulting in land alienation for poor families - were highlighted as the backlashes of inappropriate interventions. In many places, they identified several livelihood-promotion activities under the programme that were hardly viable due to absence of markets. In one place, the group was shocked to see clear wage differentiation between men and women despite strong emphasis of the programme on reducing gender disparities.
At the same time, the groups did not mince words in lauding the good initiatives of several implementing organizations. Their observations were replete with acclamations that spoke of wonderful efforts in regenerating natural resources, successful campaigns forcing government institutions to act, popular innovations in imparting elementary education, effective operations to free poor people from the misery of bonded labour, numerous livelihood-support interventions helping the poor during lean seasons, efficient collectives managing huge amounts of self-pooled resources, and ingenuous local initiatives seeking to deal with social evils.
The day the groups returned to Delhi after completing their visits to various programme areas, the atmosphere was charged with the victorious spirit of a troop conquering a fort. All their fears, apprehensions and inhibitions had been blown away by the experiences of the evaluation, and they had returned with a trove of stories from their unique adventures across snowy hills, deserts, riverbeds and seashores to share with each other. They spent three days together intensively preparing for the presentation that they had been eagerly waiting for. The support facilitators too worked very hard in helping them articulate their key observations.
The grand finale
The presentations did not betray the diligent efforts of the community representatives. Key findings and observations from the evaluation were organized around the seven thematic categories earlier identified by the groups to categorize their indicators. While the group members took turns in presenting analytical diagrams and tables drawn on chart papers, the support facilitators translated their observations to the audience. They had prepared too well to afford any mistakes on the occasion, and they surprised people with their pointed observations. The Joint Secretary from the Ministry of Rural Development, who was one of the main guests attending the presentations, was impressed by the specificity of some of their findings, and promptly sought clarifications from the representatives of the concerned organizations about the same. The Resident Representative of UNDP complimented the community representatives for their outstanding work.
For the audience gathered at the UNDP office, the experience showed that it is not only feasible and cost-effective to entrust communities with evaluations of development interventions on a large scale, but also an empowering strategy by itself. As a matter of fact, the whole evaluation across 13 programme areas located across 11 states of India was undertaken for less than Rs. 10 lakh, or equivalent of 1800 Euros. Each of the community representatives was paid Rs. 5000 as an honorarium for their month-long effort, and their actual costs of travel, food and accommodation were either reimbursed or borne by UNDP.
It was time now for the community representatives to say good-bye to each other. Long after all the presentations had been concluded, groups of women could be seen cuddling each other with a tearful twinkle in their eyes. They were aware that they had together conquered many fears that once seemed to thwart any hope of success on their mission. They also knew that they would perhaps not get to see each other again - a sense of loss that would perhaps haunt them for a long time to come! Yet, their stride away from the UNDP office indicated a newfound confidence that would propel them to take on various odds in their daily lives. Their journey of life had found a new beginning!
***
[1] The CBPPI programme was aimed at reducing poverty through collective action of poor people themselves. Implemented by non-governmental organizations and government agencies in backward districts of 11 states of the India between 1997 and 2004, the programme had a total outlay of US $ 10.7 million.
[i] I am grateful to Ms. Kirti Mishra, Ms. Madhura Pandit, Ms. Saswati Bhattacharya and Mr. Debabrata Bhuniya – the facilitators accompanying the four teams – for their valuable inputs and observations.
No comments:
Post a Comment