What follows are a series of anecdotal case-stories on evaluation methodology and related issues, presented in a lighter vein and with a twinkle. The case-stories are written as fiction and names and locations are always changed to ensure anonymity. Enjoy yourself.

If you have one or two stories to share please send these by e-mail through "contact us".

And when it rains it pours

Years ago I was in an Asian country engaged in a feasibility study. For this purpose I met several stake-holders and one of them was the manager of a UN agency engaged in development. When by chance I told this manager that I was to visit a rather remote area, he asked me if I could pick-up a scientist who was doing a study of different crop-production methods there. I agreed and a few days later, on my way back in a small pick-up truck, we found the scientist at the agreed place, with some 25 plastic bags.

The little plastic bags were carefully loaded in the back of the pick-up truck. Each of them contained the harvest of a small plot of land, the plot concerned being indicated on a label that was attached to each bag. With some effort our scientist was also accommodated in the cabin of the truck, in addition to the driver and myself. As we came nearer to our destination, the sky became darker and darker and not long before we reached the hotel where we were going to drop our scientist, the rain poured from the sky. When we arrived at the hotel, it was almost dry again but then it appeared that a catastrophe had occurred: almost all labels that had been attached to the plastic bags had fallen from the bags. They had more or less dissolved in the water. The scientist at first was terribly upset, then murmured that he would be able to reconstruct the location for each bag.

I left him and his bags and whatever could be collected of the labels in the lobby of that hotel. He must have been a man of unusual talents if he were able to repair, in a truthful way, the damage that had been caused by that rain.

Reflection: Don?t think that this story is valid in very exceptional circumstances only. Sometimes it can rain inside an office. For example, always mark each page of a questionnaire in such a way that you still know the respondent! Often questionnaires are loose sheets kept together with a paperclip or staple. For various reasons the paperclip or staple may stop functioning and this may create exactly the same problem as the one described above!

Progress and the ?process-approach?

At the beginning of the assignment for evaluating a project in Central America I was told that very little information was available and that some extra time would be allowed for preparation at the project site. Immediately after arrival in the project office I noticed a defensive attitude from the project management and within an hour it became clear that they were convinced that the evaluation mission had come with the preconceived plan to formulate conclusions that would justify termination of the project. The management had developed strong feelings against this especially as they found that after a long and difficult start the project now finally was on a most promising track with important progress made in the past 1 or 2 years.

I decided fairly soon that it would not be very fruitful to continue discussions on the issue of a hidden agenda. I had not received such an instruction (and if so, I would have refused to do the evaluation) but there was also some confused thinking as the project was nearing its termination date anyway (after two extensions). Therefore, I proposed a common starting ground and I invited the project management to come up with suggestions how such a common understanding could be reached. After some discussion it was decided that we would start by identifying action plans as these had been formulated in the past four years and then to assess the implementation rate. In other words, we would concentrate on output first and then other issues such as those related to outcome could be studied. Both project managers were confident that such output study would impress the evaluation mission as they estimated that they had realised some 90% of planned outputs over the past two years.

When work was divided over the members of the evaluation mission it soon became apparent that the outputs as planned in the annual workplans could not be taken as point of departure as these plans had been adjusted frequently. The project management explained that this was part and parcel of the ?process approach? followed within this project in which plans and actions are determined for an important part by the interest and motivation of various stakeholders, notably the target groups. To be sure, especially in projects that aim at participation, self-help and empowerment of target-groups, such a ?process approach? may be necessary or even a condition sine qua non.

However, when it comes to assessing the implementation rate of ?planned outputs? you need a plan, a benchmark, a target, even if you measure a process. Some tedious discussions followed also including other members of the project staff. It appeared that many of them had their own interpretation of ?process approach?. For some of them it meant that one has to do whatever seems fit and possible in a certain situation, while others found that the whole approach was just giving a name to hide a lack of managerial vision and capacity. This did not help in making things easier. Finally, it was concluded that the process-planning could best be retraced in the quarterly progress reports where new or adapted output plans were mentioned. However, analysis of these plans and its subsequent outputs appeared rather cumbersome as same activities were sometimes indicated with different words, numbers did not refer to same units (e.g. sometimes to farmers, sometimes to farming families, sometimes to farming communities). These issues appeared difficult to solve as the management kept on explaining to us their grand ideas on the ?process approach? where our primary concern was to get an idea on how this approach could be traced in its outputs.

In such a situation there is considerable danger to get lost in details and idle discussions while other important elements of the project may not receive the attention they deserve or need to be given on basis of the Terms of Reference. Therefore I took a sample of the kind of outputs to be identified (those identified as core-activities by the management) for detailed study. In the end it appeared that the implementation rate of core-activities of the project was somewhere between 40 and 50% of the last revised plan as could be derived of available documents. This also implied that the outputs that had been realised in general had been much more expensive that originally foreseen.

Reflection: The evaluation concerned of course much more than the outputs of individual planned activities. In this little case-story this aspect was mentioned because in the perception of the management and some staff members, the project was doing extremely well in this respect. The fact that they were continuously adapting plans and planning units, also in their monitoring, had gone completely unnoticed. It was a complete surprise for them to discover the confusion in terminology and the fact that even in terms of last revised plans their implementation rate was less than 50% where initially they had been sure that it was at least 90%. However, it is not this figure that is important here. The main issue concerns the needless antagonism toward outside parties who were accused of not ?understanding? (or not willing to understand), the specific conditions of the project and its future as perceived by those managing it. For an evaluator it is best in such circumstances to do his utmost to really try to understand what is meant (best done by summarising in own words to check if the other side agrees), to look for a consensus on matters that are of relevance to all parties involved but at the same time to avoid getting involved in endless disputes or petty details or to forget your own role as an evaluator.

Having Santa Claus in your evaluation team

It was decided to evaluate a project on regional planning in an East African country. One member of the evaluation team would take care of those parts of the 5-year plans that dealt with primary education. Although the evaluation concerned the planning effort, a number of visits to schools and other relevant institutions and personalities had been included in the programme. Our project had little to do with this, but we were asked to provide some logistical assistance. For that purpose we made some space in our guesthouse and asked our counterpart Moses to assist this evaluation mission by making appointments and to accompagny the team or individual members as required. As it happened he accompagnied the evaluator dealing with primary education.

It soon became apparent to Moses that he was assisting an extremely important man. Already during day 3 when they visited a primary school, the evaluator judged that the learning materials in this school were below all acceptable levels and promised immediate funding for new books and other materials. A few days later, when discussing salary levels of teachers, the evaluator decided that all salaries of teachers in this province were to be increased through topping-up arrangements to be financed through his Government.

Moses started to dream of such an evaluation-mission visiting our project. Not much later considerable tension appeared to grow between the leader of the evaluation mission and the educational expert, as the team leader tried to stop the incessant stream of promises trailing behind the latter. And for months Moses was asked by teachers and others where all those learning materials and topped-up salaries were hiding.

Reflection: Sometimes members of the evaluation team are not selected by the team leader but selected individually by the party commissioning the evaluation. In such a case, the team leader should not only make sure that members of the team follow a common or at least acceptable approach, s/he should also make sure that team members behave correctly. It is not always possible to do so before the mission starts. In such a case, the team leader should not allow the team to be split up immediately after arrival into individual members each following his own path. Empty promises should never be made in any case. However, preferably the team leader of the evaluation should be responsible for the selection of the individual members of the team.

Perceptions of an "angry young-lady"

This short case-story refers to an extremely angry lady in Germany, but it could also have been in France or Holland. The evaluation concerned a co-financing programme of an international organisation and the lady had been project manager of a project that was supported by the co-financing programme. I was warned about her, first by an assistant who had tried to make an appointment (in vain), then by an official of the headquarters of the co-funding organisation (HQ) who confirmed her bad temper when the appointment problem was discussed, more or less by accident. Finally I experienced it myself when I phoned her. However, she agreed to an appointment.

When we met, a few weeks in her small office in Southern Germany, she started a head on attack. As if I were in charge of the HQ myself. But then, after she calmed down, she had an incredible story to tell. To be sure, it was not representative for the functioning of the HQ and technical staff that had been hired to supervise the programme. Yet, it was an accumulation of things that went wrong. Not in her project but in HQ.

It is not important here to explain in detail what went wrong. It had to do with policy changes, with handing-over procedures, with letters not answered and problems created because an official left HQ and was replaced by a newcomer. When this newcomer by accident re-created problems that had just been remedied within the project, our lady had reacted rather angry and in response had been labeled ?difficult? by the new official in charge without the basic problem being addressed.

It was, after all, not only a pleasure to get acquainted with the project that had been managed by our ?difficult? lady, it was also enlightening to see how bureaucratic processes and the staff working within them, can yield perceptions and decisions exactly contrary with what was originally meant.

Reflection: People who are highly committed to their work, often are also emotionally involved. Therefore, they sometimes react in strong words and have little empathy with bureaucrats and bureaucratic rules and procedures. This is both their strength and their weakness. Sometimes this weakness is accentuated because of the ?image? of the person in question in the bureaucracy concerned (where also the ToR for the evaluation may be written). However, in as far as such people are reacting as they did because of their strong commitment exactly to the objectives for which the bureaucracy was made, such image-building should be treated with due care by evaluators. Not only when sampling but also when being confronted with loads of frustration.

The emperor?s clothes

The team leader was in a hurry. That was clear from the start. After this evaluation he was to do a big assignment for the same department that was commissioning this evaluation in an Arab country. After we had prepared ourselves as well as this was possible given the time constraints that had been caused by some delays in getting the mission started, we had a first meeting only days after we had arrived in the country: the first days all were spent with official meetings with our team leader being absent in the evenings for important discussions with ?stakeholders?.

This first team meeting took place in the lobby of the small hotel in some remote small town where we had arrived that same day. The team leader produced an outline of the final report and divided tasks between the members of the team on basis of the text that was to be produced for each chapter or section. The team was to split up in small units as from the following day. Places, organisations and persons to meet had been decided by the team leader already after consultation with the ?stakeholders? and all logistics also had been arranged. Some members of the team protested because they had other plans while furthermore all flexibility had been taken away. The team leader responded by pointing out that they should have voiced such objections earlier. This was a somewhat remarkable answer as it had been virtually impossible to do so given the poor preparation stage and the absence of opportunities for discussion after the team had arrived in the capital of this beautiful country. As no one really was eager to quarrel extensively with the team leader and some members of the mission felt that he deserved some credit rather than early criticism, the evaluators complied.

When the team met again one week later, experiences had been quite diverse. While some of the evaluators had seen positive and interesting projects, others had met examples of poor performance, ill conceived strategies, lack of effectiveness of supporting services end so on. This concerned participatory mechanisms, gender issues and small scale fisheries. These and other observations were exchanged amongst the team members after they arrived in the government guesthouse were the team would meet next day. That next day the team leader arrived together with the desk officer of the international organisation that had commissioned this evaluation. Once the meeting started, problems arose immediately on the agenda as no time had been reserved for a formal exchange of observations, experiences and impressions. However, once some time was given for such an exchange, problems jumped into crisis as the team leader rejected some of the observations because they did not fit into his conclusions. Apparently, he had been writing large parts of the final report in the week when the rest of the team had started the actual evaluation.

When the atmosphere deteriorated almost by the minute, the desk officer asked the word. He asked the mission to have some understanding for the team leader as the final report would be required only a few days after return to the Head Quarters because a major new loan to this country had to be signed. Furthermore he explained that participation, gender and artisanal fisheries had been a cornerstone of this programme and would be so also for the programme to be financed under the new loan. Therefore, it would be very unwise to include all sorts of critical comments on these vital issues in the evaluation. In the confusion that followed these words, a local official took the word and suggested that some members of the team had been somewhat misguided or were perhaps jumping to conclusions, confusing accidental observations with general reality.

In the afternoon and well into the night, intensive discussions followed between members of the evaluation team, the team leader and the desk officer. The next day the desk officer informed the team that each team member of course was free to report his/her own findings but that the team leader was free to make use of these findings as he thought appropriate. Furthermore, each team member was urgently invited to follow the instructions of the team leader including the programme for the second and last part of the field visit as he had prepared it.

The same three members of the (7 member) mission that first had objected, did so again. One of them decided the same day to stop his participation in the mission as he had the idea that this whole evaluation was nothing but an expensive way of window dressing. The rest completed the mission and were allowed to write down their reports as they thought fit. In the final report as written by the team leader, very little critical remarks on participation, gender or artisanal fisheries could be found. In fact, the team leader wrote the final report in the same week in which the remaining members of the evaluation team wrote their contributions.

A few years later, the member of the evaluation team that had walked out had the opportunity to discuss these events with the then director of the desk-officer who had commissioned the evaluation. The desk-officer was not highly competent and always running behind schedule. Replacing him had been considered sometimes but that would have created political and other problems. In the case of the evaluation under discussion, the director had been given the impression that the writing of the ToR was almost finished when in fact most work still had to start. Yet, the new phase of this multi-multi-million dollar programme had been made dependent on the outcome of an ?independent? evaluation. In the discussions with the government of the recipient country it then had been assumed that the evaluation would be positive so that the preparations for the new programme could go on as scheduled. However, although it had been stipulated formally that the final decision on the programme would be dependent on the outcome of the evaluation, there was in fact no way back without creating tremendous problems, not at least for the desk-officer himself. And, it may be added, for his superiors??.

Reflection: Bureaucratic organisations have an internal logic of their own and the necessity to spend funds according to funding plans is part of that logic. At the same time, key staff have often their own career plans, strengths and weaknesses and as long as these are in line with organisation policy, few directors are willing to thwart existing routines that are conducive to fund ?exhaustion?. Although Terms of Reference often are written in a manner that suggests that an ?objective? evaluation is required, desk-officers may try to influence the outcome by (a) the choice of the team leader, and (b) their power not to make use of the services of evaluators who were ?difficult? during previous evaluations.

A professional evaluator will do his best to accommodate justified wishes and to understand the points of view of those involved. However, in the end, it are ethical considerations of himself and of his company that should be decisive in the way the impartiality of the evaluation is guarded. Usually it is possible to convince one or more important parties that, in the end, it is in the interest of all to keep the evaluation impartial and professional. In the exceptional case that this is not possible, difficult choices have to be made and should be made.