Especially for larger or more complex evaluations a team of evaluators may be required. The composition of the team is often specified in the Terms of Reference for the evaluation. Certain types of sector-expertise related to the content-matter of the evaluation may be required (e.g. an agronomist, a hydro-geologist, an educational planner, an agricultural extension expert) as well as cross-sectoral expertise (e.g. on organisational or institutional development, or expertise on gender or poverty-alleviation etc, as well as experience in defined geographical areas , sometimes proficiency in one or more local languages or proven experience with certain types of research techniques or methods(surveys, auto-evaluation, or fieldwork). A minimum experience, trackrecord or seniority may also be stipulated. Furthermore, it is sometimes requested that each proposed evaluator signs a declaration stating that, if selected to do the evaluation, s/he will indeed be available.

In this article I suggest that efforts to specify in detail the types of expertise required in the team contribute comparativelty less to an effective evaluation, and it is better to specify and emphasise the role of the (team) leader of the evaluation team to ensure the fielding of teams that can really work effectively. With regard to the availability-statement it is often the case that some of the persons (proposed) are finally NOT able to make it. In the past consultancy agencies put forward their best team, to win the proposal, but deployed an available B-team. Donor agencies and principals sometimes make the actual deployment of the proposed team-member(s) a make or break condition in awarding the evaluation contract, but if this is the case I suggest that more priority is given to the deployment of the (proposed) teamleader and that some conditionalities need to apply to the Consultant as well as to the principal awarding the evaluation assignment, to make the evaluation a genuine and meaningful exercise.

Tenders and evaluations

Most large evaluations are tendered. That means that ToR are written and that evaluators are invited to submit a proposal. Often a prequalification is made, resulting in a shortlist. The companies (or sometimes the persons) included in this shortlist are then invited to submit their proposal(s). Ideally such a shortlist contains 3 names.

To keep it simple, if all three companies would be equally good, each of them would have a chance of 33% of winning this tender.: if in one year 3 evaluations would be tendered and each of the three companies would obtain one of these three evaluations , then the income from that evaluation should compensate for the costs of making the two other proposals that were made in vain.However, shortlists are often longer. Shortlists of 5 companies are not at all exceptional and there are examples of shortlists of 7 or more names (the longest I experienced contained 11 names). In case of a shortlist of 7 or more names, the chances for winning decrease strongly. In other words, if a company would invest in writing a proposal for such tender, then each winning proposal should contain provisions to regain the costs of 6 lost proposals.There are several strategies to cope with such situations. Most of these affect the quality of the proposal in a negative way.

There are of course many ToR that are of poor quality. With a bad start often the proposals tend to be sub-standard. When reading such vague or open-ended ToR it is clear that the principal (the organisation formally responsible for the evaluation) has a lack of expertise and/or does not consider the evaluation of real importance. This is almost always the case when no maximum budget is given for the evaluation. In such a case, evaluating organisations that have sufficient work will probably not submit a proposal. Organisations that are in need of work and income will react by writing a proposal that doesn?t take too much time to write. The result; the principal has to select from a set of low-cost, minimum-preparation-input and high-expectations and promises proposals.

The same is likely to happen when the shortlist contains 5 or more names. However, here there may be an exceptionwhen the ToR are (a) really good, and (b) indicate thresholds for assessing technically superior proposals. In such a case it becomes possible to make a much more realistic estimate of chances to win the tendering, and an evaluation consultant will find it worthwhile to invest in writing a good proposal..

The reason for mentioning these basic economics behind a tender procedure is that consultancy companies have to recoup their earlier investments and the bigger the shortlists and the more often a company has to submit a tender, the less likely a proposal will be really tailored to the job, and the more likely the quality of the proposal will suffer. It would be in the interest of the awarding agency to have limited shortlists, and keep a fair competition by distributing tenders in a transparent and balanced manner over all companies eligible, to these limited shortlists and thereby keeping all teammembers available and on hold for a definite and limited period of time.

Some international organisations require that all members of the proposed evaluation team sign a declaration confirming that they will be available if their organisation is selected for this assignment. This ? at face value ? common sense requirement should be studied more in-depth as it often fails to understand the high correlation between an increasing density of rules and a decreasing effectiveness of many bureaucracies.

Often it takes many months before one t even knows if the assignment was obtained or not. The reason for this is in most cases "bureaucracy" . Apart from "unforeseen events" that can trigger a delay, e.g. a military take-over in the country where the evlaution is to take place, or a natural disaster strikes the area where the field visist are to take place, there are also ?normal? delay, e.g. tendering organisations often do not decide ( refrain from immediately announcing) the winner and inform those that didn?t win. Instead they first enter into contract negotiations with the "first choice" winner. These negotiations may take weeks and sometimes months. Not because there is much to negotiate but due to all kinds of formal reasons like bank guarantees that need to be checked.. Therefore it may easily take a few months longer before the information on that the proposal becomes available 6 months is no exception and I know of a proposal for a very big project that is still pending, more than 2 years after submissions were made.

In case these teams were composed of 4 persons each, this would imply anything between 4 times 4 times X months, the X referring to the total time of the assignment minus the chance of finding other work after being informed that the assignment was not obtained.

It may be obvious from the above time-frames and the uncertainties due multiple contestants, that it is economically impossible for a company to keep most of its staff unproductive and on stand-by, and to guarantee that the proposed staff will be deployed, unless the principal or donor sets a time-limit to the award period. This conditionality would be a major improvement to the quality of the proposals to be received. It would place the responsibilities on both parties. An adjusted version of this modality would be to allow the bidding companies to change their teams (within reason) if the award period is extended beyond the agreed period. Principals and donor agencies would be certainly receive more truthful and realistic proposals, which the agencies can review in a more realistic and effective manner.

The composition of the evaluation tream

A successful football coach does depend on, and does communicate withall his or her players. But the communication ? for many reasons - is often channelled through the team captain. a Acoach who realizes that he/she can not play in the field, gladly accepts the reliance on the captain who can react more quickly to changing field situations, and the coach will be careful, and invest time when selecting the teamleader, the captain.

One skill/aptitude to look for in theteamleader is his/her ability to respond to the (changing) situation in the field without changing too much the agreed ToR Being the "chosen one? places special responsibilities on the teamleader too, in addition to sector expertise he/she is to be a peoples-person/manager. This requirement is especially relevant when the teammembers are not from one and the same company. Who has not experieneced serious infighting, heated arguments and never ending discussions, especially when the ToR leave much room for interpretation. Often evaluators are experienced experts with a strong opinion of their own. Thus the role of the teamleader is not only technical, rather in complex and bigger evaluations it is his/her team-management skills that will make the difference.

Complementarity of skills in an effective team are important. This complementarity is to be present in terms of the technical skills, and this is relatively easy to achieve, but complementarity should also be present in terms of the social skills of your evaluation team. A group that has to perform and peak in a relatively short period of 3 to 6 weeks needs to be configured in a balanced and mature way. The teamleader should be a person who can quickly gauge the group dynamics and how well the group is working and moving toward its targets

The bypass option, an evaluation team-member communicating directly with the principal, should be built-in, to serve as a check on the teamleader.The principals should make him/herself available for direct observations but keep this to a minimum not to undermine the effectiveness and leadership status of your teamleader.

Here we make the final point that it pays more to concentrate on the teamleader than on the sector specialists and other theam members. There are many ways to overcome the absence of a certain skilled expert in your team; one can find a local resource person, or it may turn out that one of the other teammembers is able and willing to double.

What should principals do?

There are of course the evergreens: write good ToR, have a good briefing, provide them with as much as possible documentation prior to the mission, ?. To this we already added, the need to limit the shortlist to 3 companies to encourgage that only the better proposals will be submitted, an added advantage is that the principal need less time to decide. Agreement to have the experts, who assess the proposals, available at the correct time and to complete with them the professional and operational selection criteria, while working in a time-efficient manner, by setting yourself deadlines, are some of the groundrules to improve the handling by the principal.

However, I would like to focus on a few specific aspects that the princip[al should consider:

In this way, it becomes interesting for specialised evaluation-organizations to weigh their chances for writing a successful proposal and if they become convinced that such chances are real, to really invest in writing a good proposal including a tailor made evaluation design and a well qualified team. To be sure, if invited, also organisations that produce routine-proposals will submit a bid. But such bids will most probably lack quality.

What should evaluating organisations do?

As stated before, I made a tentative distinction between organisations who submit proposals on almost any evaluation being tendered and organisations who prefer to write a proposal only if they feel that they are particularly qualified to do such an evaluation and if the ToR are up to minimum standards.

The first category is not necessarily doing a poor job when evaluating and the second type not necessarily a good one. Because the first category is mainly composed out of large organisations, they probably will produce an evaluation that can meet minimum standards.

Here I will elaborate somewhat on the second category, the organisations (or networks, or free lance consultants) that avail over a unique combination of evaluation and subject-matter expertise that makes them particularly qualified to submit a proposal for certain evaluations.

The first thing to do when noticing an interesting evaluation to be tendered is to decide whether or not to go for writing a proposal. If the ToR are of poor quality, if no indication is given of the maximum budget or number of expert-months, if there are no thresholds for selecting the technical proposals on basis of quality, you probably will decide not to participate.

If you decide to write a proposal, immediately start identifying a team leader, who in turn should play a key-role in writing the proposal and finding other proposed team members. In doing so, stress evaluation methodology first (the evaluation design) and only then come to phasing and individual team members. Remind yourself that resource persons may be given an important role in addition to team members and select the latter especially on their capacity to work in a team, to report in time and in according to given formats and not at least on basis of minimum evaluative skills.

In case the selection process of the tender takes much more time than indicated, weigh your chances. As long as you estimate that you have a serious chance, don?t let your team leader go. However, don?t keep the team available longer than one month after submission of the proposal. At least not at your own cost. It cannot reasonably be expected from you to keep expensive people available when no-one is going to pay for their time.

Honni soit qui mal y pense.