Development interventions of most types are usually formulated in such a way that achievement of the targets, outputs and outcomes is supposed to have a positive influence on higher-level goals.

If, for example, a credit scheme is launched for small and medium enterprises, it is assumed that such a scheme - if successful - in turn will stimulate further developments such as increased employment, a stronger export, etc. In the Logical Framework terminology: there are inputs used for activities, these activities yield outputs, these outputs yield results or outcomes and the results together constitute the achievement of the project purpose. The project purpose, however, is not the end of the chain of cause and effect: the project purpose is supposed to contribute to one or more ?higher objectives or goals?, for example the reduction of poverty.

The project management is responsible ? and to be held accountable - for achieving the project purpose. In other words, the project available resources and the organisational and managerial capacity of the project (or programme) should be timely deployed and sufficiently strong to achieve the project purpose, if no exceptional impeding events occur. Yet, in the end there is always an immediate desire ? and as we shall explain later, a longer term need - to know how the project purpose contributed, directly or otherwise to the ?higher objective(s)?. This is the domain of ?impact assessment? and it tends to reflect on lasting changes. Impact assessment was recently defined as "the systematic analysis of the lasting or significant changes - positive or negative, intended or not - in people?s lives, brought about by a given action or series of actions." (Chris Roche, Impact assessment for development agencies, page 21, Oxfam/Novib, 1999)

The traditional way of making an impact assessment was rather straightforward and consisted basically of the following series of actions:

1. Select variables (impact indicators) that reflect impact, and that can be used to measure such impact if it occurs.

2. Investigate which of these variables are already being measured by other agencies (e.g. the National Statistics Office) and which are not measured or readily available.

3. Select population/control groups comparable to the group served by the project, in all aspects except the fact that they will n?t be affected by the project or similar interventions (the control group/population).

4. Organise a survey for the (impact) variables not being measured by other organisations such as the National Statics Office, in order to measure their readings before the project would start (before & after). Do this survey both in the project population and in the control population (with and without).

5. If possible, wait till the project is implemented or at least a number of years after it started, to measure the change in values in impact indicators.

6. Repeat the surveys as described under 4.

7. Identify impact by comparing the impact indicators of the project population with those of the control group and correct for differences due to factors not related to the project intervention (e.g. seasonal effects).

This approach is also described on the World Bank website dedicated to poverty eradication as the way to measure impact at that level. (https://www.worldbank.org/poverty/impact/overview/intro.htm#whatis)

Attribution

Although the idea is easy to grasp, in practice it proves to be difficult to implement impact assessment in this way. Finding reliable statistics, organising base-line surveys yielding valid data, finding control groups, etc., there is a long list of problems. A more generic, systemic weakness is that this approach suggests a rather mechanical way in which social change (impact) takes place and how it can be measured. Furthermore, this type of impact study tends to underestimate the complexity, interplay and dynamics of the numerous factors large and small that affect social change and that have a large influence on the ?impact? of the project or programme.

Let us compare life with a pond, and the project with a stone thrown in that pond and impact with the ripples - caused by the stone that reach the other side of the pond. Assume our project would be the only actor, and the pond is tranquil, calm and almost without ripples, with no hidden currents or forces. In that case, once the project is implemented (the stone thrown in the water), we may safely assume that all ripples are caused by our stone, that is by our project. In that case it is easy to see how our ripples reach the other side of the pond, although the ripples may be very small by that time, and what the lasting effect is of our ripples on the other side of the pond. However, such a situation is completely unrealistic. In reality, all kinds of actors are throwing stones in the pond all the time, before during and after we implemented our project. In addition there are many environmental factors and forces at work. More often than not we can follow ?our? ripples only for a very short time before they disappear in the waves both large and small caused by the other stones, the wind and what other causes and forces there may be.

For this reason some funding agencies have more or less stopped making impact assessments, and they only measure "project purpose achievement". This position is difficult to defend on the longer term. After all, if it cannot be continuously assumed that our development efforts did make that lasting and desired difference. Someone from outside, or even from within our own organisation, at one time will ask WHY funding is being continued. While ?normal? evaluations would provide lessons for practical management with regard to improved project purpose achievement, also lessons are to be learned at policy and strategic levels. It is exactly for these lessons that insight is required how effective actors and factors were in stimulating and/or hampering impact levels.

The requirement, to be aware of one?s strategic and impact consequences, is even more important in recent years when developmental organisations tend to justify the support they provide increasingly in terms of ?strategic relevance?, often suggesting that by a relative modest support, lasting results can be achieved, e.g. at macro-levels, or in terms of institutional strengthening, or improved governance, or in poverty alleviation. An important realization, when expanding evaluation to impact levels, is the need to explicitly include stakeholders and outsiders in the evaluation exercise, because they will point out impact-relevant relationships and connections we may not be aware of.

Theory-based

Back to our ?theory based? evaluation approach. By comparing the data from the baseline surveys at the beginning and a number of years later, we may establish what has changed (and what not). By additionally also doing the same measurement for the control unit (group, area, cohorts) we may also get hints on what possibly could be attributed to ?our? programme or project. However, this will rarely be more than a hint because, as explained, in day-to-day social reality there may be so many actors and factors that are influencing the thinking and motivations of people, their leaders, the functioning of their organisations, etc. that it is very hard to identify what parts of behaviour and outcomes exactly were caused by our intervention(s).

It is in this context that a ?theory-based? approach may be useful. ?Theory? in this context does not refer to ?evaluation-theory? but to the complex of assumptions that people have in their mind on ?how things work?, in the world, in their society, in their organisation, in their families. These assumptions are not necessarily fully correct but that?s not the point. They channel the thinking and behaviour of those who have them and therefore are essential for understanding why things went as they did.

One may argue that such an approach only shifts the problem: if any individual has literally thousands of ?theories? in his head on how things ?work?, how then to select the theory or theories that are most relevant and how further to deal with them? Obviously, this objection is true. However, there is more to say on it.

Let?s approach the problem from the other side. At the time that the intervention was planned, also one or more ?theories? were formulated (or assumed). These theories were at the base of the ?intervention-logic?. This means that these theories formed a (causal) link between the planned inputs and the ?impact? in the end foreseen.

In practice, only a tiny part of these ?theories? is formulated in the intervention-plan. Many other parts are assumed and still more parts again have gone unnoticed, that is that assumptions have been made without knowing that this was the case.

Assumptions

The challenge of a ?theory-based? approach is not to make explicit all the assumptions as maintained by the parties involved. This is impossible. Rather, such approach should be based upon a thorough insight of the parties involved, their relative importance and the assumptions that are relevant in this context. These are the assumptions that really count.

In order to get to know these assumptions, one needs familiarity with the stakeholders and, especially, the context in which they function or try to do so. Probably, obtaining such familiarity and overview is the real condition for successfully applying the theory-based evaluation approach.

(For some more thoughts on this issue also see https://www.worldbank.org/oed/ipdet/Modules/M_02-na.pdf).