ABSTRACT
When to Evaluate In planning a project, it is important to make an early decision as to whether or not the project deserves to be evaluated. Just as there are certain projects that should be evaluated, so there are others that should not.Evaluation is properly regarded as an aid to decision-making, and is normally performed only in circumstances where some future decisions are envisaged. Being a difficult and often costly task, evaluation should not be undertaken for purely frivolous reasons, such as justification for decisions already made.There are a number of factors (or ‘principles’) which should be taken into account in making the choice whether or not to evaluate a project. A first consideration is the nature and importance of the future decisions. If the project is a pilot study, it is likely that the outcome will influence many
similar projects in the future, and so would be a strong candidate for evaluation. On the other hand, a ‘one shot’ project would be less likely to deserve evaluation.A particular case where evaluation would have a persuasive rationale occurs when alternatives are being considered: different road geometries, different public information plans, different land use arrangements. Many interventions can be implemented in a variety of strengths and configurations, and in many cases, it is worthwhile running carefully designed (and evaluated) pilot projects.A second factor to consider is the cost of evaluation. The U.S. National Highway Traffic Safety Administration manual offers a rule of thumb that ‘evaluation costs should not exceed 10 per cent to 25 per cent of the project cost unless the project is very special’. In preparing the budget for the project, the evaluation costs should be considered as carefully as the costs of the project itself.If adequate money and manpower cannot be allocated to evaluation, it may be wiser to let the project proceed without evaluation. After all, a well-designed project may have a persuasive rationale in and of itself. It has been observed that the advice given school children to ‘look in both directions before crossing the road’ has not been carefully evaluated, and probably does not need to be. It would be a mistake to assume that a project can be successfully evaluated on a shoestring budget. There are some cases, such as pilot projects which, if successful, will be widely copied, where a first class evaluation can be expected to cost as much as or more than the implementation.A third consideration must be the chance of a reliable conclusion. If an evaluation yields data which is not statistically significant, or which is unreliable in some other way, or which cannot be reliably tied to the intervention in question, the exercise may be a waste of money.This risk is greatest in dealing with imponderable effects, such as those emanating from a public relations campaign. Social action projects often have vague or ill-defined objectives, nearly impossible to evaluate objectively. It is less so in interventions, which are specific, such as the reconstruction of a particular intersection. But even in these cases, it is necessary to assess carefully the chance of a conclusive outcome. Will the necessary data be available? At what cost? Will the data be reliable? What will be the cost of coding and storing the data? What are the chances that the sample size will be so small as to give an ambiguous answer? How long will it take to accumulate an adequate sample? What other factors
might be influencing the evaluation during the experimental period? What specific problems might be encountered in forming a conclusion? What level of significance will be needed for the conclusion to be useful? These three issues: the need to know, the cost of knowing, and the chances of success, are combined to produce a decision regarding the desirability of proceeding with an evaluation.Let me give you an example of difficulties once experienced in conducting an evaluation on behalf of the U. S. Government. The topic was a reduced speed limit, to 55 m.p.h. that followed the energy panic of 1973-1974. Although this measure was designed to save fuel, there was an unexpected coincidence with a precipitous drop in the vehicle-mile fatality rate. The Nixon Administration, constrained by constitutional issues of state/federal authority and especially by the felt need to adopt measures that would be popular, relied mostly on public relations and in particular on a highly publicised 55 mph speed limit. The legacy of that approach was nearly a quarter century of wrangling involving issues having nothing to do with the energy panic, which vanished almost as soon as it came. From the point of view of evaluation, there were many complications. First, there were fifty jurisdictions, each having very slightly different laws. Second, the enforcement of these laws was partly in the hands of local government, with thousands of separate police agencies. Third, the consequence of non-compliance varied greatly, from trivial fines in some Western States, to really draconian penalties, mostly in the East. Fourth, the only really good measure of safety effectiveness was fatalities, even though, if we believe economists, fatalities are only a part of the cost of accidents. Fifth, it was not clear whether we should use fatalities per distance travelled, or fatalities per number of cars, or fatalities per capita, or any one of many other measures.The situation was complicated at this stage by the enormous degree of popular approval of 55, even - or perhaps especially - by those who seldom obeyed it. The country was polarised into pro-and contra-advocacy that often appeared to foreclose rational discussion. A technical issue had been converted by propaganda into a moral issue, and thus into a political issue. Public opinion favoured ‘saving lives’ over ‘saving time’ and regarded the speed law as doing exactly that, even as average travelling speeds crept up to and then beyond 65.The attempt to measure the effect of the law was surrounded by controversy, so that it was difficult to form a good experimental design, to collect data from myriad agencies, much less to reach a consensus. In the
end, we could say only that ‘most people’ thought the law was responsible for 20 per cent up to 100 per cent of the increased safety.
