ABSTRACT
There are major methodological problems related to assessing the effectiveness and impact of capacity building efforts in health systems. These challenges include:
Meeting donor needs for quantification. In trying to reassure donors that capacity building efforts are yielding positive results, there is a tendency to develop over-bureaucratised systems which attempt to measure too many indicators. It is thus important to ensure that indicators selected to measure the impact of capacity building efforts are reasonable and practical to collect. 1
Being multi-dimensional and user friendly. Capacity building activities in health (e.g. poverty alleviation) tend to produce multiple effects, hence the need for multi-dimensional capacity building tools. The OECD Development Cooperation Directorate for assessing effectiveness of development aid includes the following checklist. 2
Relevance: the extent to which the aid activity is suited to the priorities and policies of the target group, recipient and donor. In evaluating the relevance of a programme or a project, it is useful to consider the following questions. To what extent are the objectives of the programme still valid? Are the activities and outputs of the programme consistent with the overall goal and the attainment of its objectives? Are the activities and outputs of the programme consistent with the intended impacts and effects?
Effectiveness: a measure of the extent to which an aid activity attains its objectives. In evaluating the effectiveness of a programme or a project, it is useful to consider the following questions. To what extent were the objectives achieved/are likely to be achieved? What were the major factors influencing the achievement or non-achievement of the objectives?
Efficiency: measures the outputs – qualitative and quantitative – in relation to the inputs. It is an economic term which signifies that the aid uses the least costly resources possible in order to achieve the desired results. This generally requires comparing alternative approaches to achieving the same outputs, to see whether the most efficient process has been adopted. When evaluating the efficiency of a programme or a project, it is useful to consider the following questions. Were activities cost efficient? Were objectives achieved on time? Was the programme or project implemented in the most efficient way compared to alternatives?
Impact: the positive and negative changes produced by a development intervention, directly or indirectly, intended or unintended. This involves the main impacts and effects resulting from the activity on the local social, economic, environmental and other development indicators. The examination should be concerned with both intended and unintended results and must also include the positive and negative impact of external factors, such as changes in terms of trade and financial conditions. When evaluating the impact of a programme or a project, it is useful to consider the following questions. What has happened as a result of the programme or project? What real difference has the activity made to the beneficiaries? How many people have been affected?
Sustaiuability: is concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable. When evaluating the sustainability of a programme or a project, it is useful to consider the following questions. To what extent did the benefits of a programme or project continue after donor funding ceased? What were the major factors which influenced the achievement or non-achievement of sustainability of the programme or project?
Demonstrating attribution is a major problem with capacity building initiatives, given its diffuse implementation approaches and the multiplicity of processes that result in specific outcomes. However, viewed from the definitional perspective of ‘ability to undertake specific objectives’, ‘before and after’ assessments may facilitate adequate attribution of capacity building measures. The MEASURE Evaluation Project is actively involved with measuring and attributing capacity building initiatives. The MEASURE capacity building indicators are shown in Table 20.1. MEASURE capacity building indicators at various levels of health systems https://www.niso.org/standards/z39-96/ns/oasis-exchange/table">
Capacity building level
Inputs
Process
Outputs
Intermediates outcomes
Health system
Population per doctor
Ratio of health care spending on primary versus tertiary care
Percentage of health budget funded by external sources
Donor coordination committee meets every six months
Collaborative ‘arrangements’ exist between social sectors – e.g. meetings between health and agriculture or health and education
Number of multi-sectoral meetings held
Number of collaborative projects initiated outside health
Existence of national standards for professional qualifications
Existence of sector-wide strategy
Widely distributed sector-wide strategy
Regular auditing of system-wide accounts by independent company
Organisation
Existence of clear mission statement
Presence of operational planning system
Presence of detailed job descriptions
Clearly defined organizational structure
Coordination with other organisations evident through internal reporting mechanisms
Job descriptions are regularly updated to reflect real work requirements and responsibilities
Presence of a financial management system that regularly provides income/revenue data and cash flow analysis
Capacity to track commodities
Individual work plans are prepared for all staff
A sufficient number of sites functioning as clinical training sites to meet clinic practice needs
Realised operational targets
Ability to adjust services in response to evaluation results or emergencies
Reports generated on time
Cost-sharing revenue as a proportion of the annual MOH non-wage recurrent budget
Percentage of trained health workers that correctly diagnose 2–4 months after training
Health personnel
Adequacy of the training material/supplies has been assessed in 1 or more institutions
Adequate training supplies available in sufficient quantities to support ongoing RH/FP training in 1/more institutions
Number of training sessions to improve human resource capacity which focus on needs identified by the service providing institution
Percentage of courses where training methodology is appropriate for transfer of skills/knowledge
Number of providers trained, by type of training and cadre of provider
Number of staff trained in finance, MIS, strategic planning, financial planning
Number of managers trained, by type of training
Percentage of trainees (providers) competent in skill (i.e. met set standard when applying skill learned in training)
Percentage of trainees who apply skills (learned through training) to their subsequent work
Individual/community
Average level of education (number of years) attained in the district
Mean income level
Proportion whose partner recently died in central hospital
Existence of community leadership
Percentage who think they are at risk of contracting serious illness
Percentage who report previous poor experience of the health care system
Level of community cohesiveness
Proportion of non-users who desire to use contraception in the future
Level of participation in community health committees
Percentage of new mothers who bring their children for immunisation at the right time
Proportion whose partner recently died in central hospital
Level of community mobilisation and empowerment
Source: adapted from reference 3.Measuring intangible changes. Particularly in health systems, ‘not everything that counts can be counted’, as Einstein aptly reminds us. Improved subjective well-being is an example of a capacity building outcome that, though difficult to measure, is very important from the perspective of health consumers. Many measures of institutional and organisational capacity are subjective, and rely mainly on individual judgment and perception, thus limiting their reliability and generalisability. Harmony in working relationships between the various components of health systems is vital for efficient functioning, but it is difficult to measure or attribute to capacity building efforts.
Ensuring adequacy of skills to conduct impact assessment is a major challenge in capacity building efforts, in part due to low priority accorded such activities during health sector budgeting processes. Self-assessment may create conflict of interest issues, which might result in data manipulation in order to present results in favourable light. On the other hand, internal stakeholders may mistrust external evaluators, and misconstrue evaluation tools as a ploy for sanctions. This potential challenge may be addressed through prior agreement of the personnel and tools that will be utilised in the conduct of impact assessment.
Overall cost-effectiveness of impact assessment processes. Meaningful capacity building assessment is expensive, and its utility is strongly conditioned by assessment tools, organisational knowledge and level of investment in the evaluation process. Two approaches to determining evaluation budgets are to limit the total amount authorised, or to link it to the total budget (e.g. maximum 0.5 per cent of total budget). Evaluation personnel are best selected from a wide section of stakeholders.
