Community work, social welfare and health programs are notoriously difficult to evaluate. So many factors are involved that one evaluation instrument is likely to be inadequate and longitudinal studies are necessary. Weiss and Rein pointed out in 1970 that

...evaluation techniques for broad aim programs (such as most health and welfare services) should be many and varied, and the tight scientific experimental approach is not suitable.

In 1979, an Australian Senate Standing Committee on Social Welfare published commissioned papers on "Evaluation in Australian Health and Welfare Services". The papers covered such topics as evaluation in relation to standards of performance, methods of evaluating effectiveness of human services, the views of clients and, the need and demand in the making of planning decisions in the evaluation of health and welfare services.

Shadish et al (1991) draw together theories of evaluation practice up to 1991 via the main headings of theory of social programming, theory of use, theory of valuing and theory of knowledge. The areas of agreement and disagreement of the evaluation theorists are noted but ultimately the authors provide evaluators with a greater range of conceptual and

practical options from which to choose in thinking about and doing their work.

Evaluation defined

Owen (2006) describes evaluation as

...the process of providing information designed to assist decision making about the object being evaluated. The term `evaluand' has been used as an alternative generic term to 'object'. In practice the evaluand could be a program, a policy, an event or a training manual or text book.

He progresses to the practicalities of evaluation and the identification of five forms of evaluation and their characteristics. It is important to choose the most appropriate form of evaluation for a given purpose.

An understanding of the Forms can be achieved by considering the reason for undertaking the evaluation (orientation), the degree to which the program under review has been implemented at the time of the proposed evaluation (state), the components on which the evaluation is likely to concentrate (focus), the temporal links between evaluation and program delivery (timing), and the particular evaluation approach or approaches which are consistent with what is required (evaluation approach).

Quantitative and Qualitative evaluation

A quantitative evaluation approach tends to report with statistics, graphs, pie charts, histograms etc. and are relatively easy to work with and interpret. They convey a level of objectivity and also provide for the

development of theories and realities in the scientific manner.

Kushner (1996) criticises science methodology in evaluation, using constructivism as a specific example,

...for its failure to acknowledge that theories and realities are not `out there' waiting to be discovered or uncovered, but are constructed in the minds of individuals or in the discourses of groups.

She goes further to contend that evaluation approaches such as constructivism is an over-reaction to the problems of objective reality and to free evaluation from legacies of natural and social science approaches which are more qualitative in approach.

Constructivism as applied to evaluation risks losing sight of culture - one of the most powerful of integrating analyses.

Miles and Huberman  (1984) considered how qualitative data could be more useful than quantitative data:

Qualitative data are attractive. They are a source of well-grounded, rich description and explanation of processes occuring in local contexts. With qualitative data, one can preserve chronological flow, assess local causality, and derive fruitful explanations. Serendipitious findings and new theoretical integrations can appear. Finally, qualitative findings have a certain undeniability that is often far more convincing to a reader than pages of numbers.

They recognised at the time that there was an insufficient body of knowledge concerning qualitative techniques which was reliable or valid. Even minimally agreed-on working analysis procedures for qualitative data were not available. This difficulty is exacerbated when qualitative researchers do not clearly record their methodologies. In addressing these issues Miles and Huberman (1986) provide a conception of qualitative analysis and illustrative analysis methods including data reduction, data display, and conclusion drawing and verification which could be applied to qualitative evaluations.

It is necessary to balance the arguments 'for' and 'against' quantitative and qualitative approaches of evaluation with the possibility of utilising both. Caracelli and Riggin (1994) consider the development of mixed-method evaluation processes with particular emphasis on quality criteria for such an approach. They:

...develop a multidimensional conceptualization of quality mixed-method evaluations, that is, evaluations that use both qualitative and quantitative methods to achieve a more coherent evaluation.

The foundation has been laid, however, there is considerably more research necessary, perhaps utilising the work of Miles and Huberman (1986).

Multiple approaches to an evaluation

Enlightenment

Additionally, it is observed that utilisation of evaluation findings has a range of meanings. One  is a conceptual understanding of the influence of a social intervention, also termed enlightenment.

For practising evaluators, two aspects are particularly salient. The first is to ensure that audiences become aware of information derived from evaluation projects ...The most effective conceptual use occurs when evaluation information substantially informs the understanding of decision-makers, because, in the long run, this depth of understanding affects the quality of decisions which follow. A second aspect is to identify the nature of the information which is most needed by audiences.

Accountability or lesson-learning?

Cracknell (1996) considered the strengths and weaknesses of evaluating Development Aid, however it is an interesting study of the dilemma facing many organisations and evaluators: whether to give priority to accountability or lesson-learning as the main objective. Of particular note is the observation that

...we are finding that, at least for socially oriented projects, too much precision in the setting of objectives and the selection of indicators can well be counter-productive.

What emerges with clarity is that if socially oriented projects are to have any chance of success they must be directed at meeting the real needs of the intended beneficiaries, and must carry the full support of the target population. Development in the past has been too often for' and not 'with' people.

House et al (1996) have observed that many large bureaucracies have developed their own internal evaluation units rather than engaging external evaluators. In this study of a large bureaucracy they concluded that internal interference with findings was not realised to any great extent. The internal evaluation process necessarily involved the development of an evaluation culture which they define as:

...those social arrangements and practices that sustain honest evaluative inquiry and utilization of findings.

It is recognised that establishing an evaluation culture requires work; establishing trust by consultations, working with staff on evaluation designs, holding seminars and forums, working through problems attendant on the execution of successful evaluation activities.

...it depends on a community of people who talk to each other about evaluation in a convivial and critical manner, a self-critical but progressive and optimistic group. Expectations are important, especially that people should think about and do evaluation as a professional activity.

As previously noted, Kushner describes culture as one of the most powerful of integrating analyses and therefore needs to be taken into consideration in an evaluation process. An organisation with an anti-evaluation culture could make life extremely difficult for an evaluator.

As important as evaluation is, heed the warning not to evaluate the life out of important volunteer activities.