Evaluation 62433 225793793 2008-07-15T12:51:37Z Mysdaao 133241 Restoring See also links, references, external links, and categories previously removed {{Cleanup|date=May 2007}} {{this|characterizing and appraising something of interest}} '''Evaluation''' is systematic determination of merit, worth, and significance of something or someone. Evaluation often is used to characterize and appraise subjects of interest in a wide range of human enterprises, including the [[arts]], [[criminal justice]], [[foundation (charity)|foundations]] and [[non-profit organization]]s, [[government]], [[health care]], and other human services. ==Evaluation standards and meta-evaluation== Depending on the topic of interest, there are professional groups which look to the quality and rigor of the evaluation process. One guiding principle within the U.S. evaluation community, energetically supported by Michael Quinn-Patton has been that evaluations be useful. Further more the international organizations such as the I.M.F. and the World Bank have independent evaluation functions. The various funds, programmes, and agencies of the United Nations has a mix of independent, semi-independent and self-evaluation functions, which have organized themselves as a system-wide UN Evaluation Group (UNEG)<ref> [http://www.uneval.org UNEG]</ref>, that works together to strengthen the function, and to establish UN norms and standards for evaluation. There is also an evaluation group within the OECD-DAC, which endeavors to improve development evaluation standards. <ref>[http://www.oecd.org/site/0,2865,en_21571361_34047972_1_1_1_1_1,00.html DAC Network on Development Evaluation Home Page]</ref> The [[Joint Committee on Standards for Educational Evaluation]] <ref>[http://www.wmich.edu/evalctr/jc/ Joint Committee on Standards for Educational Evaluation]</ref> has developed standards for educational programmes, personnel, and student evaluation. The Joint Committee standards are broken into four sections: ''Utility, Feasibility, Propriety,'' and ''Accuracy''. Various European institutions have also prepared their own standards, more or less related to those produced by the Joint Committee. They provide guidelines about basing value judgments on systematic inquiry, evaluator competence and integrity, respect for people, and regard for the general and public welfare. The American Evaluation Association has created a set of Guiding [[Principle]]s <ref>[http://www.eval.org/Publications/GuidingPrinciples.asp American Evaluation Association Guiding Principles for Evaluators]</ref> for evaluators. The order of these principles does not imply priority among them; priority will vary by situation and evaluator role. The principles run as follows: * Systematic [[Inquiry]]: Evaluators conduct systematic, [[data]]-based inquiries about whatever is being evaluated. * [[Competence (human resources)|Competence]]: Evaluators provide competent performance to [[Stakeholder (corporate)|stakeholder]]s. * [[Integrity]] / [[Honesty]]: Evaluators ensure the honesty and integrity of the entire evaluation [[Process (general)|process]]. * [[Respect]] for [[People]]: Evaluators respect the [[security]], [[dignity]] and [[Self-esteem|self-worth]] of the respondents, [[Program (management)|program]] [[Participation (decision making)|participants]], [[Consumer|clients]], and other stakeholders with whom they [[Interaction|interact]]. * [[Social responsibility|Responsibilities]] for General and [[Public]] [[Quality of life|Welfare]]: Evaluators articulate and take into account the diversity of [[interest]]s and [[Value (personal and cultural)|value]]s that may be related to the general and public welfare. ==Evaluation approaches== {{main|Evaluation approaches}} [[Evaluation approaches]] are conceptually distinct ways of thinking about, designing and conducting evaluation efforts. Many of the evaluation approaches in use today make truly unique contributions to solving important problems, while others refine existing approaches in some way. ===Classification of approaches=== Two classifications of evaluation approaches by House <ref>House, E. R. (1978). Assumptions underlying evaluation models. ''Educational Researcher''. 7(3), 4-12.</ref> and Stufflebeam & Webster <ref>Stufflebeam, D. L., & Webster, W. J. (1980). An analysis of alternative approaches to evaluation. ''Educational Evaluation and Policy Analysis''. 2(3), 5-19.</ref> can be combined into a manageable number of approaches in terms of their unique and important underlying principles. House considers all major evaluation approaches to be based on a common [[ideology]], [[liberal democracy]]. Important principles of this ideology include [[freedom of choice]], the uniqueness of the [[individual]], and [[Empiricism|empirical]] [[inquiry]] grounded in [[Objectivity (philosophy)|objectivity]]. He also contends they are all based on [[Subject (philosophy)#Subjectivism|subjectivist]] [[ethics]], in which ethical conduct is based on the subjective or intuitive experience of an individual or group. One form of subjectivist ethics is [[Utilitarianism|utilitarian]], in which “[[Goodness and value theory|the good]]” is determined by what maximizes some single, explicit interpretation of happiness for society as a whole. Another form of subjectivist ethics is [[Ethical intuitionism|intuitionist]] / [[Value pluralism|pluralist]], in which no single interpretation of “the good” is assumed and these interpretations need not be explicitly stated nor justified. These ethical positions have corresponding [[Epistemology|epistemologies]]—[[Philosophy|philosophies]] of obtaining [[knowledge]]. The objectivist epistemology is associated with the utilitarian ethic. In general, it is used to acquire knowledge capable of external verification (intersubjective agreement) through publicly inspectable [[Methodology|methods]] and [[data]]. The subjectivist epistemology is associated with the intuitionist/pluralist ethic. It is used to acquire new knowledge based on existing personal knowledge and experiences that are (explicit) or are not (tacit) available for [[public]] inspection. House further divides each epistemological approach by two main [[Politics|political]] perspectives. Approaches can take an [[elite]] perspective, focusing on the interests of managers and professionals. They also can take a mass perspective, focusing on [[consumer]]s and [[Participation (decision making)|participatory]] approaches. Stufflebeam and Webster place approaches into one of three groups according to their orientation toward the role of [[Value (personal and cultural)|values]], an [[Ethics|ethical]] consideration. The political orientation promotes a positive or negative view of an object regardless of what its value actually might be. They call this pseudo-evaluation. The questions orientation includes approaches that might or might not provide answers specifically related to the value of an object. They call this quasi-evaluation. The values orientation includes approaches primarily intended to determine the value of some object. They call this true evaluation. When the above concepts are considered simultaneously, fifteen evaluation approaches can be identified in terms of epistemology, major perspective (from House), and orientation (from Stufflebeam & Webster). Two pseudo-evaluation approaches, politically controlled and public relations studies, are represented. They are based on an objectivist epistemology from an elite perspective. Six quasi-evaluation approaches use an objectivist epistemology. Five of them—experimental research, management information systems, testing programs, objectives-based studies, and content analysis—take an elite perspective. Accountability takes a mass perspective. Seven true evaluation approaches are included. Two approaches, decision-oriented and policy studies, are based on an objectivist epistemology from an elite perspective. Consumer-oriented studies are based on an objectivist epistemology from a mass perspective. Two approaches—accreditation/certification and connoisseur studies—are based on a subjectivist epistemology from an elite perspective. Finally, adversary and client-centered studies are based on a subjectivist epistemology from a mass perspective. ===Summary of approaches=== The following table is used to summarize each approach in terms of four [[Abstraction|attributes]]—organizer, purpose, strengths, and weaknesses. The organizer represents the main considerations or cues practitioners use to organize a study. The [[purpose]] represents the desired [[Outcome (Game theory)|outcome]] for a study at a very general level. Strengths and weaknesses represent other attributes that should be considered when deciding whether to use the approach for a particular study. The following narrative highlights differences between approaches grouped together. {| class="wikitable" border="1" cellpadding="5" cellspacing="0" |- ! colspan=5 align=center| '''Summary of approaches for conducting evaluations''' |- ! rowspan=2| '''Approach''' ! colspan=4 align="center"| '''Attribute''' |- ! '''Organizer''' ! '''Purpose''' ! '''Key strengths''' ! '''Key weaknesses''' |- ! '''[[Politics|Politically]] controlled''' | Threats | Get, keep or increase influence, power or money. | Secure evidence advantageous to the client in a conflict. | Violates the principle of full & frank disclosure. |- ! '''[[Public relations]]''' | Propaganda needs | Create positive public image. | Secure evidence most likely to bolster public support. | Violates the principles of balanced reporting, justified conclusions, & objectivity. |- ! '''[[Experiment|Experimental research]]''' | Causal relationships | Determine causal relationships between variables. | Strongest paradigm for determining causal relationships. | Requires controlled setting, limits range of evidence, focuses primarily on results. |- ! '''[[Management information system]]s''' | Scientific efficiency | Continuously supply evidence needed to fund, direct, & control programs. | Gives managers detailed evidence about complex programs. | Human service variables are rarely amenable to the narrow, quantitative definitions needed. |- ! '''[[Assessment|Testing]] programs''' | Individual differences | Compare test scores of individuals & groups to selected norms. | Produces valid & reliable evidence in many performance areas. Very familiar to public. | Data usually only on testee performance, overemphasizes test-taking skills, can be poor sample of what is taught or expected. |- ! '''[[Project objective|Objectives]]-based''' | Objectives | Relates outcomes to objectives. | Common sense appeal, widely used, uses behavioral objectives & testing technologies. | Leads to terminal evidence often too narrow to provide basis for judging to value of a program. |- ! '''[[Content analysis]]''' | Content of a communication | Describe & draw conclusion about a communication. | Allows for unobtrusive analysis of large volumes of unstructured, symbolic materials. | Sample may be unrepresentative yet overwhelming in volume. Analysis design often overly simplistic for question. |- ! '''[[Accountability]]''' | Performance expectations | Provide constituents with an accurate accounting of results. | Popular with constituents. Aimed at improving quality of products and services. | Creates unrest between practitioners & consumers. Politics often forces premature studies. |- ! '''[[Decision]]-oriented''' | Decisions | Provide a knowledge & value base for making & defending decisions. | Encourages use of evaluation to plan & implement needed programs. Helps justify decisions about plans & actions. | Necessary collaboration between evaluator & decision-maker provides opportunity to bias results. |- ! '''[[Policy analysis|Policy studies]]''' | Broad issues | Identify and assess potential costs & benefits of competing policies. | Provide general direction for broadly focused actions. | Often corrupted or subverted by politically motivated actions of participants. |- ! '''[[Customer|Consumer]]-oriented''' | Generalized needs & values, effects | Judge the relative merits of alternative goods & services. | Independent appraisal to protect practitioners & consumers from shoddy products & services. High public credibility. | Might not help practitioners do a better job. Requires credible & competent evaluators. |- ! '''[[Accreditation of Certification Body|Accreditation]] / [[Professional certification|certification]]''' | Standards & guidelines | Determine if institutions, programs, & personnel should be approved to perform specified functions. | Helps public make informed decisions about quality of organizations & qualifications of personnel. | Standards & guidelines typically emphasize intrinsic criteria to the exclusion of outcome measures. |- ! '''[[Connoisseur]]''' | Critical guideposts | Critically describe, appraise, & illuminate an object. | Exploits highly developed expertise on subject of interest. Can inspire others to more insightful efforts. | Dependent on small number of experts, making evaluation susceptible to subjectivity, bias, and corruption. |- ! '''Adversary''' | “Hot” issues | Present the pro & cons of an issue. | Ensures balances presentations of represented perspectives. | Can discourage cooperation, heighten animosities. |- ! '''Client-centered''' | Specific concerns & issues | Foster understanding of activities & how they are valued in a given setting & from a variety of perspectives. | Practitioners are helped to conduct their own evaluation. | Low external credibility, susceptible to bias in favor of participants. |- | colspan=5| Note. Adapted and condensed primarily from House (1978) and Stufflebeam & Webster (1980). |} '''Pseudo-evaluation''' '''[[Politics|Politically]] controlled''' and public relations studies are based on an objectivist epistemology from an elite perspective. Although both of these approaches seek to misrepresent value interpretations about some object, they go about it a bit differently. [[Information]] obtained through politically controlled studies is released or withheld to meet the special interests of the holder. '''[[Public relations]]''' information is used to paint a positive image of an object regardless of the actual situation. Neither of these approaches is acceptable evaluation practice, although the seasoned reader can surely think of a few examples where they have been used. '''Objectivist, elite, quasi-evaluation''' As a group, these five approaches represent a highly respected collection of disciplined inquiry approaches. They are considered quasi-evaluation approaches because particular studies legitimately can focus only on questions of knowledge without addressing any questions of value. Such studies are, by definition, not evaluations. These approaches can produce characterizations without producing appraisals, although specific studies can produce both. Each of these approaches serves its intended purpose well. They are discussed roughly in order of the extent to which they approach the objectivist ideal. '''[[Experiment|Experimental research]]''' is the best approach for determining [[Causality|causal relationships]] between [[variable]]s. The potential problem with using this as an evaluation approach is that its highly controlled and stylized [[methodology]] may not be sufficiently responsive to the dynamically changing needs of most human service programs. '''[[Management information system]]s''' (MISs) can give detailed information about the dynamic operations of complex programs. However, this information is restricted to readily [[Quantitative|quantifiable]] data usually available at regular intervals. '''[[Assessment|Testing]] programs''' are familiar to just about anyone who has attended school, served in the military, or worked for a large company. These programs are good at comparing individuals or groups to selected norms in a number of subject areas or to a set of standards of performance. However, they only focus on testee performance and they might not adequately sample what is taught or expected. '''[[Project objective|Objectives]]-based''' approaches relate outcomes to prespecified objectives, allowing judgments to be made about their level of attainment. Unfortunately, the objectives are often not proven to be important or they focus on outcomes too narrow to provide the basis for determining the value of an object. '''[[Content analysis]]''' is a quasi-evaluation approach because content analysis judgments need not be based on value statements. Instead, they can be based on knowledge. Such content analyses are not evaluations. On the other hand, when content analysis judgments are based on values, such studies are evaluations. '''Objectivist, mass, quasi-evaluation''' '''[[Accountability]]''' is popular with [[Constituent (politics)|constituents]] because it is intended to provide an accurate accounting of results that can improve the [[quality]] of [[Product (project management)|products]] and [[Service (economics)|services]]. However, this approach quickly can turn practitioners and [[consumer]]s into adversaries when implemented in a heavy-handed fashion. '''Objectivist, elite, true evaluation''' '''[[Decision]]-oriented''' studies are designed to provide a knowledge base for making and defending decisions. This approach usually requires the close [[collaboration]] between an evaluator and decision-maker, allowing it to be susceptible to [[Political corruption|corruption]] and [[bias]]. '''[[Policy analysis|Policy studies]]''' provide general guidance and direction on broad issues by identifying and assessing potential [[Cost-benefit analysis|costs and benefits]] of competing [[policy|policies]]. The drawback is these studies can be corrupted or subverted by the politically motivated actions of the participants. '''Objectivist, mass, true evaluation''' '''[[Customer|Consumer]]-oriented''' studies are used to judge the relative merits of goods and services based on generalized needs and values, along with a comprehensive range of effects. However, this approach does not necessarily help practitioners improve their work, and it requires a very good and credible evaluator to do it well. '''Subjectivist, elite, true evaluation''' '''[[Accreditation of Certification Body|Accreditation]] / [[Professional certification|certification]]''' programs are based on self-study and peer review of organizations, programs, and personnel. They draw on the insights, experience, and expertise of qualified individuals who use established guidelines to determine if the applicant should be approved to perform specified functions. However, unless performance-based standards are used, attributes of applicants and the processes they perform often are overemphasized in relation to measures of outcomes or effects. '''[[Connoisseur]]''' studies use the highly refined skills of individuals intimately familiar with the subject of the evaluation to critically characterize and appraise it. This approach can help others see programs in a new light, but it is difficult to find a qualified and unbiased connoisseur. '''Subjectivist, mass, true evaluation''' The '''adversary''' approach focuses on drawing out the pros and cons of [[Controversy|controversial]] issues through quasi-[[Lawsuit|legal proceedings]]. This helps ensure a balanced presentation of different perspectives on the issues, but it is also likely to discourage later [[Co-operation|cooperation]] and heighten animosities between contesting parties if “winners” and “losers” emerge. '''Client-centered''' studies address specific concerns and issues of practitioners and other clients of the study in a particular setting. These studies help people understand the activities and values involved from a variety of perspectives. However, this responsive approach can lead to low external [[credibility]] and a favorable bias toward those who participated in the study. ==Evaluation methods and techniques== Evaluation is methodologically diverse using both [[qualitative methods]] and [[quantitative methods]], including [[case studies]], [[survey research]], [[statistical analysis]], and model building among others. A more detailed list of methods, techniques and approaches for conducting evaluations would include the following: <!-- added columns for now - will expand and reorganize --> {{col-begin}} {{col-4}} * [[Accelerated aging]] * [[Action research]] * [[Advanced Product Quality Planning]] * [[Alternative assessment]] * [[Appreciative Inquiry]] * [[Assessment]] * [[Axiomatic design]] * [[Benchmarking]] * [[Case study]] * [[Change management]] * [[Clinical trial]] * [[Cohort study]] * [[Competitor analysis]] * [[Consensus decision-making]] * [[Consensus-seeking decision-making]] * [[Content analysis]] * [[Conversation analysis]] * [[Cost-benefit analysis]] {{col-4}} * [[Data mining]] * [[Delphi Technique]] * [[Discourse analysis]] * [[Educational accreditation]] * [[Electronic portfolio]] * [[Environmental scanning]] * [[Ethnography]] * [[Experiment]] * [[Experimental techniques]] * [[Factor analysis]] * [[Factorial experiment]] * [[Feasibility study]] * [[Field experiment]] * [[Fixtureless in-circuit test]] * [[Focus group]] * [[Force field analysis]] * [[Game theory]] * [[Grade (education)|Grading]] * [[Historical method]] * [[Inquiry]] {{col-4}} * [[Interview]] * [[Marketing research]] * [[Meta-analysis]] * [[Metrics]] * [[Most significant change]] * [[Multivariate statistics]] * [[Naturalistic observation]] * [[Observational techniques]] * [[Opinion poll]]ing * [[Organizational learning]] * Outcome Mapping * [[Participant observation]] * [[Participatory Impact Pathways Analysis]] * [[Policy analysis]] * [[Process improvement]] * [[Project management]] * [[Qualitative research]] * [[Quality audit]] * [[Quality circle]] * [[Quality control]] * [[Quality management]] * [[Quality Management System]] {{col-4}} * [[Quantitative research]] * [[Questionnaire]] * [[Questionnaire construction]] * [[Root cause analysis]] * [[Rubric (academic)|Rubrics]] * [[Sampling (statistics)|Sampling]] * [[Self-assessment]] * [[Six Sigma]] * [[Standardized testing]] * [[Statistical process control]] * [[Statistical survey]] * [[Statistics]] * [[Strategic planning]] * [[Structured interviewing]] * [[Systems theory]] * [[Test (student assessment)|Student testing]] * [[Total Quality Management]] * [[Triangulation (social science)|Triangulation]] {{col-end}} ==See also== * [[Assessment]] is the process of gathering and analyzing specific information as part of an evaluation. * [[Competency evaluation]] is a means for teachers to determine the ability of their students in other ways besides the standardized test. * [[Educational evaluation]] is evaluation that is conducted specifically in an educational setting. * [[Immanent evaluation]], opposed by [[Gilles Deleuze]] to [[value judgment]] * [[Performance evaluation]] is a term from the field of language testing. It stands in contrast to competence evaluation. * [[Program evaluation]] is essentially a set of philosophies and techniques to determine if a program 'works'. * [[Donald Kirkpatrick|Donald Kirkpatrick's]] Evaluation Model for training evaluation ==Notes and references== <references /> ==External links== {{sisterlinks|Evaluation}} *[http://www.evaluationwiki.org The EvaluationWiki] - The mission of EvaluationWiki is to make freely available a compendium of up-to-date information and resources to everyone involved in the science and practice of evaluation. The EvaluationWiki is presented by the non-profit [http://www.evaluationwiki.org/wiki/index.php/Evaluation_Resource_Institute Evaluation Resource Institute]. * [http://www.eval.org American Evaluation Association] - An international professional association of evaluators devoted to the application and exploration of program evaluation, personnel evaluation, technology, and many other forms of evaluation ** [http://bama.ua.edu/archives/evaltalk.html Evaltalk] - American Evaluation Association Discussion List * [http://ahe.cqu.edu.au Assessment in Higher Education] web site. * [http://evaluationcanada.ca Canadian Evaluation Society] - A Canada-wide non-profit bilingual association dedicated to the advancement of evaluation theory and practice * [http://www.sil.org/lingualinks/literacy/ReferenceMaterials/GlossaryOfLiteracyTerms/WhatIsFormativeEvaluation.htm Formative vs. Summative Evaluation] - Two general purposes for evaluation * [http://www.ibec-its.org IBEC] - International, Benefits, Evaluation and Costs Working Group for the ITS community * [http://www.education.purdue.edu/assessmentcouncil/Links/Index.htm Links to Assessment and Evaluation Resources] - List of links to resources on several topics *[http://www.mande.co.uk/ MandE NEWS] - A news service focusing on developments in monitoring and evaluation methods relevant to development projects and programmes with social development objectives. * [http://www.wmich.edu/evalctr/ Western Michigan University Evaluation Center] ** [http://www.wmich.edu/evalctr/checklists/ Checklists] ** [http://ec.wmich.edu/evaldir/ Directory of evaluators] ** [http://ec.wmich.edu/glossary/index.htm Glossaries] ** [http://ec.wmich.edu/resources/ Web resources] * [http://www.evaluation.lars-balzer.name/ Evaluation Portal] ** [http://www.evaluation.lars-balzer.name/links/ Evaluation Portal Link Collection] Evaluation link collection with information about evaluation journals, dissemination, projects, societies, how-to texts, books, and much more ** [http://www.evaluation.lars-balzer.name/calendar/ Evaluation Portal Calendar] International calendar for evaluation events (congresses, training events, methodological training, and everything useful in the field of evaluation) * [http://www.samea.org.za South African Monitoring and Evaluation Association] - A professional association of evaluation practitioners and users in South Africa **[http://www.particip.com] Particip is the leading company in developing evaluation tools for Develoment Aid Cooperation * M&E Blogs - A collection of blogs that evaluators use to communicate their ideas for various reasons **[http://weblogs.elearning.ubc.ca/mathison/ Sandra Mathison's Blog with student reading material] **[http://www.theusableblog.blogspot.com Eric Graig's Blog about using M&E with Not-for-profits] **[http://www.mandeblog.blogspot.com Benita van Wyk's Blog about M&E in South Africa] **[http://mandenews.blogspot.com/ Rick on the Road - reflections on M&E issues] *[http://gsociology.icaap.org/methods/ Free Resources for Methods in Evaluation and Social Research] [[ar:تقييم]] [[cs:Evaluace]] [[de:Evaluation]] [[nl:Evaluatie]] [[pl:Ewaluacja]] [[sk:Hodnotenie]] [[sr:Евалуација]] [[uk:Евальвація]] [[yi:עוואלוציע]] [[ru:Оценка_(значение)]] [[Category:Evaluation| ]] [[Category:Evaluation methods| ]]