Community of Evaluators-Nepal

(CoE-Nepal)

Online Repository

Investigating Possible Alternative Explanations





  • 3,273 total views, 64 views today

All impact evaluations should include some attention to identifying and (if possible) ruling out alternative explanations for the impacts that have been observed.

Options

  • Force Field Analysis: providing a detailed overview of the variety of forces that may be acting on an organizational change issue.
  • General Elimination Methodology:this involves identifying alternative explanations and then systematically investigating them to see if they can be ruled out.
  • Key informant:asking experts in these types of programmes or in the community to identify other possible explanations and/or to assess whether these explanations can be ruled out.
  • Process tracing: ruling out alternative explanatory variables at each step of the theory of change.
  • RAPID outcomes assessment:a methodology to assess and map the contribution of a project’s actions on a particular change in policy or the policy environment.
  • Ruling out technical explanations:identifying and investigating possible ways that the results might reflect technical limitations rather than actual causal relationships.
  • Searching for disconfirming evidence/Following up exceptions:Treating data that doesn’t fit the expected pattern not as outliers but as potential clues to other causal factors and then seeking to explain them.
  • Statistically controlling for extraneous variables: collecting data on the extraneous variables, as well as the independent and dependent variables is an option for removing the influence of the variable on the study of program results.

Approaches

These approaches combine ruling out possible alternative explanations with options to check the results support causal attribution.

  • Contribution Analysis: assessing whether the program is based on a plausible theory of change, whether it was implemented as intended, whether the anticipated chain of results occurred and the extent to which other factors influenced the program’s achievements.
  • Collaborative Outcomes Reporting: mapping existing data against the theory of change, and then using a combination of expert review and community consultation to check for the credibility of the evidence.
  • Multiple Lines and Levels of Evidence(MLLE): reviewing a wide range of evidence from different sources to identify consistency with the theory of change and to explain any exceptions.
  • Rapid Outcomes Assessment: assessing and mapping the contribution of a project’s actions on a particular change in policy or the policy environment.