Analysis of Competing Hypotheses

The Analysis of Competing Hypotheses (ACH) provides an unbiased methodology for evaluating multiple competing hypotheses for observed data. It was developed by Richards (Dick) J. Heuer, Jr., a 45-year veteran of the Central Intelligence Agency, in the 1970s for use by the Agency and is used by analysts in various fields who make judgments that entail a high risk of error in reasoning. It helps an analyst overcome, or at least minimize, some of the cognitive limitations that make prescient intelligence analysis so difficult to achieve.

ACH was indeed a step forward in intelligence analysis methodology, but it was first described in relatively informal terms. Producing the best available information from uncertain data remains the goal of both researchers, tool-builders, and industrial, academic, and government data analysts. Their domains include data mining, cognitive psychology and visualization, probability and statistics, etc.

Process

Heuer outlines the ACH process in considerable depth in his book, "Psychology of Intelligence Analysis". It consists of the steps:
* Hypothesis - The first step of the process is to identify all potential hypotheses, preferably using a group of analysts with different perspectives to brainstorm the possibilities. The process discourages the analyst from choosing one “likely” hypothesis and using evidence to prove its accuracy. Cognitive bias is minimized when all possible hypotheses are considered.
* Evidence - The analyst then lists evidence and arguments (including assumptions and logical deductions) for and against each hypothesis.
* Diagnostics - With the use of a matrix, the analyst applies evidence against each hypothesis in an attempt to disprove as many theories as possible. Some evidence will have greater “diagnosticity” than others—that is, some will be most helpful in judging the relative likelihood of alternative hypotheses. According to Heuer, this step is the most important. Instead of looking at one hypothesis and all evidence, the matrix forces the analyst to consider one piece of evidence and examine it against all possible hypotheses.
* Refinement - At this point the analyst reviews his/her findings and then identifies gaps and collects the additional evidence needed to refute as many of the remaining hypotheses as possible.
* Inconsistency - The analyst seeks to draw tentative conclusions about the relative likelihood of each hypothesis, with less consistency equaling less likelihood. The least consistent hypotheses are eliminated. While the matrix generates a mathematical sum, it is up to the analyst to use his/her judgment to decide on the conclusion.
* Sensitivity - The analyst then tests his/her conclusions using sensitivity analysis, which weighs how the conclusion would be affected if key evidence or arguments were wrong, misleading, or subject to a different interpretation. The validity of key evidence and the consistency of important arguments are double-checked to assure that the conclusion's linchpins and drivers are sound. citation
title= Psychology of Intelligence Analysis
first = Richards J., Jr | last = Heuer
contribution = Chapter 8: Analysis of Competing Hypotheses
url = https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/psychology-of-intelligence-analysis/art11.html | publisher = Center for the Study of Intelligence, Central Intelligence Agency]
* Conclusions and evaluation - The analyst then provides the decisionmaker with his/her conclusions, as well as a summary of alternatives that were considered and why they were rejected. The analyst also identifies milestones in the process that can serve as indicators in future analyses.

trengths

There are many benefits of doing an ACH matrix: it is auditable and helps overcome cognitive biases. Since the ACH requires the analyst to construct a matrix, the evidence and hypotheses can be backtracked. This allows the decisionmaker or other analysts to see the sequence of rules and data that led to the conclusion. Lastly, by analyzing the evidence in reference to all possible hypotheses, it prevents bias.

Weaknesses

The process to create an ACH is time consuming. The ACH matrix can be problematic when analyzing a complex project. It can be cumbersome for an analyst to manage a large database with multiple pieces of evidence.

Especially in intelligence, both governmental and business, analysts must always be aware that the opponent(s) is intelligent and may be generating information intended to deceive. citation
title = Deception Detection by Analysis of Competing Hypotheses
first1 = Frank J. | last1=Stech |first2= Christopher | last2=Elsaesser
publisher = Mitre Corporation
url = https://analysis.mitre.org/proceedings/Final_Papers_Files/94_Camera_Ready_Paper.pdf
comment = MITRE Sponsored Research Project 51MSR111, Counter-Deception Decision Support] Since deception often is the result of a cognitive trap, Stech and Elsaesser propose the use of state-based (see automata theory) and Bayesian analysis to isolate assumptions implicit in the evaluation of paths in, or conclusions of, particular hypothesis selection. As the assumptions become available, they can become the subject of separate validation. Should an assumption become false, all hypotheses depending on it are rejected. This is a form of root cause analysis.

Evidence also presents a problem if it is unreliable.The evidence used in the matrix is static and therefore it can be a snapshot in time.

tructured Analysis of Competing Hypotheses

The Structured Analysis of Competing Hypotheses offers analysts an improvement over the limitations of the original ACH.discuss|April 2008 citation
authors = Wheaton, Kristan J., "et al."
title = Structured Analysis of Competing Hypotheses: Improving a Tested Intelligence Methodology
journal = Competitive Intelligence Magazine
volume = 9
issue = 6
date = November-December 2006
pages = 12-15 |url =http://www.mcmanis-monsalve.com/assets/publications/intelligence-methodology-1-07-chido.pdf] The SACH maximizes the possible hypotheses by allowing the analyst to split one hypothesis into two complex ones.

For example, two tested hypotheses could be that Iraq has WMD or Iraq does not have WMD. If the evidence showed that it is more likely there are WMDs in Iraq then two new hypotheses could be formulated: WMD are in Baghdad or WMD are in Mosul. Or perhaps, the analyst may need to know what type of WMD Iraq has; the new hypotheses could be that Iraq has biological WMD, Iraq has chemical WMD and Iraq has nuclear WMD. By giving the ACH structure, the analyst is able to give a nuanced estimate. [citation
author = Chido, Diane E. "et al."
title = Structured Analysis Of Competing Hypotheses: Theory and Application
publisher = Mercyhurst College Institute for Intelligence Studies Press
date = 2006
page =54 ]

Other approaches to formalism

One method, by Valtorta and colleagues uses probabilistic methods, addings Bayesian analysis to ACH. citation
url = http://www.cse.sc.edu/~mgv/reports/IA-05.pdf
contribution = Extending Heuer’s Analysis of Competing Hypotheses Method to Support Complex Decision Analysis
author = Valtorta, Marco "et al."
title= International Conference on Intelligence Analysis Methods and Tools
date = May 2005] The work by Akram and Wang applies paradigms from graph theorycitation
url = http://www.huitfeldt.com/repository/ITW06.pdf
contribution = Investigative Data Mining: Connecting the dots to disconnect them
pages = 28-34
first1= Shaikh Muhammad | last1 = Akram | last2 = Wang | first2 = Jiaxin
title = Proceedings of the 2006 Intelligence Tools Workshop
date = 23 August 2006]

Other work focuses less on probabilistic methods and more on cognitive and visualization extensions to ACH, as discussed by Madsen and Hicks. citation
url = http://www.huitfeldt.com/repository/ITW06.pdf
contribution = Investigating the Cognitive Effects of Externalization Tools
pages = 4-11
first1= Fredrik H. | last1 = Madsen| first2 = David L. | last2 = Hicks
title = Proceedings of the 2006 Intelligence Tools Workshop
date = 23 August 2006] DECIDE,discussed under automation is visualization-oriented. citation
first1= Diane |last1 =Cluxton |first2 =Stephen G. | last2 = Eick
contribution = "DECIDE™ Hypothesis Visualization Tool"
title =2005 Intl conf on Intelligence Analysis
url = https://analysis.mitre.org/proceedings/Final_Papers_Files/119_Camera_Ready_Paper.pdf]

Work by Pope and Jøsang uses subjective logic, a formal mathematical methodology that explicitlydeals with uncertainty.citation
id= ADA463908
title = Analysis of Competing Hypotheses using Subjective Logic (ACH-SL)
publisher = Queensland University, Brisbane, Australia
last1= Pope | first1= Simon |last2 = Josang | first2= Audun
date = June 2005
url = http://handle.dtic.mil/100.2/ADA463908 ] . This was used to create the ShEBA software.

Automation

A few online and downloadable tools help automate the ACH process. These programs leave a visual trail of evidence and allow the analyst to weigh evidence.

PARC ACH 2.0citation
url = http://www2.parc.com/istl/projects/ach/ach.html
authors = Xerox Palo Alto Research Center and Richards J. Heuer
title =ACH2.0.3 Download Page: Analysis of Competing Hypotheses (ACH)] was developed by Palo Alto Research Center (PARC) in collaboration with Richards J. Heuer, Jr. It is a standard ACH program that allows an analysts to enter evidence and rate its credibility and relevance. Another useful program is the Decision Command™ software created by Dr. Willard Zangwill citation
url = http://www.quantinus.com/aboutus/biodrwillardzangwill.asp | title = Quantinus Biography
first = Willard | last = Zangwill ]

SSS Research, Inc. is an analytic research firm that created DECIDE.citation
url = http://www.cs.umd.edu/hcil/VASTcontest06/SUBMITTED/SSSResearch-DECIDE/index.html
author = Lankenau, Russell A. "et al."
title = SSS Research, Inc. – DECIDE, VAST 2006 Contest Submission
date = July 2006] DECIDE not only allows analysts to manipulate ACH, but it provides multiple visualization products.citation
url = http://www.sss-research.com/decide.aspx
title = DECIDE™: from Complexity to Clarity
author =SSS Research]

ee also

*Intelligence analysis
*Intelligence (information gathering)
*Cognitive bias
*Words of Estimative Probability

Notes