Causal Testing: Finding Defects' Root Causes
share
Isolating and repairing unexpected or buggy software behavior typically involves identifying and understanding the root cause of that behavior. We develop Causal Testing, a new method of root-cause analysis that relies on the theory of statistical causal inference to identify a set of executions that likely hold the key causal information necessary to understand and repair buggy behavior. Given one or more faulty executions, Causal Testing finds a small set of minimally different executions that do not exhibit the faulty behavior. Reducing the differences any further causes the faulty behavior to reappear, and so the differences between these minimally different executions and the faulty executions capture causal execution information, which can aid system understanding and debugging tasks. Evaluating Causal Testing on a subset of the Defects4J benchmark, we find that Causal Testing could be applied to 71 real-world defects, and for 77 root cause of the defect. We implement and make public Holmes, a Causal Testing Eclipse plug-in that automatically computes and presents causal information. A controlled experiment with 16 developers showed that Holmes improves the subjects' ability to identify the cause of the defect: Users with standard testing tools identified the cause 81 did so 92
READ FULL TEXT