Inter-Coder Agreement for Improving Reliability in Software Engineering Qualitative Research
share
In recent years, the research on empirical software engineering that uses qualitative data analysis (e.g. thematic analysis, content analysis, and grounded theory) is increasing. However, most of this research does not deep into the reliability and validity of findings, specifically in the reliability of coding, despite there exist a variety of statistical techniques known as Inter-Coder Agreement (ICA) for analyzing consensus in team coding. This paper aims to establish a novel theoretical framework that enables a methodological approach for conducting this validity analysis. This framework is based on a set of statistics for measuring the degree of agreement that different coders achieve when judging a common matter. We analyze different reliability coefficients and provide detailed examples of calculation, with special attention to Krippendorff's α coefficients. We systematically review several variants of Krippendorff's α reported in the literature and provide a novel common mathematical framework in which all of them are unified through a universal α coefficient. Finally, this paper provides a detailed guide of the use of this theoretical framework in a large case study on DevOps culture. We explain how α coefficients is computed and interpreted using a widely used software tool for qualitative analysis like Atlas.ti.
READ FULL TEXT