Psychometrics in Behavioral Software Engineering: A Methodological Introduction with Guidelines
share
Humans are what constitutes the most complex and complicated, yet fascinating, component of a software engineering endeavor. A meaningful and deep understanding of the human aspects of software engineering requires psychological constructs to be taken into account. We argue that psychology and statistics theory can facilitate the development and adoption of valid and reliable instruments to assess these constructs. In particular, to ensure high quality, the psychometric properties of measurement instruments need evaluation. In this paper, we provide an introduction to psychometric theory for the evaluation of measurement instruments (e.g., psychological tests and questionnaires) for software engineering researchers. We present guidelines that enable using existing instruments and developing new ones adequately. We conducted a comprehensive review of the psychology literature, including journal articles, textbooks, and society standards, framed by the Standards for Educational and Psychological Testing (American Educational Research Association et al., 2014). We detail activities used when operationalizing new psychological constructs, such as item analysis, factor analysis, standardization and normalization, reliability, validity, and fairness in testing and test bias. With this paper, we hope to encourage a culture change in software engineering research towards the adoption of established methods from social science. To improve the quality of behavioral research in software engineering, we believe that studies focusing on introducing, validation, and then using psychometric instruments need to be more common. Finally, we present an example of a psychometric evaluation based on our guidelines, to which we openly provide code and dataset.
READ FULL TEXT