TY - GEN
T1 - An Empirical Analysis of Blind Tests
AU - Baral, Kesina
AU - Offutt, Jeff
N1 - Publisher Copyright: © 2020 IEEE.
PY - 2020/10
Y1 - 2020/10
N2 - Modern software engineers automate as many tests as possible. Test automation allows tests to be run hundreds or thousands of times: hourly, daily, and sometimes continuously. This saves time and money, ensures reproducibility, and ultimately leads to software that is better and cheaper. Automated tests must include code to check that the output of the program on the test matches expected behavior. This code is called the test oracle and is typically implemented in assertions that flag the test as passing if the assertion evaluates to true and failing if not. Since automated tests require programming, many problems can occur. Some lead to false positives, where incorrect behavior is marked as correct, and others to false negatives, where correct behavior is marked as incorrect. This paper identifies and studies a common problem where test assertions are written incorrectly, leading to incorrect behavior that is not recognized. We call these tests blind because the test does not see the incorrect behavior. Blind tests cause false positives, essentially wasting the tests. This paper presents results from several human-based studies to assess the frequency of blind tests with different software and different populations of users. In our studies, the percent of blind tests ranged from a low of 39% to a high of 95%.
AB - Modern software engineers automate as many tests as possible. Test automation allows tests to be run hundreds or thousands of times: hourly, daily, and sometimes continuously. This saves time and money, ensures reproducibility, and ultimately leads to software that is better and cheaper. Automated tests must include code to check that the output of the program on the test matches expected behavior. This code is called the test oracle and is typically implemented in assertions that flag the test as passing if the assertion evaluates to true and failing if not. Since automated tests require programming, many problems can occur. Some lead to false positives, where incorrect behavior is marked as correct, and others to false negatives, where correct behavior is marked as incorrect. This paper identifies and studies a common problem where test assertions are written incorrectly, leading to incorrect behavior that is not recognized. We call these tests blind because the test does not see the incorrect behavior. Blind tests cause false positives, essentially wasting the tests. This paper presents results from several human-based studies to assess the frequency of blind tests with different software and different populations of users. In our studies, the percent of blind tests ranged from a low of 39% to a high of 95%.
KW - Test automation
KW - assertions
KW - test oracle
UR - https://www.scopus.com/pages/publications/85091563139
U2 - 10.1109/ICST46399.2020.00034
DO - 10.1109/ICST46399.2020.00034
M3 - Conference contribution
T3 - Proceedings - 2020 IEEE 13th International Conference on Software Testing, Verification and Validation, ICST 2020
SP - 254
EP - 262
BT - Proceedings - 2020 IEEE 13th International Conference on Software Testing, Verification and Validation, ICST 2020
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 13th IEEE International Conference on Software Testing, Verification and Validation, ICST 2020
Y2 - 23 March 2020 through 27 March 2020
ER -