Skip to main navigation Skip to search Skip to main content

An Empirical Analysis of Blind Tests

  • George Mason University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

7 Scopus citations

Abstract

Modern software engineers automate as many tests as possible. Test automation allows tests to be run hundreds or thousands of times: hourly, daily, and sometimes continuously. This saves time and money, ensures reproducibility, and ultimately leads to software that is better and cheaper. Automated tests must include code to check that the output of the program on the test matches expected behavior. This code is called the test oracle and is typically implemented in assertions that flag the test as passing if the assertion evaluates to true and failing if not. Since automated tests require programming, many problems can occur. Some lead to false positives, where incorrect behavior is marked as correct, and others to false negatives, where correct behavior is marked as incorrect. This paper identifies and studies a common problem where test assertions are written incorrectly, leading to incorrect behavior that is not recognized. We call these tests blind because the test does not see the incorrect behavior. Blind tests cause false positives, essentially wasting the tests. This paper presents results from several human-based studies to assess the frequency of blind tests with different software and different populations of users. In our studies, the percent of blind tests ranged from a low of 39% to a high of 95%.

Original languageEnglish
Title of host publicationProceedings - 2020 IEEE 13th International Conference on Software Testing, Verification and Validation, ICST 2020
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages254-262
Number of pages9
ISBN (Electronic)9781728157771
DOIs
StatePublished - Oct 2020
Event13th IEEE International Conference on Software Testing, Verification and Validation, ICST 2020 - Porto, Portugal
Duration: Mar 23 2020 → Mar 27 2020

Publication series

NameProceedings - 2020 IEEE 13th International Conference on Software Testing, Verification and Validation, ICST 2020

Conference

Conference13th IEEE International Conference on Software Testing, Verification and Validation, ICST 2020
Country/TerritoryPortugal
CityPorto
Period03/23/20 → 03/27/20

Keywords

  • Test automation
  • assertions
  • test oracle

Fingerprint

Dive into the research topics of 'An Empirical Analysis of Blind Tests'. Together they form a unique fingerprint.

Cite this