Mutation Testing (MT)

Mutation Testing

Mutation Testing, also called Mutation Analysis, is not an approach to testing software, but rather, a method to measure and judge the coverage of a software test set. Similar to branch and path coverage, it provides a test adequacy metric. The goal is to make this adequacy metric become 1.0 by adding more test cases (and improving existing test cases) to achieve complete test coverage.

Unfortunately, mutation testing has a number of shortcomings that make it less than ideal for this purpose. In our ongoing research we observed some shortcomings of mutation testing and discovered an alternative approach that in our tests has proven to provide a better test adequacy metric.

As this research is not yet published, we cannot reveal the details here, but to understand some of the shortcomings and misconceptions of mutation testing, we recommend examining the many "Myths of Mutation Testing" listed in section 7 of the 2009 paper "Higher Order Mutation Testing" by Y Jia.