Whether you are working for a large enterprise, a startup or yourself, unit testing is not only useful, but is often indispensable. We use unit tests to test the code, but what if our tests are wrong or incomplete? What can we use to test our tests? Who will supervise the inspector? 
Key Points
Mutation test
No, it's not that kind of mutation. Variation testing (or Variation analysis) is a technique used to create and evaluate the quality of software tests. It involves modifying the test in a very small way. Each modified version is called a variant, and the test detects and rejects variants by causing the original version to behave differently from the variant. Mutations are errors in the original code, and analysis checks whether our tests detect these errors. In short, if the test is still valid after the mutation, it is not a good test.
Mutation test using Humbug
Humbug is a variant testing framework for PHP. In order for Humbug to generate code coverage, we must install and enable XDebug on our machine. We can then install it as a global tool.
composer global require 'humbug/humbug'
, if we run
humbug
command should be able to see some Humbug installation information and an error indicating that we do not have a humbug.json file.
Boot program
Before we configure and use Humbug, we need a project that can be tested. We will create a small PHP calculator package where we run our unit tests and mutation tests. Let's create a /Calculator folder. In it, let's create our /src and /tests folders. In our /src folder, we will have our application code; the /tests folder will contain our unit tests. We also need to use PHPUnit in our package. The best way is to use Composer. Let's install PHPUnit using the following command:
composer global require 'humbug/humbug'
Let's create our calculator. In the /src folder, create a Calculator.php file and add the following:
humbug
This is a fairly simple program. A simple calculator with basic arithmetic, percentage and logarithmic operations and functions that return π values. Next, in our /tests folder, let's create unit tests for our calculator. If you need help with unit testing in PHP, check out this tutorial. Create a CalculatorTest.php file and add the following:
composer global require phpunit/phpunit
This will be our initial test stack. If we run the phpunit command, we will see it execute successfully and our 4 tests and 4 assertions will pass. It is important that all tests must be passed, otherwise Humbug will fail.
Configuration Humbug
Humbug can be configured manually by creating a humbug.json.dist file, or automatically by running the following command:
<?php
namespace package\Calculator;
class Calculator {
/**
* 基本运算
*/
public function add($a1, $a2) {
return $a1 + $a2;
}
public function subtract($a1, $a2) {
return $a1 - $a2;
}
public function multiply($a1, $a2) {
return $a1 * $a2;
}
public function divide($a1, $a2) {
if ($a2 === 0) {
return false;
}
return $a1 / $a2;
}
/*
* 百分比
*/
// 这将返回 a1 的 a2 百分比
public function percentage($a1, $a2) {
return ( $a1 / $a2 ) * 100;
}
/*
* π
*/
// 返回 π 的值
public function pi() {
return pi();
}
/*
* 对数
*/
// 返回以 10 为底的基本对数
public function log($a) {
return log10($a);
}
}Running this command will ask us to answer some questions:
Use Humbug
Now that we have run the application and tested it, and we have Humbug installed, let's run Humbug and check the results.
<?php
use package\Calculator\Calculator;
class CalculatorTest extends PHPUnit_Framework_TestCase {
public function testAdd() {
$calculator = new Calculator();
$result = $calculator->add(2, 3);
$this->assertEquals($result, 5);
}
public function testSubtract() {
$calculator = new Calculator();
$result = $calculator->subtract(6, 3);
$this->assertEquals($result, 3);
}
public function testMultiply() {
$calculator = new Calculator();
$result = $calculator->multiply(6, 3);
$this->assertEquals($result, 18);
}
public function testDivide() {
$calculator = new Calculator();
$result = $calculator->divide(6, 3);
$this->assertEquals($result, 2);
}
}The result should be close to this: 
The number of mutations created is just the number of small changes introduced by Humbug to test our tests. The killed mutation (.) is the mutation that causes the test to fail. Don't be confused, this is a positive result! Escape mutation (M) is the mutation that still passes the test. This is not a positive result, we should go back to our tests and check what is missing. Uncovered mutation (S) is a mutation that occurs in rows that are not covered by unit tests. Fatal error (E) and timeout (T) are the mutations that create fatal error and create infinite loops, respectively.
Variation score metrics represent the percentage of mutations detected. Our goal is 100%. Variation code coverage represents the percentage of tests covered by the mutation. Variation score metrics can give you an idea of the effectiveness of existing tests. Analyzing our humbug logs, we can see that we have 9 uncovered mutations, as well as some very bad metrics. Check out the humblogjson.json file. This file is automatically generated like the humblog.txt file and contains more detailed information about the causes, location, and reasons for the failure. We did not test our percentage, π, and logarithmic functions. Additionally, we need to cover the case where the number is divided by 0. Let's add more tests to cover the missing situation:
composer global require 'humbug/humbug'
This time, 100% means that all mutations are killed and we have full code coverage.
The biggest disadvantage of mutation testing and Humbug is performance. Variation testing is a slow process because it relies on many factors such as interactions between lines of code, number of tests, code coverage level, and the performance of code and tests. Humbug also performs initial test runs, logging, and code coverage, which increases the total duration. Additionally, Humbug is PHPUnit-specific and can be a problem for users using other test frameworks. That is, Humbug is under active development and will continue to improve.
Conclusion
Humbug can be an important tool for maintaining the life of your application. As application complexity increases, so does testing complexity—and it becomes very important to always keep 100% testing, especially when dealing with the enterprise ecosystem. The code used in this tutorial can be cloned here. Have you used Humbug? Are you doing mutation testing in other ways? Please tell us all your thoughts on this!
Frequently Asked Questions about "Who will supervise the inspector?" (FAQ)
"Who will supervise the inspector?" This sentence originated from the Latin phrase "Quis custodiet ipsos custodes?", created by the Roman poet Juvenal. This sentence is often used in discussions that question the integrity and accountability of those in power. It's basically asking, "Who will guard the guard?" or "Who will monitor the people who monitor us?" This sentence has appeared in various forms of media, including Alan Moore, Dave Gibbons and John ·Higgins' popular graphic novel series "Watchman".
In the context of software testing, "Who will supervise the inspector?" is a metaphorical question that solves the reliability and accuracy of the test. It questions who or what is monitoring the tests to ensure they can function properly and produce accurate results. This is a key aspect of software development because it ensures the quality and reliability of the software being developed.
Test testing, also known as test verification, is a key part of software development. It ensures that testing accurately measures the functionality and performance of the software. If there is no test verification, there is a risk that the test may produce false positives or missed reports, resulting in an inaccurate assessment of software quality and reliability.
Ensure the reliability and accuracy of the test involves several steps. First, you should thoroughly check your tests to make sure they are designed and implemented correctly. Second, you should verify your tests regularly by comparing the test results with known results. Finally, you should continuously monitor and update your tests to ensure they remain accurate as the software evolves.
Some common pitfalls in software testing include insufficient testing, wrong testing, and not understanding the purpose of testing. Other pitfalls include over-reliance on automated tests without understanding their limitations, and periodic review and update of tests.
The graphic novel "Watchman" uses the sentence "Who supervises the inspector?" to question the responsibility and integrity of those in power. In the context of software testing, this sentence can be used to question the reliability and accuracy of the test itself. Just as the Watcher should protect society, testing should protect the quality and reliability of software. But just as the Watcher needs to be monitored, the test needs to be monitored.
The role of software testers is to ensure the quality and reliability of the software by designing and implementing testing. These tests are used to identify and fix errors, verify functionality, and evaluate performance. Software testers must also monitor and update these tests to ensure they remain accurate during the software development.
Improving your software testing skills requires continuous learning and practice. You should be aware of the latest testing methods and tools and practice designing and implementing tests regularly. You should also seek feedback on your tests and be happy to learn from your mistakes.
There are many resources available to learn more about software testing. These include online courses, books, blogs and forums. Some recommended books include "The Art of Software Testing" by Glenford J. Myers, "Software Testing: The Craftsman Method" by Paul C. Jorgensen, and Seim Karnar and Jack Falke "Testing Computer Software" by Huang Q. Ruan.
The future of software testing may be strongly affected by technological advances. This includes the increasing use of automation and artificial intelligence in testing, as well as the development of new testing methods to adapt to emerging technologies such as virtual reality and blockchain. However, the basic principles of software testing—to ensure the quality and reliability of the software—will remain the same.
The above is the detailed content of Testing Your Tests? Who Watches the Watchmen?. For more information, please follow other related articles on the PHP Chinese website!
![[Web front-end] Node.js quick start](https://img.php.cn/upload/course/000/000/067/662b5d34ba7c0227.png)
