Myths about AI testing

In recent years, I have been paying attention to AI-related testing and actively participated in multiple national testing communities and communities. In these communities, I communicate with testing experts from different companies and fields to discuss topics related to AI testing, including experts from top companies in the industry and well-known domestic testing scholars. I also participated in many conferences, listened to many sharings on AI testing topics, and tried a variety of AI-related testing tools, from which I gained a lot of knowledge and insights.

In these testing communities and communities, I encountered many questions about AI testing, such as what is AI testing, how to conduct AI testing, what tools and methods are available for AI testing, etc. However, when I searched for books related to AI testing on the Internet, I found a large number of books related to AI development, but few books specifically introduced AI testing. This shows that the testing industry is still groping forward in chaos.

In order to share the knowledge and experience I have learned about AI testing, I sorted out my learning experience, tried to answer some common questions I encountered about AI testing, and organized these contents into articles , communicate and discuss with everyone.

The Myth of AI Testing

When discussing AI testing, there are usually two understandings:

• The first one It uses AI to assist current software testing, such as using AI systems to learn test analysis and test design, and then automatically generate test cases and automatically implement these test cases.
• The second is to test the AI ​​system. Although the industry still uses conventional testing methods for testing AI systems, such as functional testing, performance testing, and security testing, it is often difficult to obtain clear test data and acceptance conditions when testing the effectiveness of its functions. In this case, testing can only be done through a deep understanding of the algorithm and by generating or finding data based on experience, and roughly assessing the validity of the functional test results.

The use of AI to assist current automated testing is an emerging field. Using AI (such as deep learning) systems to assist testing work is definitely one of the hottest testing trends in recent years, including automatically generating and executing automated tests, large-scale test result analysis, automated exploratory testing, defect location, etc.

Many companies in the United States have launched commercial AI testing tools. In Teacher Zhu Shaomin's public account "Software Quality Report", there is an article titled "The future is here, artificial intelligence testing is unstoppable: Introducing 9 AI testing tools", which introduces 9 AI-based testing tools. However, these AI testing tools commonly have problems such as test case accuracy and large-scale test case maintainability.

The first question: Can AI-assisted testing really be used?

Although many companies have begun to study AI-assisted testing and many tools have come out, they all have one significant problem: accuracy. Due to the limitations of the existing AI learning algorithm itself, the accuracy of the test cases and verification conditions learned and generated is not very high. I have attended several conferences where the accuracy rate of AI-assisted testing shared by some of China's first-tier Internet manufacturers was only slightly above 80% and less than 90%. This kind of accuracy is difficult to be recognized in some systems that require high accuracy, such as finance.

Secondly, when the scale of automated test cases is large, it is difficult to rely on manual maintenance of test cases and can only rely on tools. Due to the inaccuracy of AI testing tools, the accuracy of maintenance work is not very ideal.

Nonetheless, in large systems with low quality requirements, AI-assisted testing can greatly reduce testing costs, so in these systems, AI-assisted automated testing has been applied. In addition, in projects with high quality requirements and sufficient resources, AI-assisted testing can be used as an extension of manual automated testing and as a tool for automated exploratory testing to further ensure software quality.

Second question: To what extent has AI-assisted testing developed?

Currently, AI-assisted testing is still in its infancy. I divide AI-assisted testing into three stages:

• The first stage is to automatically generate the input of test cases through the deep learning model, and then manually verify the correctness of the output results.
• The second stage is to automatically generate test case input through the deep learning model, and automatically verify the correctness of the output results through the rule model.
• The third stage is to automatically generate the input and output of the test case through the deep learning model, and automatically verify the correctness of the output results.

At present, the industry has basically realized the first stage, and some companies have also begun to realize the second stage. However, only a handful of large companies have achieved stage three, and the accuracy of these companies has yet to be improved. Therefore, AI-assisted testing still has a long way to go.

The third question: Which software systems can be tested with AI assistance?

Theoretically, any software system can use AI to assist in automated testing. However, due to the current status of AI testing systems, it cannot truly be used for all types of software systems. Many actual projects only use AI testing on specific interface layers of certain systems, such as Web Service APIs, etc.

In different industry fields, the usage of AI testing is also different. For example, in industries with high quality requirements such as finance and military industry, AI testing can currently only be used as an extension of existing functional automated testing. In industries with low quality requirements such as the Internet, some companies with strong technical capabilities have adopted AI testing as one of the main automated testing methods. However, AI-assisted automated testing is undoubtedly the future of automated testing!

Summary

Through the answers to the questions above, I hope to solve everyone’s confusion about AI testing, including understanding what AI testing is, whether your project is suitable for AI testing, and whether it is needed in the future. Issues such as investing in AI testing.

To sum up, AI testing is still in the early stages of development, so it is not suitable for large-scale use and can only be considered for use in specific projects. Whether you use AI or manual methods to implement automated testing, the core is the effectiveness of the test, the accuracy of the test results, and the maintainability of the test cases. These are critical factors in the success of automated testing at scale.

Therefore, whether it is AI automated testing or manual automated testing, its core essence is the same: relying on a series of human thinking such as knowledge learning, analysis, and summary to solve test analysis, test design, and automated test implementation work. However, the current development of the AI ​​system itself is not enough to replace human work in test analysis and test design. As a result, the accuracy and maintainability of automated tests implemented by AI are worse than manual methods. However, AI testing has obvious advantages in terms of time and cost, which is why it is becoming more and more popular.

Therefore, in the field of testing, AI needs to work harder to truly replace human work. Before considering using AI testing, the specific needs and conditions of the project should be carefully evaluated to determine whether AI testing is appropriate. In the future, with the continuous development and improvement of AI technology, AI testing will become an important trend in automated testing, and investing in AI testing related technologies and talents will also be a wise choice.

The above is the detailed content of Myths about AI testing. For more information, please follow other related articles on the PHP Chinese website!