Beihang University breaks down modal barriers and introduces a universal physical combat method across visible and infrared modes.

In recent years, research on the safety assessment of visual perception systems has gradually deepened. Researchers have successfully developed visible light modal safety assessment technology based on different carriers such as glasses, stickers, clothing, etc., and there are also Some new attempts at infrared modalities. However, these technologies can only be applied to a single modality

With the development of artificial intelligence technology, visible-thermal infrared imaging technology has been widely used in security monitoring, autonomous driving and other security On a critical mission. Visible light imaging can provide rich texture information during the day, while infrared imaging can clearly display the thermal radiation distribution of the target at night. Using the two together, the visual perception system can achieve 24-hour full coverage and is not subject to environmental restrictions, which has many advantages. Therefore, there is a need to study a unified security assessment method for multi-modal visual perception systems

However, achieving multi-modal assessment is extremely challenging. First, it is difficult to apply universal attack methods under different imaging mechanisms. Previous methods were proposed based on the imaging characteristics of specific target modalities, and are difficult to work in other modalities. Furthermore, it is difficult to balance stealth performance, production cost and flexible application. It is not easy to be dual effective in visible light and the more difficult infrared mode, and it is even more difficult to achieve low-cost and convenient production and use.

Faced with many challenges, researchers from Beihang Institute of Artificial Intelligence explored the common shape attributes between visible light and infrared modalities, and innovatively proposed "cross-modal universality" "Anti-patch" to achieve visible light-infrared synchronized stealth. It selects materials that are easy to obtain, low cost, and have excellent thermal insulation properties to make convenient patches, which are ready to use. While filling the gaps in the robustness evaluation technology of visible light-infrared multi-modal detection systems in the current physical world, it also takes into account The ease and immediacy of physical implementation. Experiments demonstrate the effectiveness of this method under different detection models and modalities, as well as its generalization in multiple scenarios. Currently, this paper has been accepted by ICCV 2023.

## Please click the following link to view the paper: https://arxiv.org/abs/2307.07859

Code link: https://github.com/Aries-iai/Cross-modal_Patch_Attack

Technical points

This research uses evolutionary algorithms as the basic framework to conduct program design and effect improvement from three perspectives: shape modeling, shape optimization and modal balance. The specific process is shown in the figure below:

1. Multi-anchor shape modeling based on spline interpolation

#For the basic shape modeling part, the researchers designed a new paradigm of point optimization modeling. By changing the coordinates of the points, the shape of the patch can be directly adjusted without being affected by restrictions such as direction, distance, etc. This effectively increases the search space for patch shapes. In order to ensure the naturalness of the shape, the researchers also used spline interpolation methods to achieve smooth connections, allowing the splines to follow the control points more closely

2. Boundary defining shape The optimization algorithm is based on the principle of differential evolution

Effective optimization methods are required to achieve combat. For this reason, researchers consider the evolutionary algorithm as the basic framework from the perspectives of time cost, actual effect, etc. , and improved from the two perspectives of boundary setting and fitness function:

The content that needs to be rewritten is: (1) Boundary setting: By setting the boundary of the anchor point, It can improve the effectiveness of deformation and reduce time costs. The specific settings are as follows: loops or self-intersections will not be formed in the curve segment; cusps will not easily appear in the curve segment; they will not appear in the invalid area

Take anchor point as an example. The blue part in the figure below is the boundary setting legend, and the orange part is the error instance:

About the boundary determination of the anchor point The mathematical expression is as follows:

(2) Fitness function: This study is different from previous work that only conducted strike evaluation on a single mode. Instead, it focuses on the two modes of visible light and infrared, and realizes that there are natural differences between the two modes. The issue of differences in balance effects. To avoid over-optimizing a single modality while neglecting the other, the researchers proposed an innovative cross-modal fitness function based on the detector's confidence score perception, which aims to encourage exploration of successful directions and balance the two effect differences between modalities. In the end, survival of the fittest will be based on scores. In order to take into account the difference in strike difficulty between the initial stage and the later stage, this function uses an exponential function instead of a linear function to highlight the differences in strike progress at different stages

The algorithm iterates the exploration process until both modes are successfully attacked and outputs the optimal shape strategy. The complete optimization process is as follows:

Experimental results

Experiment 1: For different series of detectors Verification of cross-modal strike performance

Experiment 2: Ablation experiment for shapes

Experiment 3: Ablation experiment for cross-modal fitness function

Experiment 4: Method under physical implementation deviation Robustness verification

Experiment 5: Method effectiveness verification under different physical conditions

Perform performance verification at different angles, distances, postures and scenarios, and present the results visually

Summary

The core of this research is natural shape optimization, combined with deformation patches and cross-modal strikes, to design a visible-infrared multi-mode in a physical environment State robustness evaluation method. This method can evaluate the robustness of a multi-modal (visible-infrared) target detection system and effectively correct the detector model based on the evaluation results, while improving the accuracy of target image detection in both visible and infrared modes. The method is implemented and applied in a physical environment, contributing to the robustness evaluation and improvement of multi-modal detection systems

The above is the detailed content of Beihang University breaks down modal barriers and introduces a universal physical combat method across visible and infrared modes.. For more information, please follow other related articles on the PHP Chinese website!