


Harbin Institute of Technology and Nanyang Institute of Technology propose the world's first 'Multi-modal DeepFake Detection and Positioning' model: giving AIGC no place to hide fakes
Due to the rapid development of visual generative models such as Stable Diffusion, high-fidelity face images can be automatically forged, creating an increasingly serious DeepFake problem.
With the emergence of large language models such as ChatGPT, a large number of fake articles can also be easily generated and maliciously spread false information.
To this end, a series of single-modal detection models have been designed to deal with the forgery of the above AIGC technology in image and text modalities. However, these methods cannot cope well with multi-modal fake news tampering in new forgery scenarios.
Specifically, in multi-modal media tampering, the faces of important figures in pictures of various news reports (the face of the French President in Figure 1) are replaced, and the text Key phrases or words have been tampered with (in Figure 1, the positive phrase "is welcome to" has been tampered with the negative phrase "is forced to resign").
This will change or cover up the identity of key news figures, as well as modify or mislead the meaning of news text, creating multi-modal fake news that is spread on a large scale on the Internet.
## Figure 1. This paper proposes the task of detecting and locating multi-modal media tampering (DGM4). Different from existing single-modal DeepFake detection tasks, DGM4 not only predicts whether the input image-text pair is true or false, but also attempts to detect more fine-grained tampering types and locate image tampered areas and text tampering. word. In addition to true and false binary classification, this task provides a more comprehensive explanation and deeper understanding of tamper detection.
Table 1: Proposed DGM4 versus existing image and text forgery detection Comparison of related tasks
Detecting and locating multi-modal media tampering tasksTo understand this new challenge, research from Harbin Institute of Technology (Shenzhen) and Nanyang Technological University The researchers proposed the task of detecting and locating multi-modal media tampering (DGM4), built and open sourced the DGM4 data set, and also proposed a multi-modal hierarchical tampering inference model. Currently, this work has been included in CVPR 2023.
On Article address: https://arxiv.org/abs /2304.02556
GitHub:https://github.com/rshaojimmy/MultiModal-DeepFake
Project homepage: https://rshaojimmy.github.io/Projects/MultiModal-DeepFake
As shown in Figure 1 and Table 1, detection And the difference between Detecting and Grounding Multi-Modal Media Manipulation (DGM4) and existing single-modal tampering detection is:
1) Different from existing DeepFake image detection and fake text detection methods that can only detect single-modal fake information, DGM4 requires simultaneous detection of multi-modality in image-text pairs. State tampering;
2) Unlike existing DeepFake detection that focuses on binary classification, DGM4 further considers locating image tampered areas and text tampered words. This requires the detection model to perform more comprehensive and in-depth reasoning for tampering between image-text modalities.
Detect and locate multi-modal media tampering data setIn order to support the research on DGM4, as shown in Figure 2, the contribution of this work Developed the world's firstdetection and location of multi-modal media tampering (DGM4) data set.
Figure 2. DGM4Dataset
DGM The 4 data set investigates 4 types of tampering, face replacement tampering (FS), face attribute tampering (FA), text replacement tampering (TS), and text attribute tampering (TA).
Figure 2 shows the overall statistical information of DGM4, including (a) the distribution of the number of tampering types; (b) the tampered areas of most images are small in size , especially for face attribute tampering; (c) text attribute tampering has fewer tampered words than text replacement tampering; (d) distribution of text sentiment scores; (e) number of samples for each tampering type.
This data generated a total of 230,000 image-text pair samples, including 77,426 original image-text pairs and 152,574 tampered sample pairs. The tampered sample pairs include 66722 face replacement tampering, 56411 face attribute tampering, 43546 text replacement tampering and 18588 text attribute tampering.
Multimodal hierarchical tampering inference model
This article believes that multimodal tampering will cause subtle semantic inconsistencies between modalities. Therefore, detecting the cross-modal semantic inconsistency of tampered samples by fusing and inferring semantic features between modalities is the main idea of this article to deal with DGM4.
Figure 3. The proposed multi-modal hierarchical tampering inference model HierArchical Multi-modal Manipulation rEasoning tRansformer (HAMMER)
Based on this idea, as shown in Figure 3, this article proposes a multi-modal hierarchical tampering inference model HierArchical Multi-modal Manipulation rEasoning tRansformer (HAMMER).
This model is built on the model architecture of multi-modal semantic fusion and reasoning based on the dual-tower structure, and integrates the detection and location of multi-modal tampering in a fine-grained and hierarchical manner through shallow Layer and deep tamper reasoning are implemented.
Specifically, as shown in Figure 3, the HAMMER model has the following two characteristics:
1) In shallow tampering inference , through Manipulation-Aware Contrastive Learning to align the single-modal semantic features of images and texts extracted by the image encoder and text encoder. At the same time, the single-modal embedded features use the cross-attention mechanism for information interaction, and a local patch attention aggregation mechanism (Local Patch Attentional Aggregation) is designed to locate the image tampering area;
2) In deep tamper reasoning, the modality-aware cross-attention mechanism in the multi-modal aggregator is used to further fuse multi-modal semantic features. On this basis, special multi-modal sequence tagging and multi-modal multi-label classification are performed to locate text tampered words and detect altered words. Fine-grained tampering types.
Experimental results
As shown below, the experimental results show that the HAMMER proposed by the research team can detect more accurately compared with multi-modal and single-modal detection methods and locating multimodal media tampering.
Figure 4. Visualization of multi-modal tamper detection and location results
Figure 5 . Model Tamper Detection Attention Visualization on Tampered Text
Figure 4 provides some visualization results of multi-modal tamper detection and localization, illustrating that HAMMER can accurately and simultaneously Tamper detection and localization tasks. Figure 5 provides the model attention visualization results on tampered words, further demonstrating that HAMMER performs multi-modal tampering detection and localization by focusing on image areas that are semantically inconsistent with the tampered text.
Summary
- This work proposes a new research topic: the task of detecting and locating multi-modal media tampering to deal with multi-modal fake news.
- This work contributes the first large-scale data set for detecting and locating multi-modal media tampering, and provides detailed and rich annotations for tamper detection and location. The team believes it can well help future research on multi-modal fake news detection.
- This work proposes a powerful multi-modal hierarchical tampering inference model as a good starting solution for this new topic.
The code and data set link of this work have been shared on the GitHub of this project. Everyone is welcome to Star this GitHub Repo and use the DGM4 data set and HAMMER Let’s study the DGM4 problem. The field of DeepFake is not only about single-modality detection of images, but also a broader multi-modal tampering detection problem that needs to be solved urgently!
The above is the detailed content of Harbin Institute of Technology and Nanyang Institute of Technology propose the world's first 'Multi-modal DeepFake Detection and Positioning' model: giving AIGC no place to hide fakes. For more information, please follow other related articles on the PHP Chinese website!

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics

Imagine an artificial intelligence model that not only has the ability to surpass traditional computing, but also achieves more efficient performance at a lower cost. This is not science fiction, DeepSeek-V2[1], the world’s most powerful open source MoE model is here. DeepSeek-V2 is a powerful mixture of experts (MoE) language model with the characteristics of economical training and efficient inference. It consists of 236B parameters, 21B of which are used to activate each marker. Compared with DeepSeek67B, DeepSeek-V2 has stronger performance, while saving 42.5% of training costs, reducing KV cache by 93.3%, and increasing the maximum generation throughput to 5.76 times. DeepSeek is a company exploring general artificial intelligence

Earlier this month, researchers from MIT and other institutions proposed a very promising alternative to MLP - KAN. KAN outperforms MLP in terms of accuracy and interpretability. And it can outperform MLP running with a larger number of parameters with a very small number of parameters. For example, the authors stated that they used KAN to reproduce DeepMind's results with a smaller network and a higher degree of automation. Specifically, DeepMind's MLP has about 300,000 parameters, while KAN only has about 200 parameters. KAN has a strong mathematical foundation like MLP. MLP is based on the universal approximation theorem, while KAN is based on the Kolmogorov-Arnold representation theorem. As shown in the figure below, KAN has

Boston Dynamics Atlas officially enters the era of electric robots! Yesterday, the hydraulic Atlas just "tearfully" withdrew from the stage of history. Today, Boston Dynamics announced that the electric Atlas is on the job. It seems that in the field of commercial humanoid robots, Boston Dynamics is determined to compete with Tesla. After the new video was released, it had already been viewed by more than one million people in just ten hours. The old people leave and new roles appear. This is a historical necessity. There is no doubt that this year is the explosive year of humanoid robots. Netizens commented: The advancement of robots has made this year's opening ceremony look like a human, and the degree of freedom is far greater than that of humans. But is this really not a horror movie? At the beginning of the video, Atlas is lying calmly on the ground, seemingly on his back. What follows is jaw-dropping

The performance of JAX, promoted by Google, has surpassed that of Pytorch and TensorFlow in recent benchmark tests, ranking first in 7 indicators. And the test was not done on the TPU with the best JAX performance. Although among developers, Pytorch is still more popular than Tensorflow. But in the future, perhaps more large models will be trained and run based on the JAX platform. Models Recently, the Keras team benchmarked three backends (TensorFlow, JAX, PyTorch) with the native PyTorch implementation and Keras2 with TensorFlow. First, they select a set of mainstream

AI is indeed changing mathematics. Recently, Tao Zhexuan, who has been paying close attention to this issue, forwarded the latest issue of "Bulletin of the American Mathematical Society" (Bulletin of the American Mathematical Society). Focusing on the topic "Will machines change mathematics?", many mathematicians expressed their opinions. The whole process was full of sparks, hardcore and exciting. The author has a strong lineup, including Fields Medal winner Akshay Venkatesh, Chinese mathematician Zheng Lejun, NYU computer scientist Ernest Davis and many other well-known scholars in the industry. The world of AI has changed dramatically. You know, many of these articles were submitted a year ago.

The latest video of Tesla's robot Optimus is released, and it can already work in the factory. At normal speed, it sorts batteries (Tesla's 4680 batteries) like this: The official also released what it looks like at 20x speed - on a small "workstation", picking and picking and picking: This time it is released One of the highlights of the video is that Optimus completes this work in the factory, completely autonomously, without human intervention throughout the process. And from the perspective of Optimus, it can also pick up and place the crooked battery, focusing on automatic error correction: Regarding Optimus's hand, NVIDIA scientist Jim Fan gave a high evaluation: Optimus's hand is the world's five-fingered robot. One of the most dexterous. Its hands are not only tactile

Target detection is a relatively mature problem in autonomous driving systems, among which pedestrian detection is one of the earliest algorithms to be deployed. Very comprehensive research has been carried out in most papers. However, distance perception using fisheye cameras for surround view is relatively less studied. Due to large radial distortion, standard bounding box representation is difficult to implement in fisheye cameras. To alleviate the above description, we explore extended bounding box, ellipse, and general polygon designs into polar/angular representations and define an instance segmentation mIOU metric to analyze these representations. The proposed model fisheyeDetNet with polygonal shape outperforms other models and simultaneously achieves 49.5% mAP on the Valeo fisheye camera dataset for autonomous driving

This paper explores the problem of accurately detecting objects from different viewing angles (such as perspective and bird's-eye view) in autonomous driving, especially how to effectively transform features from perspective (PV) to bird's-eye view (BEV) space. Transformation is implemented via the Visual Transformation (VT) module. Existing methods are broadly divided into two strategies: 2D to 3D and 3D to 2D conversion. 2D-to-3D methods improve dense 2D features by predicting depth probabilities, but the inherent uncertainty of depth predictions, especially in distant regions, may introduce inaccuracies. While 3D to 2D methods usually use 3D queries to sample 2D features and learn the attention weights of the correspondence between 3D and 2D features through a Transformer, which increases the computational and deployment time.
