


End-to-end template-free response prediction model based on dual tasks
Re-formatted| It’s Hu Haozhe, a master’s student
Paper link: https://doi.org/10.1007/s10489-023-05048-8
When tested on three major reaction prediction data sets, testing on the retrosynthetic and forward reaction prediction tasks, and comprehensive ablation experiments on the above modules, we demonstrate that BiG2S is able to handle both retrosynthetic and forward reactions with a single model at appropriate parameter scales. Response prediction task. Compared with existing template-free methods based on pre-training and data enhancement, the overall prediction ability of BiG2S is equally excellent
Research backgroundRetrosynthesis and forward Synthesis is a fundamental challenge in the fields of organic chemistry, computer-aided synthesis planning (CASP), and computer-aided drug design (CADD)When rewriting the content, the original text needs to be rewritten into Chinese while maintaining the original meaning.
Early retrosynthetic planning systems directly relied on reaction rules pre-coded by domain experts, or calculations based on physical chemistry, but with the rapid development of deep learning. The current mainstream method in the field is to build a task-specific neural network framework to complete the reaction prediction task from a data-driven perspective. Among them, the template-free method, which does not rely on specific prior chemical knowledge, has gradually become one of the mainstream development directions in the field through its simplicity and flexibility similar to end-to-end machine translation.
Currently, the input and output of most template-free retrosynthetic models are SMILES strings of molecules, that is, using the sequence-to-sequence (Seq2Seq) process. This method can make good use of the existing model framework in the field of natural language processing, as well as the mature data processing flow for the SMILES representation method
However, since SMILES as a one-dimensional string sequence cannot be easily To better characterize and utilize the two-dimensional/three-dimensional structural information contained in molecular graphs, the graph-to-sequence (Graph2Seq) method that uses molecular graphs instead of SMILES as model input has gradually emerged in this field, or the additional structural information of molecular graphs has gradually emerged. Sequence-to-sequence methods embedded in SMILES sequences. Both methods can make good use of the rich structural features from molecular graphs
Based on this, this paper is based on the emerging graph-to-sequence method and combines retrosynthesis and forward reaction in the original SMILES-based model. Based on the related exploration benchmark of simultaneous training of prediction tasks, we further comprehensively explore the construction and experiment of this type of dual-task model, and also preliminarily explore and analyze the difficulty imbalance and Top-k matching displayed by the model during the training process. The problem of rate fluctuation; the BiG2S model built on this basis can better handle the retrosynthesis and forward reaction prediction tasks in mainstream data sets, and achieve results consistent with other template-free retrosynthesis models without using data enhancement. Response prediction ability
The overall framework needs to be rewrittenThe overall structure of BiG2S is an end-to-end encoder-decoder, as shown in Figure 1. The encoder side uses a local directed message passing graph network and a global graph Transformer that incorporates graph structure bias information to generate the final molecular graph node representation. The decoder uses a standard Transformer decoder to generate the SMILES sequence of the target molecule in an autoregressive mannerIt should be noted that in order to learn retrosynthesis and forward reaction prediction at the same time, the input to the decoder additionally contains Dual-task labeling of location information. At the same time, the normalization layer and the final linear layer on the decoder side have two sets of parameters, which are used to learn the retrosynthesis task and the forward reaction prediction task respectively
Figure 1: BiG2S overall framework diagramRetrosynthesis and forward reaction prediction are two related tasks. The retrosynthesis task uses products as input and reactants as target output, while the forward reaction prediction task does the opposite. There is a close connection between these two tasks, as they can be transformed into a forward reaction prediction task by exchanging the input and target output of the retrosynthesis task
Therefore, some template-free models based on SMILES have An attempt was made to improve the understanding of chemical reactions by using reverse synthesis and forward reaction prediction as training objectives, and achieved certain results. Based on this idea, the author further tried to introduce dual-task training into the graph-to-sequence model
Specifically, the author based on the parameter sharing strategy previously used on other methods, in the normalization layer of the decoder and Two sets of task-specific parameters are constructed within the final linear layer. In other modules, the two types of tasks share a set of parameters. At the same time, additional dual-task labels are added to the input molecular graph nodes and the initial input sequence of the decoder. In this way, even while controlling the overall model size, the model is able to distinguish between the two types of tasks and learn their different data distributions
Requires training and inference optimization
In During the training process, the author further recorded and analyzed two types of problems reflected by the model during the training process
First, the author recorded the frequency of occurrence of different SMILES characters in USPTO-50k and their corresponding ones during training The prediction accuracy is shown in Figure 2. During the training process, for S and Br, which accounted for 0.4% and 0.3% respectively in the training set, the absolute difference in overall prediction accuracy reached 8%. This initially shows that there are obvious differences in the difficulty of prediction between different molecular structures/fragments. Therefore, the author alleviates such problems by introducing an unbalanced loss function (such as Focal Loss), so that the model can pay more attention to the accuracy during training. Lower molecular fragments
Figure 2: In the USPTO-50k training set, the frequency of occurrence of different SMILES characters and their overall prediction accuracy during training
In addition, the author also recorded the quality changes of the model's prediction results on the validation set during training, as shown in Figure 3. The author found that in the middle and late training stages of the USPTO-50k data set, the Top-1 accuracy of the model on the validation set was still improving, but there was a decline in the prediction quality of Top-3, Top-5, and Top-10 Significant decrease
In order to improve the top-1 prediction quality of the model while maintaining the overall quality of the model's top ten reactant generation results, we additionally constructed a type of model integration strategy based on custom evaluation indicators. Specifically, we build a queue to store models and sort the stored models according to predefined evaluation indicators (such as Top-1 accuracy, weighted Top-k accuracy, etc.). Throughout the training process, we dynamically store candidate models and automatically generate ensemble models based on the top 3-5 in the queue, thereby retaining the Top-k models with the highest prediction quality. In the inference stage, we also rebuilt a beam search strategy based on the new framework that focuses more on search breadth to improve the overall quality of the results generated by the model Top-k
Requires a benchmark data set for dual-task experiments
The author conducted experiments in the retrosynthesis task and the forward reaction prediction task, using the data sets USPTO-50k and USPTO-MIT containing 50,000, 500,000, and 1 million chemical reaction data. , USPTO-full. In the experiment, the performance of the dual-task model and the single-task model were compared. According to the test results in Figure 4,
In the small-scale data set, BiG2S achieved leading prediction accuracy in the retrosynthesis task based on dual-task training, while also maintaining a high forward reaction prediction accuracy; however, In the USPTO-MIT data set that is biased towards forward response prediction and the large-scale data set USPTO-full, due to the limitation of the overall parameter amount of the model, the performance of the model after dual-task training has decreased. Nonetheless, the ability to simultaneously process the retrosynthesis task and the forward reaction prediction task was obtained from the dual-task model with almost the same number of parameters and a small reduction in response prediction ability (the absolute difference in Top-k accuracy is around 0.5%). From the perspective of capabilities, the BiG2S model has achieved the expected goals
Reanalyze the ablation experiment
The author further verified the new beam search algorithm and the optimal temperature hyperparameters of BiG2S when predicting in different data sets after using imbalance loss through ablation experiments. The temperature hyperparameter here refers to the temperature parameter T used in Softmax to control the output probability distribution. The experimental results are shown in Figure 5 and Figure 6
In the experiment on the beam search algorithm, it can be observed that OpenNMT expanded the search width to 3 times while the search time only expanded to 1.74 times, while the new When the top-1 accuracy of the beam search algorithm is consistent with OpenNMT, the overall search time increases by 1-2 times; but in terms of the quality of the top-10 prediction results, the new beam search algorithm has at least 3% improvement compared to OpenNMT. The absolute accuracy advantage and the effective molecule proportion advantage of 2%. It can be said that the new beam search algorithm has significantly improved the quality of the overall Top-k search results of the model at the cost of search time.
In terms of temperature When conducting experiments on hyperparameters, the researchers found that using larger temperature parameters on small-scale data sets can significantly improve the overall Top-k prediction accuracy. In larger data sets, since the BiG2S model size cannot fully adapt to all reaction data, choosing smaller temperature parameters at this time often helps model search
The conclusion of the study shows...
In this article, the author proposes a template-free reaction prediction model called BiG2S, which can handle both retrosynthesis tasks and forward reaction prediction tasks. By adopting appropriate parameter sharing strategies and additional dual-task labels, BiG2S can complete retrosynthesis tasks and reaction prediction tasks on data sets of different sizes with a smaller number of parameters, and its overall prediction ability is comparable to mainstream models
In order to solve the problem of uneven prediction difficulty of different SMILES characters and fluctuation of Top-k prediction accuracy in model training, the author introduced imbalance loss, automatic model integration strategy based on custom evaluation indicators and beam search algorithm based on new framework To alleviate these problems
BiG2S has shown good dual-task prediction capabilities on three mainstream data sets of different sizes, and further ablation experiments also proved the effectiveness of the additionally introduced training and inference strategies. sex
The above is the detailed content of End-to-end template-free response prediction model based on dual tasks. For more information, please follow other related articles on the PHP Chinese website!

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics

Written previously, today we discuss how deep learning technology can improve the performance of vision-based SLAM (simultaneous localization and mapping) in complex environments. By combining deep feature extraction and depth matching methods, here we introduce a versatile hybrid visual SLAM system designed to improve adaptation in challenging scenarios such as low-light conditions, dynamic lighting, weakly textured areas, and severe jitter. sex. Our system supports multiple modes, including extended monocular, stereo, monocular-inertial, and stereo-inertial configurations. In addition, it also analyzes how to combine visual SLAM with deep learning methods to inspire other research. Through extensive experiments on public datasets and self-sampled data, we demonstrate the superiority of SL-SLAM in terms of positioning accuracy and tracking robustness.

In modern manufacturing, accurate defect detection is not only the key to ensuring product quality, but also the core of improving production efficiency. However, existing defect detection datasets often lack the accuracy and semantic richness required for practical applications, resulting in models unable to identify specific defect categories or locations. In order to solve this problem, a top research team composed of Hong Kong University of Science and Technology Guangzhou and Simou Technology innovatively developed the "DefectSpectrum" data set, which provides detailed and semantically rich large-scale annotation of industrial defects. As shown in Table 1, compared with other industrial data sets, the "DefectSpectrum" data set provides the most defect annotations (5438 defect samples) and the most detailed defect classification (125 defect categories

The open LLM community is an era when a hundred flowers bloom and compete. You can see Llama-3-70B-Instruct, QWen2-72B-Instruct, Nemotron-4-340B-Instruct, Mixtral-8x22BInstruct-v0.1 and many other excellent performers. Model. However, compared with proprietary large models represented by GPT-4-Turbo, open models still have significant gaps in many fields. In addition to general models, some open models that specialize in key areas have been developed, such as DeepSeek-Coder-V2 for programming and mathematics, and InternVL for visual-language tasks.

For AI, Mathematical Olympiad is no longer a problem. On Thursday, Google DeepMind's artificial intelligence completed a feat: using AI to solve the real question of this year's International Mathematical Olympiad IMO, and it was just one step away from winning the gold medal. The IMO competition that just ended last week had six questions involving algebra, combinatorics, geometry and number theory. The hybrid AI system proposed by Google got four questions right and scored 28 points, reaching the silver medal level. Earlier this month, UCLA tenured professor Terence Tao had just promoted the AI Mathematical Olympiad (AIMO Progress Award) with a million-dollar prize. Unexpectedly, the level of AI problem solving had improved to this level before July. Do the questions simultaneously on IMO. The most difficult thing to do correctly is IMO, which has the longest history, the largest scale, and the most negative

Editor | ScienceAI Based on limited clinical data, hundreds of medical algorithms have been approved. Scientists are debating who should test the tools and how best to do so. Devin Singh witnessed a pediatric patient in the emergency room suffer cardiac arrest while waiting for treatment for a long time, which prompted him to explore the application of AI to shorten wait times. Using triage data from SickKids emergency rooms, Singh and colleagues built a series of AI models that provide potential diagnoses and recommend tests. One study showed that these models can speed up doctor visits by 22.3%, speeding up the processing of results by nearly 3 hours per patient requiring a medical test. However, the success of artificial intelligence algorithms in research only verifies this

Editor |KX To this day, the structural detail and precision determined by crystallography, from simple metals to large membrane proteins, are unmatched by any other method. However, the biggest challenge, the so-called phase problem, remains retrieving phase information from experimentally determined amplitudes. Researchers at the University of Copenhagen in Denmark have developed a deep learning method called PhAI to solve crystal phase problems. A deep learning neural network trained using millions of artificial crystal structures and their corresponding synthetic diffraction data can generate accurate electron density maps. The study shows that this deep learning-based ab initio structural solution method can solve the phase problem at a resolution of only 2 Angstroms, which is equivalent to only 10% to 20% of the data available at atomic resolution, while traditional ab initio Calculation

Editor |ScienceAI Question Answering (QA) data set plays a vital role in promoting natural language processing (NLP) research. High-quality QA data sets can not only be used to fine-tune models, but also effectively evaluate the capabilities of large language models (LLM), especially the ability to understand and reason about scientific knowledge. Although there are currently many scientific QA data sets covering medicine, chemistry, biology and other fields, these data sets still have some shortcomings. First, the data form is relatively simple, most of which are multiple-choice questions. They are easy to evaluate, but limit the model's answer selection range and cannot fully test the model's ability to answer scientific questions. In contrast, open-ended Q&A

Editor | KX In the field of drug research and development, accurately and effectively predicting the binding affinity of proteins and ligands is crucial for drug screening and optimization. However, current studies do not take into account the important role of molecular surface information in protein-ligand interactions. Based on this, researchers from Xiamen University proposed a novel multi-modal feature extraction (MFE) framework, which for the first time combines information on protein surface, 3D structure and sequence, and uses a cross-attention mechanism to compare different modalities. feature alignment. Experimental results demonstrate that this method achieves state-of-the-art performance in predicting protein-ligand binding affinities. Furthermore, ablation studies demonstrate the effectiveness and necessity of protein surface information and multimodal feature alignment within this framework. Related research begins with "S
