Table of Contents
Introduction
Home Technology peripherals AI Effectively integrate language models, graph neural networks, and text graph training framework GLEM to achieve new SOTA

Effectively integrate language models, graph neural networks, and text graph training framework GLEM to achieve new SOTA

Apr 11, 2023 pm 01:28 PM
AI train

Effectively integrate language models, graph neural networks, and text graph training framework GLEM to achieve new SOTA

  • Main units: Montreal Algorithm Learning Artificial Intelligence Laboratory (Mila), Microsoft Research Asia, etc.
  • Paper address: https://arxiv.org/abs/2210.14709
  • Code address: https://github.com /andyjzhao/glem

Introduction

Effectively integrate language models, graph neural networks, and text graph training framework GLEM to achieve new SOTA

##Figure 1: (a) Text graph (b) Graph neural network (c) Language model

Graph is a universal data structure that models the structural relationship between nodes. In real life, many nodes contain rich text features, and this graph is called a text-attributed graph [2]. For example, the paper citation network contains the text of the paper and the citation relationship between the papers; the social network contains the user's text description and the user's direct interactive relationship. The representation learning model on text graphs can be applied to tasks such as node classification and link prediction, and has wide application value.

Text graph contains two aspects of information: text information of nodes and graph structure information between nodes. The modeling of traditional text graphs can be divided into two perspectives: text modeling and graph modeling. Among them, the text modeling method (shown in Figure 1.b) usually uses a Transformer-based language model (LM) to obtain the text representation of a single node and predict the target task; the modeling method of graph modeling ( As shown in Figure 1.c), a graph neural network (GNN) is usually used to model the interaction between node features and predict target tasks through a message propagation mechanism.

However, the two models can only model the text and graph structure in the text graph respectively: the traditional language model cannot directly consider the structural information, and the graph neural network cannot directly consider the original text information. Modeling. In order to model text and graph structures at the same time, researchers try to integrate language models and graph neural networks and update the parameters of the two models simultaneously. However, existing work [2, 3] cannot model a large number of neighbor texts at the same time, has poor scalability, and cannot be applied to large text graphs.

GLEM framework

In order to more effectively integrate graph neural networks and language models, this article proposes Graph and L anguage Learning by Expectation Maximization (GLEM) framework. The GLEM framework is based on the variational expectation maximum algorithm (Variational EM) and alternately learns graph neural networks and language models, thus achieving good scalability.

Effectively integrate language models, graph neural networks, and text graph training framework GLEM to achieve new SOTA

Figure 2: GLEM framework

Specifically, taking the node classification task as an example, in the E step, GLEM trains the language model based on the real labels and the pseudo labels predicted by the graph neural network; in the M step , GLEM trains a graph neural network based on real labels and pseudo-labels predicted by the language model . In this way, the GLEM framework effectively mines local textual information and global structural interaction information. Both graph neural networks (GLEM-GNN) and language models (GLEM-LM) trained through the GLEM framework can be used to predict node labels.

Experiment

The experimental part of the paper mainly discusses the GLEM framework from the following aspects:

  • Effectiveness: The GLEM model can effectively integrate graph neural networks and language models, significantly improving both models. The GLEM framework achieved first place on three text graph node classification tasks at OGB.
  • Scalability: By alternately training graph neural networks and language models, the GLEM framework can train large language models and deep GNNs simultaneously.
  • Structure-free inductive reasoning ability: The traditional GNN model performs poorly when facing new nodes without graph structure. In contrast, GLEM-LM enables efficient inference using only textual features (without graph structure).
  • Model convergence: GLEM uses the EM iteration algorithm, and can converge in one EM iteration on some data sets.

Effectively integrate language models, graph neural networks, and text graph training framework GLEM to achieve new SOTA

Figure 3: The GLEM framework won first place on the OGBN-arxiv, products, papers100M data set

The above is the detailed content of Effectively integrate language models, graph neural networks, and text graph training framework GLEM to achieve new SOTA. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Repo: How To Revive Teammates
4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Bytedance Cutting launches SVIP super membership: 499 yuan for continuous annual subscription, providing a variety of AI functions Bytedance Cutting launches SVIP super membership: 499 yuan for continuous annual subscription, providing a variety of AI functions Jun 28, 2024 am 03:51 AM

This site reported on June 27 that Jianying is a video editing software developed by FaceMeng Technology, a subsidiary of ByteDance. It relies on the Douyin platform and basically produces short video content for users of the platform. It is compatible with iOS, Android, and Windows. , MacOS and other operating systems. Jianying officially announced the upgrade of its membership system and launched a new SVIP, which includes a variety of AI black technologies, such as intelligent translation, intelligent highlighting, intelligent packaging, digital human synthesis, etc. In terms of price, the monthly fee for clipping SVIP is 79 yuan, the annual fee is 599 yuan (note on this site: equivalent to 49.9 yuan per month), the continuous monthly subscription is 59 yuan per month, and the continuous annual subscription is 499 yuan per year (equivalent to 41.6 yuan per month) . In addition, the cut official also stated that in order to improve the user experience, those who have subscribed to the original VIP

Context-augmented AI coding assistant using Rag and Sem-Rag Context-augmented AI coding assistant using Rag and Sem-Rag Jun 10, 2024 am 11:08 AM

Improve developer productivity, efficiency, and accuracy by incorporating retrieval-enhanced generation and semantic memory into AI coding assistants. Translated from EnhancingAICodingAssistantswithContextUsingRAGandSEM-RAG, author JanakiramMSV. While basic AI programming assistants are naturally helpful, they often fail to provide the most relevant and correct code suggestions because they rely on a general understanding of the software language and the most common patterns of writing software. The code generated by these coding assistants is suitable for solving the problems they are responsible for solving, but often does not conform to the coding standards, conventions and styles of the individual teams. This often results in suggestions that need to be modified or refined in order for the code to be accepted into the application

Seven Cool GenAI & LLM Technical Interview Questions Seven Cool GenAI & LLM Technical Interview Questions Jun 07, 2024 am 10:06 AM

To learn more about AIGC, please visit: 51CTOAI.x Community https://www.51cto.com/aigc/Translator|Jingyan Reviewer|Chonglou is different from the traditional question bank that can be seen everywhere on the Internet. These questions It requires thinking outside the box. Large Language Models (LLMs) are increasingly important in the fields of data science, generative artificial intelligence (GenAI), and artificial intelligence. These complex algorithms enhance human skills and drive efficiency and innovation in many industries, becoming the key for companies to remain competitive. LLM has a wide range of applications. It can be used in fields such as natural language processing, text generation, speech recognition and recommendation systems. By learning from large amounts of data, LLM is able to generate text

Can fine-tuning really allow LLM to learn new things: introducing new knowledge may make the model produce more hallucinations Can fine-tuning really allow LLM to learn new things: introducing new knowledge may make the model produce more hallucinations Jun 11, 2024 pm 03:57 PM

Large Language Models (LLMs) are trained on huge text databases, where they acquire large amounts of real-world knowledge. This knowledge is embedded into their parameters and can then be used when needed. The knowledge of these models is "reified" at the end of training. At the end of pre-training, the model actually stops learning. Align or fine-tune the model to learn how to leverage this knowledge and respond more naturally to user questions. But sometimes model knowledge is not enough, and although the model can access external content through RAG, it is considered beneficial to adapt the model to new domains through fine-tuning. This fine-tuning is performed using input from human annotators or other LLM creations, where the model encounters additional real-world knowledge and integrates it

Kuaishou version of Sora 'Ke Ling' is open for testing: generates over 120s video, understands physics better, and can accurately model complex movements Kuaishou version of Sora 'Ke Ling' is open for testing: generates over 120s video, understands physics better, and can accurately model complex movements Jun 11, 2024 am 09:51 AM

What? Is Zootopia brought into reality by domestic AI? Exposed together with the video is a new large-scale domestic video generation model called "Keling". Sora uses a similar technical route and combines a number of self-developed technological innovations to produce videos that not only have large and reasonable movements, but also simulate the characteristics of the physical world and have strong conceptual combination capabilities and imagination. According to the data, Keling supports the generation of ultra-long videos of up to 2 minutes at 30fps, with resolutions up to 1080p, and supports multiple aspect ratios. Another important point is that Keling is not a demo or video result demonstration released by the laboratory, but a product-level application launched by Kuaishou, a leading player in the short video field. Moreover, the main focus is to be pragmatic, not to write blank checks, and to go online as soon as it is released. The large model of Ke Ling is already available in Kuaiying.

To provide a new scientific and complex question answering benchmark and evaluation system for large models, UNSW, Argonne, University of Chicago and other institutions jointly launched the SciQAG framework To provide a new scientific and complex question answering benchmark and evaluation system for large models, UNSW, Argonne, University of Chicago and other institutions jointly launched the SciQAG framework Jul 25, 2024 am 06:42 AM

Editor |ScienceAI Question Answering (QA) data set plays a vital role in promoting natural language processing (NLP) research. High-quality QA data sets can not only be used to fine-tune models, but also effectively evaluate the capabilities of large language models (LLM), especially the ability to understand and reason about scientific knowledge. Although there are currently many scientific QA data sets covering medicine, chemistry, biology and other fields, these data sets still have some shortcomings. First, the data form is relatively simple, most of which are multiple-choice questions. They are easy to evaluate, but limit the model's answer selection range and cannot fully test the model's ability to answer scientific questions. In contrast, open-ended Q&A

SOTA performance, Xiamen multi-modal protein-ligand affinity prediction AI method, combines molecular surface information for the first time SOTA performance, Xiamen multi-modal protein-ligand affinity prediction AI method, combines molecular surface information for the first time Jul 17, 2024 pm 06:37 PM

Editor | KX In the field of drug research and development, accurately and effectively predicting the binding affinity of proteins and ligands is crucial for drug screening and optimization. However, current studies do not take into account the important role of molecular surface information in protein-ligand interactions. Based on this, researchers from Xiamen University proposed a novel multi-modal feature extraction (MFE) framework, which for the first time combines information on protein surface, 3D structure and sequence, and uses a cross-attention mechanism to compare different modalities. feature alignment. Experimental results demonstrate that this method achieves state-of-the-art performance in predicting protein-ligand binding affinities. Furthermore, ablation studies demonstrate the effectiveness and necessity of protein surface information and multimodal feature alignment within this framework. Related research begins with "S

Five schools of machine learning you don't know about Five schools of machine learning you don't know about Jun 05, 2024 pm 08:51 PM

Machine learning is an important branch of artificial intelligence that gives computers the ability to learn from data and improve their capabilities without being explicitly programmed. Machine learning has a wide range of applications in various fields, from image recognition and natural language processing to recommendation systems and fraud detection, and it is changing the way we live. There are many different methods and theories in the field of machine learning, among which the five most influential methods are called the "Five Schools of Machine Learning". The five major schools are the symbolic school, the connectionist school, the evolutionary school, the Bayesian school and the analogy school. 1. Symbolism, also known as symbolism, emphasizes the use of symbols for logical reasoning and expression of knowledge. This school of thought believes that learning is a process of reverse deduction, through existing

See all articles