10 lines of code to complete the graph Transformer, the graph neural network framework DGL ushered in version 1.0

In 2019, New York University and Amazon Cloud Technology jointly launched the graph neural network framework DGL (Deep Graph Library). Now DGL 1.0 is officially released! DGL 1.0 summarizes the various needs for graph deep learning and graph neural network (GNN) technology in academia or industry in the past three years. From academic research on state-of-the-art models to scaling GNNs to industrial applications, DGL 1.0 provides a comprehensive and easy-to-use solution for all users to better take advantage of graph machine learning.

##DGL 1.0 for different scenarios solutions provided.

DGL 1.0 adopts a layered and modular design to meet various user needs. Key features of this release include:

• More than 100 out-of-the-box GNN model examples, more than 15 top-ranked ones on Open Graph Benchmark (OGB) Baseline model;
• More than 150 commonly used GNN modules, including GNN layers, data sets, graph data conversion modules, graph samplers, etc., which can be used to build new model architectures or GNN-based Solution;
• Flexible and efficient message passing and sparse matrix abstraction for developing new GNN modules;
• Multi-GPU and distributed Training capabilities support training on tens of billions of graphs.

#DGL 1.0 Technology stack diagram

##Address: https:/ /github.com/dmlc/dgl

One of the highlights of this version is the introduction of DGL-Sparse, a new programming interface that uses sparse matrices as the core programming abstraction. DGL-Sparse not only simplifies the development of existing GNN models such as graph convolutional networks, but also works with the latest models, including diffusion-based GNNs, hypergraph neural networks, and graph Transformers.

The release of DGL version 1.0 has aroused enthusiastic responses on the Internet. Scholars such as Yann Lecun, one of the three giants of deep learning, and Xavier Bresson, associate professor at the National University of Singapore, have all liked and forwarded it.

#In the following article, the author Two mainstream GNN paradigms are outlined, namely message passing view and matrix view. These paradigms can help researchers better understand the inner working mechanism of GNN, and the matrix perspective is also one of the motivations for the development of DGL Sparse.

Message passing view and matrix view

There is a saying in the movie "Arrival": "The language you use determines your The way you think and affects your view of things." This sentence also applies to GNN.

means that graph neural networks have two different paradigms. The first, called the message passing view, expresses the GNN model from a fine-grained, local perspective, detailing how messages are exchanged along edges and how node states are updated accordingly. The second is the matrix perspective. Since graphs have algebraic equivalence with sparse adjacency matrices, many researchers choose to express GNN models from a coarse-grained, global perspective, emphasizing operations involving sparse adjacency matrices and eigenvectors.

The message passing perspective reveals the connection between GNNs and the Weisfeiler Lehman (WL) graph isomorphism test, which also relies on aggregating information from neighbors. The matrix perspective understands GNN from an algebraic perspective, leading to some interesting discoveries, such as the over-smoothing problem.

In short, these two perspectives are indispensable tools for studying GNN. They complement each other and help researchers better understand and describe the nature and characteristics of GNN models. It is for this reason that one of the main motivations for the release of DGL 1.0 is to add support for the matrix perspective based on the existing message passing interface.

DGL Sparse: A sparse matrix library designed for graph machine learning

A new library called DGL Sparse has been added to DGL version 1.0 (dgl.sparse), together with the message passing interface in DGL, improves support for all types of graph neural network models. DGL Sparse provides sparse matrix classes and operations specifically for graph machine learning, making it easier to write GNNs from a matrix perspective. In the next section, the authors demonstrate several GNN examples, showing their mathematical formulations and corresponding code implementations in DGL Sparse.

Graph Convolutional Network

GCN is one of the pioneers of GNN modeling. GCN can be represented with both message passing view and matrix view. The following code compares the differences between these two methods in DGL.

#Implementing GCN using messaging API

##Use DGL Sparse to implement GCN

GNN based on graph diffusion

Graph diffusion is the process of spreading or smoothing node features or signals along edges. Many classic graph algorithms such as PageRank fall into this category. A series of studies have shown that combining graph diffusion with neural networks is an effective and efficient way to enhance model predictions. The following equation describes the core calculation of one of the more representative models, APPNP. It can be implemented directly in DGL Sparse.

##Hypergraph Neural Network

Hypergraph is a generalization of graph where edges can connect any number of nodes (called hyperedges). Hypergraphs are particularly useful in scenarios where higher-order relationships need to be captured, such as co-purchasing behavior in e-commerce platforms, or co-authorship in citation networks. A typical feature of a hypergraph is its sparse correlation matrix, so hypergraph neural networks (HGNN) are often defined using sparse matrices. The following is a hypergraph convolutional network (Feng et al., 2018) and its code implementation.

#Graph Transformer

Transformer model has become the most successful model architecture in natural language processing. Researchers are also beginning to extend Transformer to graph machine learning. Dwivedi et al. pioneered the idea of ​​limiting all multi-head attention to connected pairs of nodes in the graph. This model can be easily implemented with only 10 lines of code using the DGL Sparse tool.

Key Features of DGL Sparse

Compared with sparse matrix libraries such as scipy.sparse or torch.sparse, the overall design of DGL Sparse is to serve graph machine learning, which includes the following key features:

• Automatic sparse format selection: DGL Sparse is designed so that users do not have to worry about choosing the correct data structure to store sparse matrices (also known as sparse formats). Users only need to remember that dgl.sparse.spmatrix creates a sparse matrix, and DGL will automatically select the optimal format internally based on the operator called;
• # Scalar or vector non-zero elements : Many GNN models learn multiple weights on the edges (such as the multi-head attention vectors demonstrated in the Graph Transformer example). To accommodate this situation, DGL Sparse allows non-zero elements to have vector shapes and extends common sparse operations such as sparse-dense-matrix multiplication (SpMM), etc. You can refer to the bspmm operation in the Graph Transformer example.

By leveraging these design features, DGL Sparse reduced code length on average compared to the previous implementation of the matrix view model using a message passing interface. 2.7 times . The simplified code also reduces the framework's overhead by 43%. In addition, DGL Sparse is compatible with PyTorch and can be easily integrated with various tools and packages in the PyTorch ecosystem. Get started with DGL 1.0

DGL 1.0 has been released on all platforms and can be easily installed using pip or conda. In addition to the examples introduced earlier, the first version of DGL Sparse also includes 5 tutorials and 11 end-to-end examples, all of which can be experienced directly in Google Colab without the need for local installation.

To learn more about the new features of DGL 1.0, please refer to the author's release log. If you encounter any problems or have any suggestions or feedback while using DGL, you can also contact the DGL team through the Discuss forum or Slack.

The above is the detailed content of 10 lines of code to complete the graph Transformer, the graph neural network framework DGL ushered in version 1.0. For more information, please follow other related articles on the PHP Chinese website!