Universal Graph Transformer Self-Attention NetworksTwitter

GitHub top languageGitHub repo size GitHub last commit GitHub forks GitHub stars GitHub

This program provides the implementation of our graph transformer, named UGformer, as described in our paper, where we leverage the transformer self-attention network to learn graph representations.

Variant 1 Variant 2



  • 20-08-2021: Release a Pytorch implementation to apply the Variant 2 for inductive text classification.

  • 04-05-2021: Release a Pytorch 1.5.0 implementation (i.e., Variant 2) to leverage the transformer directly on the input graph.

  • 17-05-2020: Release a Pytorch 1.5.0 implementation.

  • 11-12-2019: Release a Tensorflow 1.14 implementation.


  • Variant 1: Leveraging the transformer on sampled neighbors of each node:

      $ python train_UGformerV1_Sup.py --dataset IMDBBINARY --batch_size 4 --ff_hidden_size 1024 --fold_idx 1 --num_neighbors 8 --num_epochs 50 --num_timesteps 4 --learning_rate 0.0005 --model_name IMDBBINARY_bs4_fold1_1024_8_idx0_4_1
  • Variant 2: Leveraging the transformer directly on all nodes of the input graph:

      $ python train_UGformerV2.py --dataset PTC --ff_hidden_size 1024 --fold_idx 1 --num_epochs 50 --num_timesteps 3 --learning_rate 0.0005 --model_name PTC_fold1_1024_idx0_1
  • Applying the Variant 2 for inductive text classification:

      $ python train_TextGNN.py --dataset mr --learning_rate 0.0001 --batch_size 4096 --num_epochs 150 --num_GNN_layers 2 --hidden_size 384 --model GatedGT


  • Python 3.x
  • Tensorflow 1.14 & Tensor2tensor 1.13
  • Pytorch >= 1.5.0
  • Networkx 2.3
  • Scikit-learn 0.21.2


Please cite the paper whenever our graph transformer is used to produce published results or incorporated into other software:

    author={Dai Quoc Nguyen and Tu Dinh Nguyen and Dinh Phung},
    title={Universal Graph Transformer Self-Attention Networks},
    journal={arXiv preprint arXiv:1909.11855},


As a free open-source implementation, Graph-Transformer is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. All other warranties including, but not limited to, merchantability and fitness for purpose, whether express, implied, or arising by operation of law, course of dealing, or trade usage are hereby disclaimed. I believe that the programs compute what I claim they compute, but I do not guarantee this. The programs may be poorly and inconsistently documented and may contain undocumented components, features or modifications. I make no guarantee that these programs will be suitable for any application.

Graph-Transformer is licensed under the Apache License 2.0.

Graph Transformer

Transformer for Graph Classification (Pytorch and Tensorflow)

Graph Transformer Info

⭐ Stars322
🔗 Source Codegithub.com
🕒 Last Update10 months ago
🕒 Created3 years ago
🐞 Open Issues2
➗ Star-Issue Ratio161
😎 Authordaiquocnguyen