Audio-Visual Event Localization in Unconstrained Videos (To appear in ECCV 2018)

Project ArXiv Demo Video

Watch the video

AVE Dataset & Features

AVE dataset can be downloaded from

Audio feature and visual feature (7.7GB) are also released. Please put videos of AVE dataset into /data/AVE folder and features into /data folder before running the code.

Scripts for generating audio and visual features: (Feel free to modify and use it to precess your audio and visual data).


Python-3.6, Pytorch-0.3.0, Keras, ffmpeg.

Visualize attention maps

Run: python to generate audio-guided visual attention maps.


Supervised audio-visual event localization


A+V-att model in the paper: python --model_name AV_att

DMRN model in the paper: python --model_name DMRN


python --model_name AV_att --train

Weakly-supervised audio-visual event localization

We add some videos without audio-visual events into training data. Therefore, the labels of these videos are background. Processed visual features can be found in visual_feature_noisy.h5. Put the feature into data folder.


W-A+V-att model in the paper: python


python --train

Cross-modality localization

For this task, we developed a cross-modal matching network. Here, we used visual feature vectors via global average pooling, and you can find here. Please put the feature into data folder. Note that the code was implemented via Keras-2.0 with Tensorflow as the backend.





[1] Rouditchenko, Andrew, et al. "Self-supervised Audio-visual Co-segmentation." ICASSP, 2019. [Paper]

[2] Lin, Yan-Bo, Yu-Jhe Li, and Yu-Chiang Frank Wang. "Dual-modality seq2seq network for audio-visual event localization." ICASSP, 2019 [Paper]

[3] Rana, Aakanksha, Cagri Ozcinar, and Aljosa Smolic. "Towards Generating Ambisonics Using Audio-visual Cue for Virtual Reality." ICASSP, 2019. [Paper]

[4] Yu Wu, Linchao Zhu, Yan Yan, Yi Yang. "Dual Attention Matching for Audio-Visual Event Localization", ICCV, 2019. (oral) [website]

[5] Jinxing Zhou, Liang Zheng, Yiran Zhong, Shijie Hao, Meng Wang. Positive Sample Propagation along the Audio-Visual Event Line, CVPR 2021. [paper][code]


If you find this work useful, please consider citing it.

  author={Tian, Yapeng and Shi, Jing and Li, Bochen and Duan, Zhiyao and Xu, Chenliang},
  title={Audio-Visual Event Localization in Unconstrained Videos},
  booktitle = {ECCV},
  year = {2018}


Audio features are extracted using vggish and the audio-guided visual attention model was implemented highly based on adaptive attention. We thank the authors for sharing their codes. If you use our codes, please also cite their nice works.

Ave Eccv18

Audio-Visual Event Localization in Unconstrained Videos, ECCV 2018

Ave Eccv18 Info

⭐ Stars 111
🔗 Homepage
🔗 Source Code
🕒 Last Update 8 days ago
🕒 Created 4 years ago
🐞 Open Issues 5
➗ Star-Issue Ratio 22
😎 Author YapengTian