Tian Han

2024

1. ICML

   Learning Latent Space Hierarchical EBM Diffusion Models

   Jiali Cui, and Tian Han

   In International Conference on Machine Learning (ICML), 2024

   Abs Bib PDF

   This work studies the learning problem of the energy-based prior model and the multi-layer generator model. The multi-layer generator model, which contains multiple layers of latent variables organized in a top-down hierarchical structure, typically assumes the Gaussian prior model. Such a prior model can be limited in modelling expressivity, which results in a gap between the generator posterior and the prior model, known as the prior hole problem. Recent works have explored learning the energy-based (EBM) prior model as a second-stage, complementary model to bridge the gap. However, the EBM defined on a multi-layer latent space can be highly multi-modal, which makes sampling from such marginal EBM prior challenging in practice, resulting in ineffectively learned EBM. To tackle the challenge, we propose to leverage the diffusion probabilistic scheme to mitigate the burden of EBM sampling and thus facilitate EBM learning. Our extensive experiments demonstrate a superior performance of our diffusion-learned EBM prior on various challenging tasks.

   @inproceedings{DBLP:conf/icml/Cui024,
     author = {Cui, Jiali and Han, Tian},
     title = {Learning Latent Space Hierarchical {EBM} Diffusion Models},
     booktitle = {International Conference on Machine Learning (ICML)},
     publisher = {OpenReview.net},
     year = {2024},
     url = {https://openreview.net/forum?id=o9uOuIwhZK},
     timestamp = {Mon, 02 Sep 2024 16:55:26 +0200},
     biburl = {https://dblp.org/rec/conf/icml/Cui024.bib},
     bibsource = {dblp computer science bibliography, https://dblp.org}
   }

2023

1. NeurIPS

   Learning Energy-based Model via Dual-MCMC Teaching

   Jiali Cui, and Tian Han

   In Advances in Neural Information Processing Systems (NeurIPS), 2023

   Abs Bib PDF

   This paper studies the fundamental learning problem of the energy-based model (EBM). Learning the EBM can be achieved using the maximum likelihood estimation (MLE), which typically involves the Markov Chain Monte Carlo (MCMC) sampling, such as the Langevin dynamics. However, the noise-initialized Langevin dynamics can be challenging in practice and hard to mix. This motivates the exploration of joint training with the generator model where the generator model serves as a complementary model to bypass MCMC sampling. However, such a method can be less accurate than the MCMC and result in biased EBM learning. While the generator can also serve as an initializer model for better MCMC sampling, its learning can be biased since it only matches the EBM and has no access to empirical training examples. Such biased generator learning may limit the potential of learning the EBM. To address this issue, we present a joint learning framework that interweaves the maximum likelihood learning algorithm for both the EBM and the complementary generator model. In particular, the generator model is learned by MLE to match both the EBM and the empirical data distribution, making it a more informative initializer for MCMC sampling of EBM. Learning generator with observed examples typically requires inference of the generator posterior. To ensure accurate and efficient inference, we adopt the MCMC posterior sampling and introduce a complementary inference model to initialize such latent MCMC sampling. We show that three separate models can be seamlessly integrated into our joint framework through two (dual-) MCMC teaching, enabling effective and efficient EBM learning.

   @inproceedings{CuiW023b,
     author = {Cui, Jiali and Han, Tian},
     booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
     editor = {Oh, A. and Naumann, T. and Globerson, A. and Saenko, K. and Hardt, M. and Levine, S.},
     pages = {28861--28872},
     publisher = {Curran Associates, Inc.},
     title = {Learning Energy-based Model via Dual-MCMC Teaching},
     volume = {36},
     year = {2023}
   }

2. CVPR

   Learning Joint Latent Space EBM Prior Model for Multi-layer Generator

   Jiali Cui, Ying Nian Wu, and Tian Han

   In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

   Abs Bib PDF Website

   This paper studies the fundamental problem of learning multi-layer generator models. The multi-layer generator model builds multiple layers of latent variables as a prior model on top of the generator, which benefits learning complex data distribution and hierarchical representations. However, such a prior model usually focuses on modeling inter-layer relations between latent variables by assuming non-informative (conditional) Gaussian distributions, which can be limited in model expressivity. To tackle this issue and learn more expressive prior models, we propose an energy-based model (EBM) on the joint latent space over all layers of latent variables with the multi-layer generator as its backbone. Such joint latent space EBM prior model captures the intra-layer contextual relations at each layer through layer-wise energy terms, and latent variables across different layers are jointly corrected. We develop a joint training scheme via maximum likelihood estimation (MLE), which involves Markov Chain Monte Carlo (MCMC) sampling for both prior and posterior distributions of the latent variables from different layers. To ensure efficient inference and learning, we further propose a variational training scheme where an inference model is used to amortize the costly posterior MCMC sampling. Our experiments demonstrate that the learned model can be expressive in generating high-quality images and capturing hierarchical features for better outlier detection.

   @inproceedings{CuiW023,
     author = {Cui, Jiali and Wu, Ying Nian and Han, Tian},
     title = {Learning Joint Latent Space {EBM} Prior Model for Multi-layer Generator},
     booktitle = {{IEEE/CVF} Conference on Computer Vision and Pattern Recognition (CVPR)},
     pages = {3603--3612},
     publisher = {{IEEE}},
     year = {2023},
     url = {https://doi.org/10.1109/CVPR52729.2023.00351},
     doi = {10.1109/CVPR52729.2023.00351},
     timestamp = {Mon, 28 Aug 2023 16:14:40 +0200},
     biburl = {https://dblp.org/rec/conf/cvpr/CuiW023.bib},
     bibsource = {dblp computer science bibliography, https://dblp.org}
   }

2022

1. NeurIPS

   Adaptive Multi-stage Density Ratio Estimation for Learning Latent Space Energy-based Model

   Zhisheng Xiao, and Tian Han

   In Advances in Neural Information Processing Systems (NeurIPS) , 2022

   Abs Bib PDF Poster

   This paper studies the fundamental problem of learning energy-based model (EBM) in the latent space of the generator model. Learning such prior model typically requires running costly Markov Chain Monte Carlo (MCMC). Instead, we propose to use noise contrastive estimation (NCE) to discriminatively learn the EBM through density ratio estimation between the latent prior density and latent posterior density. However, the NCE typically fails to accurately estimate such density ratio given large gap between two densities. To effectively tackle this issue and learn more expressive prior models, we develop the adaptive multi-stage density ratio estimation which breaks the estimation into multiple stages and learn different stages of density ratio sequentially and adaptively. The latent prior model can be gradually learned using ratio estimated in previous stage so that the final latent space EBM prior can be naturally formed by product of ratios in different stages. The proposed method enables informative and much sharper prior than existing baselines, and can be trained efficiently. Our experiments demonstrate strong performances in image generation and reconstruction as well as anomaly detection.

   @inproceedings{XiaoH22,
     author = {Xiao, Zhisheng and Han, Tian},
     title = {Adaptive Multi-stage Density Ratio Estimation for Learning Latent
                       Space Energy-based Model},
     booktitle = {Advances in Neural Information Processing Systems (NeurIPS) },
     year = {2022},
     timestamp = {Thu, 11 May 2023 17:08:21 +0200},
     biburl = {https://dblp.org/rec/conf/nips/XiaoH22.bib},
     bibsource = {dblp computer science bibliography, https://dblp.org}
   }

2020

1. NeurIPS

   Learning Latent Space Energy-Based Prior Model

   Bo Pang, Tian Han, Erik Nijkamp, Song-Chun Zhu, and Ying Nian Wu

   In Advances in Neural Information Processing Systems (NeurIPS) , 2020

   Abs Bib PDF Website

   We propose to learn energy-based model (EBM) in the latent space of a generator model, so that the EBM serves as a prior model that stands on the top-down network of the generator model. Both the latent space EBM and the top-down network can be learned jointly by maximum likelihood, which involves short-run MCMC sampling from both the prior and posterior distributions of the latent vector. Due to the low dimensionality of the latent space and the expressiveness of the top-down network, a simple EBM in latent space can capture regularities in the data effectively, and MCMC sampling in latent space is efficient and mixes well. We show that the learned model exhibits strong performances in terms of image and text generation and anomaly detection.

   @inproceedings{Pang0NZW20,
     author = {Pang, Bo and Han, Tian and Nijkamp, Erik and Zhu, Song{-}Chun and Wu, Ying Nian},
     editor = {Larochelle, Hugo and Ranzato, Marc'Aurelio and Hadsell, Raia and Balcan, Maria{-}Florina and Lin, Hsuan{-}Tien},
     title = {Learning Latent Space Energy-Based Prior Model},
     booktitle = {Advances in Neural Information Processing Systems (NeurIPS) },
     year = {2020},
     timestamp = {Tue, 19 Jan 2021 15:57:02 +0100},
     biburl = {https://dblp.org/rec/conf/nips/Pang0NZW20.bib},
     bibsource = {dblp computer science bibliography, https://dblp.org}
   }

2. CVPR

   Joint Training of Variational Auto-Encoder and Latent Energy-Based Model

   Tian Han, Erik Nijkamp, Linqi Zhou, Bo Pang, Song-Chun Zhu, and Ying Nian Wu

   In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2020

   Abs Bib PDF Code

   This paper proposes a joint training method to learn both the variational auto-encoder (VAE) and the latent energy-based model (EBM). The joint training of VAE and latent EBM are based on an objective function that consists of three Kullback-Leibler divergences between three joint distributions on the latent vector and the image, and the objective function is of an elegant symmetric and anti-symmetric form of divergence triangle that seamlessly integrates variational and adversarial learning. In this joint training scheme, the latent EBM serves as a critic of the generator model, while the generator model and the inference model in VAE serve as the approximate synthesis sampler and inference sampler of the latent EBM. Our experiments show that the joint training greatly improves the synthesis quality of the VAE. It also enables learning of an energy function that is capable of detecting out of sample examples for anomaly detection.

   @inproceedings{0001NZPZW20,
     author = {Han, Tian and Nijkamp, Erik and Zhou, Linqi and Pang, Bo and Zhu, Song{-}Chun and Wu, Ying Nian},
                       Model},
     booktitle = {{IEEE/CVF} Conference on Computer Vision and Pattern Recognition (CVPR) },
     pages = {7975--7984},
     publisher = {Computer Vision Foundation / {IEEE}},
     year = {2020},
     doi = {10.1109/CVPR42600.2020.00800},
     timestamp = {Tue, 31 Aug 2021 14:00:04 +0200},
     biburl = {https://dblp.org/rec/conf/cvpr/0001NZPZW20.bib},
     bibsource = {dblp computer science bibliography, https://dblp.org}
   }

2019

1. CVPR

   Divergence Triangle for Joint Training of Generator Model, Energy-Based Model, and Inferential Model

   Tian Han, Erik Nijkamp, Xiaolin Fang, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu

   In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019

   Abs Bib PDF Code

   This paper proposes the divergence triangle as a framework for joint training of generator model, energy-based model and inference model. The divergence triangle is a compact and symmetric (anti-symmetric) objective function that seamlessly integrates variational learning, adversarial learning, wake-sleep algorithm, and contrastive divergence in a unified probabilistic formulation. This unification makes the processes of sampling, inference, energy evaluation readily available without the need for costly Markov chain Monte Carlo methods. Our experiments demonstrate that the divergence triangle is capable of learning (1) an energy-based model with well-formed energy landscape, (2) direct sampling in the form of a generator network, and (3) feed-forward inference that faithfully reconstructs observed as well as synthesized data. The divergence triangle is a robust training method that can learn from incomplete data.

   @inproceedings{HanNFHZW19,
     author = {Han, Tian and Nijkamp, Erik and Fang, Xiaolin and Hill, Mitch and Zhu, Song{-}Chun and Wu, Ying Nian},
     title = {Divergence Triangle for Joint Training of Generator Model, Energy-Based
                       Model, and Inferential Model},
     booktitle = {{IEEE} Conference on Computer Vision and Pattern Recognition (CVPR)},
     pages = {8670--8679},
     publisher = {Computer Vision Foundation / {IEEE}},
     year = {2019},
     doi = {10.1109/CVPR.2019.00887},
     timestamp = {Tue, 15 Aug 2023 14:52:14 +0200},
     biburl = {https://dblp.org/rec/conf/cvpr/HanNFHZW19.bib},
     bibsource = {dblp computer science bibliography, https://dblp.org}
   }

2017

1. AAAI

   Alternating Back-Propagation for Generator Network

   Tian Han, Yang Lu, Song-Chun Zhu, and Ying Nian Wu

   In the Thirty-First AAAI Conference on Artificial Intelligence (AAAI), 2017

   Abs Bib PDF Website

   This paper proposes an alternating back-propagation algorithm for learning the generator network model. The model is a non-linear generalization of factor analysis. In this model, the mapping from the continuous latent factors to the observed signal is parametrized by a convolutional neural network. The alternating back-propagation algorithm iterates the following two steps: (1) Inferential back-propagation, which infers the latent factors by Langevin dynamics or gradient descent. (2) Learning back-propagation, which updates the parameters given the inferred latent factors by gradient descent. The gradient computations in both steps are powered by back-propagation, and they share most of their code in common. We show that the alternating back-propagation algorithm can learn realistic generator models of natural images, video sequences, and sounds. Moreover, it can also be used to learn from incomplete or indirect training data.

   @inproceedings{HanLZW17,
     author = {Han, Tian and Lu, Yang and Zhu, Song{-}Chun and Wu, Ying Nian},
     editor = {Singh, Satinder and Markovitch, Shaul},
     title = {Alternating Back-Propagation for Generator Network},
     booktitle = {the Thirty-First {AAAI} Conference on Artificial Intelligence (AAAI)},
     pages = {1976--1984},
     publisher = {{AAAI} Press},
     year = {2017},
     url = {https://doi.org/10.1609/aaai.v31i1.10902},
     doi = {10.1609/aaai.v31i1.10902},
     timestamp = {Mon, 04 Sep 2023 16:50:24 +0200},
     biburl = {https://dblp.org/rec/conf/aaai/HanLZW17.bib},
     bibsource = {dblp computer science bibliography, https://dblp.org}
   }