1. Generative Adversarial Networks (GANs)
James Engelmann
CS 6890 – Deep Learning
April 12, 2018
Generative Adversarial Networks

2. Generative Adversarial Networks
Generative Modeling
What is the point?
To generate examples that could belong to the training set but are new
Generative Adversarial Networks (GANs)
Variational Autoencoders (VAEs)
Generative Adversarial Networks

3. Variational Autoencoder (VAE)
Unsupervised Learning
Trained to reproduce input data, 𝑥
𝑥 is mapped into latent variable 𝑧 (innate features) by Encoder network
Reparameterization Trick
Allows for gradient Backpropagation
Sample from distribution generated from latent feature representation of 𝑥
𝑥 𝑧 is the input to the Decoder network
The Decoder network produces 𝑥
log 𝑃 𝑥 − 𝐷 𝐾𝐿 (𝑄(𝑧|𝑥)| 𝑃 𝑧 𝑥 = 𝔼 𝑧~𝑄 log 𝑃 𝑥 𝑧 − 𝐷 𝐾𝐿 (𝑄(𝑧|𝑥)| 𝑃(𝑧)
𝑝 𝑥 𝑧 𝑧 =Ɲ( μ 𝑧 , Σ 𝑧 )
Lower bound on log 𝑃 𝑥 ‼!
Reference: Tutorial on Variational Autoencoders
Generative Adversarial Networks

4. Variational Autoencoder (VAE)
Advantages:
Quality of the model can be evaluated (log-likelihood)
Easier to train than GANs
Disadvantages:
Results in lower (than state-of-the-art) quality in reproduced images
Trained to maximize lower bound of likelihood
Models can create images close to training datasets probability distribution
Doesn’t mean they look like training examples for complex images
Reference: Tutorial on Variational Autoencoders
Generative Adversarial Networks

5. Generative Adversarial Networks
Original paper: Generative Adversarial Nets by Goodfellow et al.
Team from the University of Montreal
Presented at NIPS in 2014
Two deep neural networks: Generator and Discriminator
Uses “adversarial nets” framework
Trained to play a minimax game
Generator tries to fool Discriminator
Discriminator tries to determine
whether samples are real or fake
Generative Adversarial Networks

6. Generative Adversarial Networks
The Generator
The Generator’s job is to fool the Discriminator
Produce fake samples, 𝐺(𝑧), from noise, 𝑧, the Discriminator, 𝐷 𝑥 can’t distinguish from real samples, 𝑥
During training:
Input is random noise samples z from probability distribution, 𝑝 𝑧
Use SGD or other gradient update algorithm like Adam to update parameters, 𝜃 𝐺 with ∇ 𝜃 𝐺 𝐽 𝐺 (𝑊,𝑏)
Outputs fake samples, 𝐺(𝑧) with
probability distribution 𝑝 𝑔
Minimax GAN
In practice Minimax GAN saturates quickly, Non-Saturating GAN alleviates that problem
𝐽 𝐺 (𝑊,𝑏) = 1 𝑚 𝑖=1 𝑚 log 1 −𝐷 𝐺 𝑧 𝑖 (𝑀𝐺𝐴𝑁) − log (𝐷 𝐺 𝑧 𝑖 (𝑁𝑆𝐺𝐴𝑁)
Non-Saturating GAN
Reference: Generative Adversarial Nets
Generative Adversarial Networks

7. Generative Adversarial Networks
The Discriminator
The Discriminator’s job is detect fake samples
Determine probability input is real or fake
During training:
Input is both fake samples, 𝐺(𝑧), and real samples, 𝑥, from data probability distribution, 𝑝 𝑑𝑎𝑡𝑎
Update parameters 𝜃 𝐷 by gradient ascent with ∇ 𝜃 𝐷 𝐽 𝐷 (𝑊,𝑏)
Output is probability 𝐷(𝑥)
𝐽 𝐷 (𝑊,𝑏)= 1 𝑚 𝑖=1 𝑚 log 𝐷 𝑥 𝑖 − log 1 −𝐷 𝐺 𝑧 𝑖
Reference: Generative Adversarial Nets
Generative Adversarial Networks

8. Generative Adversarial Networks
The Game GANs Play
The Generator wants 𝐷(𝐺(𝑧)) to be high
The Discriminator wants 𝐷(𝐺(𝑧)) to be low
Generator and Discriminator play minimax game with value function 𝑉(𝐷,𝐺):
Type equation here.
GANs are trained to approximate
this relationship:
𝑚𝑖𝑛 𝐺 𝑚𝑎𝑥 𝐷 𝑉 𝐷,𝐺 = 𝔼 𝑥~ 𝑝 𝑑𝑎𝑡𝑎 (𝑥) log 𝐷 𝑥 + 𝔼 𝑧~ 𝑝 𝑧 (𝑥) [log⁡(1−𝐷 𝐺 𝑧 )]
𝑝 𝑔 (𝑧)= 𝑝 𝑑𝑎𝑡𝑎 (𝑥)
Reference: Generative Adversarial Nets
Generative Adversarial Networks

9. Generative Adversarial Networks
The Algorithm
for # of training epochs do
for k steps do
Sample minibatch of m noise samples {𝑧 1 , 𝑧 (2) ,…, 𝑧 𝑚 } from distribution 𝑝 𝑧 (𝑧)
Sample minibatch of m data examples {𝑥 1 , 𝑥 (2) ,…, 𝑥 𝑚 } from distribution 𝑝 𝑑𝑎𝑡𝑎 (𝑥)
Update the discriminator through gradient ascent:
end for
Update the generator through gradient descent:
Reference: Generative Adversarial Nets
k is a hyperparameter
Updating the Discriminator before the Generator necessary to avoid Mode Collapse
Gradient updates can be done through any gradient-based learning rule
∇ 𝜃 𝐷 𝑚 𝑖=1 𝑚 log 𝐷 𝑥 𝑖 − log 1 −𝐷 𝐺 𝑧 𝑖
∇ 𝜃 𝐺 𝑚 𝑖=1 𝑚 log 1 −𝐷 𝐺 𝑧 𝑖 (𝑀𝐺𝐴𝑁) − log (𝐷 𝐺 𝑧 𝑖 ) 𝑁𝑆𝐺𝐴𝑁
Generative Adversarial Networks

10. Generative Adversarial Networks
Pros:
State-of-the-art Image Generation
Gradients calculated through Backpropagation
Cons:
No explicit representation of generator distribution, 𝑝 𝑔 (𝑥)
𝐺 and 𝐷 need to be synchronously trained to avoid Mode Collapse
Mode Collapse happens when the Generator finds a “crack” in the Discriminator’s armor and continues to attack the weakness
Produces similar outputs with little variation between examples or between features in examples
Unstable/Difficult to train
Generative Adversarial Networks

11. Generative Adversarial Networks
GAN trained on MNIST
GAN trained on CIFAR-10 with
fully connected
architecture
The five frames on the left of each box are generated examples.
The yellow frame is the nearest training example to the sample on its left.
Reference: Generative Adversarial Nets
GAN trained on CIFAR-10 with
Convolutional
Discriminator
And “Deconvolutional”
Generator
GAN trained on TFD
Generative Adversarial Networks

12. Generative Adversarial Networks
Upgrade: DCGAN
Original paper: Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks by Alec Radford, Luke Metz (indigo Research) and Soumith Chintala (Facebook AI Research)
Conference paper at ICLR 2016
Contributions:
Deep Convolutional Generative Adversarial Network (DCGAN) architecture
Originally proposed in Generative Adversarial Nets but architecture wasn’t discussed
Continued GAN training optimization
Visualized what kernels/filters learned
Showed Discriminator to be near state-of-the-art classifier
Showed interesting vector arithmetic properties of the Generator
Generative Adversarial Networks

13. Generative Adversarial Networks
DCGAN: Architecture
The Generator is a “DeConvnet” which uses transposed convolution and max value “switches” to perform spatial up-sampling
Vector Arithmetic is done with Generated example’s representation
(embedding)
in the 𝑧 space
The Discriminator is a standard CNN using strided convolutions for spatial down-sampling
Reference: Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
Generative Adversarial Networks

14. DCGAN: Convolutional Layers
DCGAN – CNN architecture specs: (USES ORIGINAL TRAINING ALGORITHM)
Use “All-Convolutional” Nets: No pooling operations for (down/up) sampling
Eliminate fully-connected layers at the end of convolutional layers
Apply Batch Normalization
DON’T apply to Generator output layer or Discriminator input layer
Activation Functions
Uses ReLU throughout Generator except Tanh at output
Use Leaky ReLU throughout Discriminator (especially for high-res applications)
Reference: Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
Generative Adversarial Networks

15. DCGAN for Classification?
Radford, Metz, and Chintala’s DCGAN trained on Imagenet-1k dataset
Discriminator’s convolutional features (all layers) down-sampled through max-pooling to 4x4 grid
4x4 grid flattened, concatenated into 28,672 dimensional vector and fed into L2-SVM for classification
CIFAR-10 classification results
Model
Accuracy
Accuracy (400 per class)
Max # of features units
Layer K-means
80.6%
63.7% (± 0.7%)
4800
Layer K-means Learned RF
82.0%
70.7% (± 0.7%)
3200
View Invariant K-means
81.9%
72.6% (± 0.7%)
6400
Exemplar CNN
84.3%
77.4% (± 0.2%)
1024
DCGAN + L2-SVM (R,M,C)
82.8%
73.8% (± 0.4%)
512
Reference: Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
Generative Adversarial Networks

16. DCGAN: Vector Arithmetic
Generated images are classified by the human eye
“Smiling Woman”
“Neutral Man”
The Generator input vector 𝑧, that created these images can be thought of (after training) as the latent variables that represent generated image traits/features
The human classified examples are stored as their latent representation 𝑧
The latent representations are averaged by category
Produces the latent representation, 𝑧, of the “average” “smiling woman”, etc.
Averaged latent representations are passed into the Generator to generate the image representation of the model’s “average” “smiling woman” etc.
Reference: Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
Generative Adversarial Networks

17. DCGAN: Vector Arithmetic
Demonstrated it’s possible to perform arithmetic in the latent “𝑧 space”
Uniform noise added to average categorical representation to yield variable results shown to the right
Because the input to the Generator is random noise (latent “𝑧 space”) this shows that the model has learned a representation of “man” and “woman” inside
Reference: Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
Generative Adversarial Networks

18. Generative Adversarial Networks
DCGAN: LSUN Bedrooms
After one pass through the training set!
Reference: Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
Generative Adversarial Networks

19. Welcome to the GAN Jungle
Reference: The GAN Zoo on GitHub - Hindu Puravinash ~ 300 GAN Papers!!!! Mostly GAN variations
3D-ED-GAN - Shape Inpainting using 3D Generative Adversarial Network and Recurrent Convolutional Networks
3D-GAN - Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling (github)
3D-IWGAN - Improved Adversarial Systems for 3D Object Generation and Reconstruction (github)
3D-RecGAN - 3D Object Reconstruction from a Single Depth View with Adversarial Learning (github)
ABC-GAN - ABC-GAN: Adaptive Blur and Control for improved training stability of Generative Adversarial Networks(github)
ABC-GAN - GANs for LIFE: Generative Adversarial Networks for Likelihood Free Inference
AC-GAN - Conditional Image Synthesis With Auxiliary Classifier GANs
acGAN - Face Aging With Conditional Generative Adversarial Networks
ACtuAL - ACtuAL: Actor-Critic Under Adversarial Learning
AdaGAN - AdaGAN: Boosting Generative Models
AdvGAN - Generating adversarial examples with adversarial networks
AE-GAN - AE-GAN: adversarial eliminating with GAN
AEGAN - Learning Inverse Mapping by Autoencoder based Generative Adversarial Nets
AffGAN - Amortised MAP Inference for Image Super-resolution
AL-CGAN - Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts
ALI - Adversarially Learned Inference (github)
AlignGAN - AlignGAN: Learning to Align Cross-Domain Images with Conditional Generative Adversarial Networks
AM-GAN - Activation Maximization Generative Adversarial Nets
AnoGAN - Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery
APE-GAN - APE-GAN: Adversarial Perturbation Elimination with GAN
ARAE - Adversarially Regularized Autoencoders for Generating Discrete Structures (github)
ARDA - Adversarial Representation Learning for Domain Adaptation
ARIGAN - ARIGAN: Synthetic Arabidopsis Plants using Generative Adversarial Network
ArtGAN - ArtGAN: Artwork Synthesis with Conditional Categorial GANs
AttGAN - Arbitrary Facial Attribute Editing: Only Change What You Want
AttnGAN - AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
TV-GAN - TV-GAN: Generative Adversarial Network Based Thermal to Visible Face Recognition
UGACH - Unsupervised Generative Adversarial Cross-modal Hashing
UGAN - Enhancing Underwater Imagery using Generative Adversarial Networks
Unim2im - Unsupervised Image-to-Image Translation with Generative Adversarial Networks (github)
Unrolled GAN - Unrolled Generative Adversarial Networks (github)
VAE-GAN - Autoencoding beyond pixels using a learned similarity metric
VariGAN - Multi-View Image Generation from a Single-View
VAW-GAN - Voice Conversion from Unaligned Corpora using Variational Autoencoding Wasserstein Generative Adversarial Networks
VEEGAN - VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning (github)
VGAN - Generating Videos with Scene Dynamics (github)
VGAN - Generative Adversarial Networks as Variational Training of Energy Based Models (github)
VGAN - Text Generation Based on Generative Adversarial Nets with Latent Variable
ViGAN - Image Generation and Editing with Variational Info Generative Adversarial Networks
VIGAN - VIGAN: Missing View Imputation with Generative Adversarial Networks
VoiceGAN - Voice Impersonation using Generative Adversarial Networks
VRAL - Variance Regularizing Adversarial Learning
WaterGAN - WaterGAN: Unsupervised Generative Network to Enable Real-time Color Correction of Monocular Underwater Images
WaveGAN - Synthesizing Audio with Generative Adversarial Networks
weGAN - Generative Adversarial Nets for Multiple Text Corpora
WGAN - Wasserstein GAN (github)
WGAN-GP - Improved Training of Wasserstein GANs (github)
WS-GAN - Weakly Supervised Generative Adversarial Networks for 3D Reconstruction
XGAN - XGAN: Unsupervised Image-to-Image Translation for many-to-many Mappings
ZipNet-GAN - ZipNet-GAN: Inferring Fine-grained Mobile Traffic Patterns via a Generative Adversarial Neural Network
α-GAN - Variational Approaches for Auto-Encoding Generative Adversarial Networks (github)
Δ-GAN - Triangle Generative Adversarial Networks
Generative Adversarial Networks

20. Generative Adversarial Networks
Comparing GANs
GANs provide objectively good images (especially under direct comparison to other generative models like VAEs)
How do we compare GAN performance between models?
No way to compute probability 𝑝 𝑔 𝑥 , can’t compute log-likelihood
Original paper: Are GANs Created Equal? A Large-Scale Study by Lucic et al. at Google Brain
Contributions:
Comparison of state-of-the-art GANs
Empirical evidence showing comparing GAN comparison needs summary of results not just best
Assess Fréchet Inception Distance (FID) for GAN comparison
Their Code on GitHub
Generative Adversarial Networks

21. Comparing GANs: IS and FID
Inception Score (IS):
Use pretrained Inception v3 network trained on ImageNet-1k , calculate the score on 𝐺 𝑧
Well correlated with human perception but has issues, A Note on the Inception Score
Fréchet Inception Distance (FID):
𝐺 𝑧 /𝑥 embedded by specific layer in Inception v3
Embedding layer viewed as multivariate Gaussian
Mean and covariance for layer with real data (𝜇 𝑥 , Σ 𝑥 ) and generated data (𝜇 𝑔 , Σ 𝑔 )
Consistent with humans, more robust to noise than IS, and can detect intra-class mode dropping
Negative correlation between FID and visual quality
𝐼𝑆 𝐺(𝑧) =exp⁡( 𝔼 𝑥~ 𝑝 𝑔 [ 𝐷 𝐾𝐿 𝑝 𝑦 𝑥 𝑝 𝑦 )
𝐹𝐼𝐷 𝑥,𝑔 = 𝜇 𝑥 − 𝜇 𝑔 𝑇𝑟 Σ 𝑥 + Σ 𝑔 −2 Σ 𝑥 Σ 𝑔
Reference: Are GANs Created Equal? A Large-Scale Study
Generative Adversarial Networks

22. Generative Adversarial Networks
Comparing WHICH GANs?
MGAN/NSGAN- Many Paths to Equilibrium: GANs Do Not Need To Decrease A Divergence At Every Step
WGAN - Wasserstein GANs
Uses Earth-Mover (EM) Distance as Loss, clipped to enforce 𝐺(𝑧) to be 1-Lipschitz function
WGAN GP - Improved Training of Wasserstein GANs
Uses Gradient Penalty (GP) instead of clipping weights
LSGAN - Least Squares Generative Adversarial Networks
Discriminator uses Least Square Loss Function
DRAGAN - On Convergence and Stability of GANs
Deep Regret Analytic GANs – Uses Gradient Penalty scheme
BEGAN - BEGAN: Boundary Equilibrium Generative Adversarial Networks
Includes proportionality control variable, 𝑘, in Discriminator loss
Reference: Are GANs Created Equal? A Large-Scale Study
Generative Adversarial Networks

23. Comparing GANs: Fairness
How did they compare so many different GANs while keeping it fair?
Architecture kept the same for all models
Four Test Sets
MNIST, FASHION-MNIST, CIFAR-10, CelebA
Hyperparameters searches were random and done for each dataset and also inferred across models
Random seed was varied
Comparisons were made across computational budget range
THIS WAS VERY IMPORTANT!!
Reference: Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
Generative Adversarial Networks

24. Comparing GANs: The Results
Table shows the best FID obtained during their hyperparameter search for each data set.
Testing procedure:
Search for hyperparameters
Select best model
Re-run training of best model with 50 different seeds
Report mean FID, standard deviation
Excluding outliers
No one GAN dominates
MNIST
FASHION
CIFAR
CELEBA
MGAN
9.8 ± 0.9
29.6 ± 1.6
72.7 ± 3.6
65.6 ± 4.2
NSGAN
6.8 ± 0.5
26.5 ± 1.6
58.5 ± 1.9
55.0 ± 3.3
LSGAN
7.8 ± 0.6
30.7 ± 2.2
87.1 ± 47.5
53.9 ± 2.8
WGAN
6.7 ± 0.4
21.5 ± 1.6
55.2 ± 2.3
41.3 ± 2.0
WGAN GP
20.3 ± 5.0
24.5 ± 2.1
55.8 ± 0.9
30.0 ± 1.0
DRAGAN
7.6 ± 0.4
27.7 ± 1.2
69.8 ± 2.0
42.3 ± 3.0
BEGAN
13.1 ± 1.0
22.9 ± 0.9
71.4 ± 1.6
38.9 ± 0.9
VAE
23.8 ± 0.6
58.7 ± 1.2
155.7 ± 11.6
85.7 ± 3.8
Average FID for each model trained on each dataset
Reference: Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
Generative Adversarial Networks

25. Comparing GANs: Conclusions
It is necessary to report distributions of FID for a fixed computational budget
As computational budget increases a “bad” algorithm can outperform a “good” algorithm
Different models have different precision and recall abilities
No empirical evidence to suggest an algorithm superior to original GANs
NSGAN obtained one of the lowest FID on MNIST, and best F1 score on TRIANGLES dataset
Reference: Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
Generative Adversarial Networks

26. Summary and GANclusions
Generator and Discriminator (sometimes called Critic)
Generator wants to produce samples that fool the Discriminator
Generator wants to learn real data distribution
Discriminator wants to catch the fakes
Original concept played minimax game during training
DCGANs sparked GAN growth
The ever long search for optimization
Training GANs is tough!
Instability
Mode Collapse
FID: Useful GAN comparison
Different GANs may have
advantages but no one is best
𝑝 𝑔 (𝑧)= 𝑝 𝑑𝑎𝑡𝑎 (𝑥)
For training tips and tricks: GAN Hacks (GitHub)
Generative Adversarial Networks

27. Pretty Pictures from GANs
BEGAN trained on CelebA - BEGAN: Boundary Equilibrium Generative Adversarial Networks
Generative Adversarial Networks

28. Pretty Pictures from GANs
MMGAN trained on CelebA - MMGAN: Manifold-Matching Generative Adversarial Network
Generative Adversarial Networks

29. Pretty Pictures from GANs
LSGAN - Least Squares Generative Adversarial Networks
Generative Adversarial Networks

30. Generative Adversarial Networks
My Figures
Training
Testing 𝑥 𝑥 𝑥 𝑧 𝑧 𝜇𝑧 𝑥
𝑥 𝑧 Σ𝑧
Encoder
Decoder
Generative Adversarial Networks

31. Generative Adversarial Networks
My Figures
Generated samples, 𝐺(𝑧) -
...
...
...
...
...
... 1
𝐺(𝑧)
𝐷(𝑥)
𝐺(𝑧) 𝑧
FAKE
REAL
Random noise, 𝑧 𝑥
Data samples, x
Generator
Discriminator
Generative Adversarial Networks