Generative Models (GANs, VAEs)

Generative models learn to generate new data that resembles the training data. Unlike discriminative models that learn the boundary between classes, generative models learn the underlying distribution of the data itself.

Discriminative vs Generative Models

Aspect	Discriminative Models	Generative Models
Goal	Learn P(y	x) — the probability of a label given input
Output	A class label or value	New data samples (images, text, audio)
Examples	Logistic regression, CNNs, SVMs	GANs, VAEs, diffusion models
Analogy	A critic who judges art	An artist who creates art

Autoencoders

An autoencoder learns to compress data into a low-dimensional representation (the latent space) and then reconstruct it. While not directly generative, autoencoders are the foundation for variational autoencoders.

Architecture

Input → [Encoder] → Latent Code (z) → [Decoder] → Reconstructed Input

import torch.nn as nn

class Autoencoder(nn.Module):
    def __init__(self, input_dim=784, latent_dim=32):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(input_dim, 256),
            nn.ReLU(),
            nn.Linear(256, latent_dim),
        )
        self.decoder = nn.Sequential(
            nn.Linear(latent_dim, 256),
            nn.ReLU(),
            nn.Linear(256, input_dim),
            nn.Sigmoid(),
        )

    def forward(self, x):
        z = self.encoder(x)
        reconstructed = self.decoder(z)
        return reconstructed

Variational Autoencoders (VAEs)

A Variational Autoencoder extends the autoencoder by making the latent space continuous and structured. Instead of encoding each input to a single point, the encoder outputs the mean and variance of a probability distribution, and the latent code is sampled from this distribution.

Key Differences from Standard Autoencoders

Aspect	Autoencoder	VAE
Latent space	Unstructured, discrete points	Structured, continuous distribution
Sampling	Deterministic encoding	Stochastic — samples from a distribution
Loss function	Reconstruction loss only	Reconstruction loss + KL divergence
Generation	Cannot reliably generate new samples	Can generate new samples by sampling from the latent space

VAE in PyTorch

import torch
import torch.nn as nn

class VAE(nn.Module):
    def __init__(self, input_dim=784, latent_dim=32):
        super().__init__()
        # Encoder
        self.encoder = nn.Sequential(
            nn.Linear(input_dim, 256),
            nn.ReLU(),
        )
        self.fc_mu = nn.Linear(256, latent_dim)
        self.fc_logvar = nn.Linear(256, latent_dim)
        # Decoder
        self.decoder = nn.Sequential(
            nn.Linear(latent_dim, 256),
            nn.ReLU(),
            nn.Linear(256, input_dim),
            nn.Sigmoid(),
        )

    def encode(self, x):
        h = self.encoder(x)
        return self.fc_mu(h), self.fc_logvar(h)

    def reparameterise(self, mu, logvar):
        std = torch.exp(0.5 * logvar)
        eps = torch.randn_like(std)
        return mu + eps * std

    def decode(self, z):
        return self.decoder(z)

    def forward(self, x):
        mu, logvar = self.encode(x)
        z = self.reparameterise(mu, logvar)
        return self.decode(z), mu, logvar

VAE Loss Function

def vae_loss(reconstructed, original, mu, logvar):
    # Reconstruction loss (how well the decoder reconstructs the input)
    recon_loss = nn.functional.binary_cross_entropy(
        reconstructed, original, reduction='sum'
    )
    # KL divergence (how close the latent distribution is to a standard normal)
    kl_loss = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    return recon_loss + kl_loss

Generative Adversarial Networks (GANs)

GANs (Goodfellow et al., 2014) consist of two neural networks that compete against each other in a game:

Generative Models