Regularization-Based Robust Training

Adversarial training is the most common approach to building robust networks: train on adversarially perturbed examples to learn invariance. But adversarial training is expensive (requires solving inner maximization for each batch), can hurt clean accuracy, and doesn’t provide certified guarantees during training.

Regularization-based training offers an alternative: add penalty terms to the loss that encourage robustness properties—small Lipschitz constants, large margins, smooth decision boundaries. These regularizers don’t require generating adversarial examples during training, provide clearer optimization objectives, and often integrate naturally with certified verification methods.

This guide explores regularization-based approaches to robust training: what robustness regularizers exist, how they compare to adversarial training, and when they’re the right choice.

Why Regularization for Robustness?

Standard training minimizes empirical risk on clean examples:

\min_\theta \mathbb{E}_{(x,y) \sim \mathcal{D}} [\mathcal{L}(f_\theta(x), y)]

This says nothing about behavior on perturbed inputs. Two strategies add robustness:

Adversarial training: Augment training with worst-case perturbations:

\min_\theta \mathbb{E}_{(x,y)} \left[ \max_{\|\delta\|_p \leq \epsilon} \mathcal{L}(f_\theta(x + \delta), y) \right]

Regularization-based: Add penalties encouraging robustness properties:

\min_\theta \mathbb{E}_{(x,y)} [\mathcal{L}(f_\theta(x), y)] + \lambda R(f_\theta)

where $R(f_\theta)$ is a regularizer (e.g., Lipschitz constant, smoothness).

Key Differences

Adversarial Training:

Trains on worst-case perturbations

Requires solving inner maximization (expensive)

Implicit robustness (through data augmentation)

No certified guarantees during training

Regularization-Based:

Penalizes network properties directly

Closed-form or efficiently computable penalties

Explicit robustness (through regularization term)

Can integrate with certified bounds

Types of Robustness Regularizers

Lipschitz Regularization

Idea: Constrain the Lipschitz constant $L_f$ to limit sensitivity to input perturbations.

Regularizer: Penalize large Lipschitz constant:

R_{\text{Lip}}(f_\theta) = L_f^2 = \left( \sup_x \|\nabla f_\theta(x)\|_2 \right)^2

Approximations: Computing the exact Lipschitz constant is hard. Practical approaches:

Spectral normalization: Normalize weight matrices by spectral norm (Yoshida and Miyato 2017):

W_{\text{normalized}} = \frac{W}{\|W\|_2}

This ensures each layer has Lipschitz constant at most 1, giving $L_f \leq 1$ for the network.

Gradient penalty: Penalize large gradients at training points:

R_{\text{grad}}(f_\theta) = \mathbb{E}_x \|\nabla_x f_\theta(x)\|_2^2

This encourages small local Lipschitz constant near training data.

Benefits: Networks with small Lipschitz constants have:

Certified robustness radius $\epsilon_{\text{cert}} = \Delta / L_f$
Better generalization (bounded complexity)
Smoother decision boundaries

Implementation:

import torch
import torch.nn as nn
from torch.nn.utils import spectral_norm

# Spectral normalization (hard constraint)
class RobustNetwork(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = spectral_norm(nn.Linear(784, 256))
        self.fc2 = spectral_norm(nn.Linear(256, 10))

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        return self.fc2(x)

# Gradient penalty (soft constraint)
def gradient_penalty(model, x):
    x.requires_grad = True
    output = model(x)
    gradients = torch.autograd.grad(
        outputs=output, inputs=x,
        grad_outputs=torch.ones_like(output),
        create_graph=True
    )[0]
    penalty = (gradients.norm(2, dim=1) ** 2).mean()
    return penalty

# Training loop
for x, y in dataloader:
    loss_task = criterion(model(x), y)
    loss_reg = gradient_penalty(model, x)
    loss_total = loss_task + lambda_reg * loss_reg

    optimizer.zero_grad()
    loss_total.backward()
    optimizer.step()

Margin-Based Regularization

Idea: Maximize the classification margin—the distance from decision boundary to training examples.

Regularizer: Penalize small margins:

R_{\text{margin}}(f_\theta) = -\mathbb{E}_{(x,y)} \left[ f_\theta(x)_y - \max_{c \neq y} f_\theta(x)_c \right]

Larger margin leads to more confident correct classification, which is harder to flip with perturbations.

Connection to robustness: For a network with Lipschitz constant $L_f$ , certified radius is:

\epsilon_{\text{cert}} = \frac{\text{margin}}{L}

Maximizing margin while minimizing $L$ improves certified robustness.

Cross-entropy vs margin: Standard cross-entropy doesn’t explicitly maximize margin. Adding margin regularization or using margin-based losses (e.g., hinge loss) encourages larger separability.

Smoothness Regularization

Idea: Encourage smooth network outputs—small second derivatives (Hessian).

Curvature penalty: Penalize large curvature:

R_{\text{smooth}}(f_\theta) = \mathbb{E}_x \|\nabla^2 f_\theta(x)\|_F^2

where $\nabla^2 f$ is the Hessian matrix.

Why it helps: Smooth functions change slowly, making them robust to small perturbations. High curvature means rapid output change, leading to vulnerability.

Computational challenge: Computing and penalizing the Hessian is expensive (quadratic in dimension). Approximations:

Finite differences: Approximate curvature via output changes:

R_{\text{smooth}} \approx \mathbb{E}_{x, \delta} \|f_\theta(x + \delta) - f_\theta(x) - \nabla f_\theta(x)^T \delta\|^2

Random projection: Project Hessian onto random directions for efficient estimation.

Jacobian Regularization

Jacobian Frobenius norm: Penalize large Jacobian $J = \nabla_x f_\theta(x)$ :

R_{\text{Jac}}(f_\theta) = \mathbb{E}_x \|J(x)\|_F^2 = \mathbb{E}_x \sum_{i,j} \left( \frac{\partial f_\theta(x)_i}{\partial x_j} \right)^2

Relationship to Lipschitz: For $\ell_2$ norm, $L_f = \sup_x \|J(x)\|_2$ (spectral norm of Jacobian). Frobenius norm is an upper bound: $\|J\|_2 \leq \|J\|_F$ .

Efficient computation: Jacobian regularization can be computed via backpropagation without explicit Jacobian construction:

def jacobian_regularization(model, x):
    """Compute Frobenius norm of Jacobian."""
    x.requires_grad = True
    output = model(x)

    jac_penalty = 0
    for i in range(output.shape[1]):  # For each output dimension
        grad = torch.autograd.grad(
            output[:, i].sum(), x,
            create_graph=True, retain_graph=True
        )[0]
        jac_penalty += (grad ** 2).sum()

    return jac_penalty / (output.shape[0] * output.shape[1])

Comparison with Adversarial Training

Adversarial Training (PGD-AT)

Standard PGD (Madry et al. 2018) adversarial training:

\min_\theta \mathbb{E}_{(x,y)} \left[ \max_{\|\delta\|_\infty \leq \epsilon} \mathcal{L}(f_\theta(x + \delta), y) \right]

Inner maximization: Solve for worst-case perturbation via PGD:

\delta^{(t+1)} = \Pi_{\|\delta\|_\infty \leq \epsilon} \left( \delta^{(t)} + \alpha \cdot \text{sign}(\nabla_\delta \mathcal{L}(f_\theta(x + \delta^{(t)}), y)) \right)

Pros:

Strong empirical robustness, often among the best-performing empirical defenses
Directly optimizes worst-case objective
Well-studied, mature techniques

Cons:

Expensive: an inner maximization per training step, against none for a plain objective
Accuracy-robustness tradeoff, and how large it is depends on the dataset, the architecture and $\epsilon$ — large enough to be worth measuring rather than assuming
No certified guarantees (only empirical robustness)
Unstable training (requires careful tuning)

Regularization-Based Training

Lipschitz/gradient regularization:

\min_\theta \mathbb{E}_{(x,y)} [\mathcal{L}(f_\theta(x), y)] + \lambda \mathbb{E}_x \|\nabla f_\theta(x)\|^2

Pros:

Fast (no inner maximization, just gradient penalty)
Better clean accuracy (smaller accuracy-robustness tradeoff)
Certified robustness (via Lipschitz bounds)
Stable training (convex regularization term)

Cons:

Weaker empirical robustness than PGD-AT (adversarial accuracy typically lower)
Requires tuning regularization strength $\lambda$
Certified bounds may be loose (conservative)

Comparing the three approaches. The columns below are the ones that can be filled in without a benchmark run, and the reason each is filled in qualitatively is that the obvious alternative is a number without a configuration behind it. Training cost depends on the depth and the hardware; the accuracy cost depends on the dataset, the architecture and $\epsilon$ . A figure quoted without those is not a claim a reader can check, so what follows is ordered and categorised instead:

Method	Relative training cost	What kind of robustness	What can fail
Standard	1x	none	nothing to certify; accuracy is undefended
PGD-AT	several times standard	empirical, attack-dependent	a stronger attack than the one trained against
Lipschitz / gradient regularization	slightly above standard	certified, from a Lipschitz bound	the bound is conservative and may certify little
Certified adversarial training	above both	certified, from a relaxation bound	the relaxation is loose, so the certificate is weak

Two columns carry most of the information, and for different reasons.

Cost is ordered by what each objective requires, which is why the ordering holds without a benchmark. A method with no inner maximization is cheapest because there is no second problem to solve per step. PGD-AT is not, because it solves one. The certified methods sit above both because they compute bounds during training as well as gradients.

Guarantee kind is categorical rather than graded, and a table that ranked it on a scale would be wrong. A certificate here comes from a Lipschitz bound or from a relaxation bound, and “no attack was found” is not a weaker version of either — it is a different kind of statement that does not become a certificate with enough searching. The last column is therefore about what each guarantee can fail at, which is the part a practitioner is actually choosing between: an empirical defense fails to a stronger attack, a Lipschitz certificate fails by being conservative about the same network it got right.

What genuinely cannot be stated without running something: the clean-accuracy cost of each method relative to undefended training, and the adversarial accuracy at a fixed $\epsilon$ and attack budget. Those are the numbers a reader should ask a paper for, and a table that supplies them without a configuration is worse than no table, because it looks like an answer.

Hybrid Approaches: TRADES

TRADES (Zhang et al. 2019) (Trade-off between Robustness and Accuracy) combines adversarial training with regularization:

\min_\theta \mathbb{E}_{(x,y)} \left[ \mathcal{L}(f_\theta(x), y) + \lambda \max_{\|\delta\| \leq \epsilon} D_{\text{KL}}(f_\theta(x) \| f_\theta(x + \delta)) \right]

Key insight: Separate natural accuracy (first term) from robustness (second term). The parameter $\lambda$ controls the tradeoff.

Advantages over PGD-AT:

Better natural accuracy (explicit natural loss term)
More stable training (KL divergence smoother than cross-entropy)
Tunable robustness-accuracy tradeoff (via $\lambda$ )

Relationship to regularization: The KL divergence term $D_{\text{KL}}(f(x) \| f(x + \delta))$ penalizes output sensitivity—similar spirit to Lipschitz/gradient regularization but applied to worst-case perturbations.

Implementation:

def trades_loss(model, x, y, epsilon=0.031, alpha=0.007, num_steps=10, beta=6.0):
    """TRADES loss combining natural and robust objectives."""
    # Natural loss
    logits_natural = model(x)
    loss_natural = F.cross_entropy(logits_natural, y)

    # Generate adversarial examples
    x_adv = x.detach() + 0.001 * torch.randn_like(x)
    for _ in range(num_steps):
        x_adv.requires_grad = True
        with torch.enable_grad():
            logits_adv = model(x_adv)
            loss_kl = F.kl_div(
                F.log_softmax(logits_adv, dim=1),
                F.softmax(logits_natural, dim=1),
                reduction='batchmean'
            )
        grad = torch.autograd.grad(loss_kl, x_adv)[0]
        x_adv = x_adv.detach() + alpha * grad.sign()
        x_adv = torch.min(torch.max(x_adv, x - epsilon), x + epsilon)
        x_adv = torch.clamp(x_adv, 0.0, 1.0)

    # Robust loss (KL divergence)
    logits_adv = model(x_adv)
    loss_robust = F.kl_div(
        F.log_softmax(logits_adv, dim=1),
        F.softmax(logits_natural, dim=1),
        reduction='batchmean'
    )

    return loss_natural + beta * loss_robust

Practical Considerations

Choosing Regularization Strength

The regularization parameter $\lambda$ controls the robustness-accuracy tradeoff:

Too small: Insufficient robustness, close to standard training
Too large: Over-regularization, poor accuracy

Heuristics for tuning:

Grid search: Try $\lambda \in \{0.001, 0.01, 0.1, 1.0, 10.0\}$ , select based on validation accuracy and certified radius
Warmup: Start with $\lambda = 0$ , gradually increase during training (annealing schedule)
Adaptive: Adjust $\lambda$ based on current margin/robustness (increase if under-regularized, decrease if over-regularized)

Combining Multiple Regularizers

Different regularizers capture different robustness aspects. Combining them can improve overall robustness:

R_{\text{total}} = \lambda_1 R_{\text{Lip}} + \lambda_2 R_{\text{margin}} + \lambda_3 R_{\text{smooth}}

Example: Lipschitz regularization + margin maximization ensures both small sensitivity (Lipschitz) and large separation (margin).

Challenge: More hyperparameters to tune. Start with single regularizers, add others if needed.

Integration with Certified Training

Regularization integrates naturally with certified training methods:

IBP (Gowal et al. 2019) training: Compute bounds via interval propagation, maximize worst-case correct margin:

\min_\theta \mathbb{E}_{(x,y)} \left[ \mathcal{L}_{\text{IBP}}(f_\theta, x, y, \epsilon) \right] + \lambda R_{\text{Lip}}(f_\theta)

The Lipschitz regularization tightens IBP bounds, improving certified accuracy.

CROWN (Zhang et al. 2018) training: Use CROWN bounds for certified loss + Lipschitz regularization.

When to Use Regularization-Based Training

Use regularization when:

Clean accuracy is important (minimal accuracy drop acceptable)
Want certified robustness guarantees (Lipschitz-based certification)
Training budget is limited (faster than adversarial training)
Network architecture benefits from regularization (e.g., GANs with spectral normalization)
Deploying in settings where certified guarantees are valued

Use adversarial training when:

Empirical robustness is paramount (need highest adversarial accuracy)
Willing to accept accuracy-robustness tradeoff
Computational budget allows expensive training
Certification not required (only need empirical robustness)

Use hybrid (TRADES) when:

Want balance between natural and robust accuracy
Need tunability (adjust $\lambda$ for desired tradeoff)
Willing to pay moderate training cost (between regularization and full PGD-AT)

Complementary Approaches

Regularization and adversarial training aren’t mutually exclusive:

Start with regularization (Lipschitz, margin) for good initialization

Fine-tune with adversarial training for empirical robustness

Use TRADES to balance both objectives

This combines certified guarantees (regularization) with strong empirical robustness (adversarial training).

Current Research Directions

Tighter regularization: Developing regularizers that more directly correspond to certified robustness (e.g., tightening Lipschitz constant estimation).

Architecture-aware regularization: Exploiting specific architectures (CNNs, transformers) for more efficient regularization.

Learned regularization: Using meta-learning to automatically tune regularization strengths or learn custom regularizers.

Scalable certified training: Combining regularization with scalable certified bounds (IBP, CROWN) for certified training on large networks.

Multi-task regularization: Regularizing for multiple robustness properties (adversarial, natural distribution shift, fairness) simultaneously.

Limitations

Weaker empirical robustness: Regularization-based methods typically achieve lower adversarial accuracy than PGD adversarial training on strong attacks.

Loose certified bounds: Lipschitz-based certification can be conservative, especially for deep networks where product bounds accumulate.

Hyperparameter sensitivity: Performance depends on regularization strength $\lambda$ ; requires careful tuning.

Not defense against all attacks: Regularization helps against $\ell_p$ perturbations but may not defend against other attack types (e.g., semantic, physical).

Key Takeaways

Regularization-based training provides an elegant alternative to adversarial training: instead of augmenting data with adversarial examples, directly encourage robustness properties through regularization. This approach is faster, maintains better clean accuracy, and integrates naturally with certified verification methods.

While regularization alone may not achieve the empirical robustness of intensive adversarial training, it offers a better accuracy-robustness-cost tradeoff for many applications. Methods like TRADES show that combining both philosophies—explicit regularization and adversarial examples—yields the best of both worlds.

Understanding regularization-based training clarifies the relationship between network properties (Lipschitz constant, margin, smoothness) and robustness. This perspective guides both training (what to regularize) and verification (what bounds to expect), providing a unified framework for building and certifying robust networks.

Next Phase

That is the end of Phase 3: Practice. Phase 4 takes up what happens when the network is too large, too structured, or too exposed for the methods here to certify, and what you can still say about it anyway.

Regularization-Based Robust Training

What you will be able to do

Assumes you have read