Gradient descent

Purpose: Minimize a loss function $C(w)$ by iteratively updating parameters $w$ (weights/biases) in the direction of the negative gradient.

Update Rule:

• $\alpha$: Learning rate (step size).
• ${\mathrm{\nabla }}_{w}C(w)$: Gradient of loss w.r.t. $w$.

Variants:

1. Batch GD: Uses full dataset (slow, precise).
2. Stochastic GD (SGD): One random sample per step (fast, noisy).
3. Mini-batch SGD: Compromise (common in deep learning).

Key Idea: Follow the steepest descent to find a (local) minimum.

Challenge: Choosing $\alpha$ (too small → slow; too large → overshoot).
Advanced Optimizers (Adam, RMSprop) adapt $\alpha$ dynamically.