LLM-training / Algorithm Basics

After doing some training work recently, I realized that I had started to forget some of the fundamentals of LLMs.

— Revised and expanded on May 6, 2026

Loss

Loss is one of the most fundamental concepts. It measures the distance between a model’s output and its target, and falls mainly into two categories:

• Classification loss (discrete): when the desired result is discrete—for example, choosing one or k results from the final vocabulary using an LLM’s output logits—a classification loss is required.
  • Cross entropy, for example, measures the difference between distributions.
  • KL divergence can serve as a metric for distillation: $D_{KL}(P \parallel Q) = \sum_x P(x) \log \frac{Q(x)}{P(x)}$
• Regression loss (continuous): when the desired result is a continuous scalar or vector, a regression loss measures its “distance” from the target.
  • MSE (Mean Squared Error): the mean of the L2 loss, based on squared differences.
  • MAE (Mean Absolute Error): the mean of the L1 loss.

Optimize

Starting with basic convex optimization: although the many nonlinear layers in deep learning destroy convexity, many ideas from convex optimization remain useful, including gradient descent and Newton’s method.

Gradient descent simply and effectively moves in the direction opposite to the gradient. The gradient only provides a direction, so we must choose a step size—the learning rate, or lr: $\theta = \theta_{0} - \eta \nabla f_{\theta}$.

Newton’s method additionally introduces second-order curvature information.

Optimizer

An optimizer uses the gradient obtained through backpropagation ($\nabla \theta$) to decide how to update the model parameters $\theta$.

The following are several common optimizers and their mathematical principles, where $\eta$ is the learning rate and $g_t$ is the gradient.

1. SGD (Stochastic Gradient Descent)

This is the most basic optimizer, and probably one of the earliest. It moves directly in the direction opposite to the gradient.

• Mathematical principle:

  $\theta_{t+1} = \theta_t - \eta \cdot g_t$

  Here, $\eta$ is the learning rate and $g_t$ is the gradient at the current step.

• Advantages and disadvantages: it is simple, but because each batch is used to compute a gradient, the update path can zigzag severely. It may also become trapped in a local optimum or at a saddle point.

2. SGD with Momentum

To reduce SGD’s oscillation, momentum introduces the idea of “inertia”: it considers not only the current gradient but also the previous direction of movement.

• Mathematical principle:

  1. Compute the momentum term: $v_t = \gamma v_{t-1} + \eta g_t$

  2. Update the parameters: $\theta_{t+1} = \theta_t - v_t$

     Here, $\gamma$ (usually 0.9) is the momentum coefficient.

• Intuition: when the current direction of movement agrees with the gradient, the momentum term adds the previous direction on top of the gradient, effectively providing a bonus. When the two disagree, the optimizer is oscillating, so momentum acts as a penalty that suppresses unnecessary back-and-forth movement.

---

3. AdaGrad (Adaptive Gradient Algorithm)

AdaGrad introduces an adaptive learning rate. Its core idea is to decrease the learning rate for frequently updated parameters and increase it for infrequently updated parameters.

• Mathematical principle:

  $\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{G_t + \epsilon}} \cdot g_t$

  Here, $G_t = \sum_{\tau=1}^t g_\tau^2$ is the cumulative sum of all past squared gradients.

• Advantages and disadvantages: it works well with sparse data. However, because $G_t$ continually accumulates positive values, the denominator grows and the effective gradient coefficient rapidly decays toward zero, which can halt training prematurely at later stages.

4. RMSProp

RMSProp was proposed to address AdaGrad’s vanishing learning rate. Instead of accumulating every historical gradient, it uses an exponential moving average (EMA) to retain primarily recent gradient information.

• Mathematical principle:

  1. Compute the moving average of squared gradients: $E[g^2]_t = \beta E[g^2]_{t-1} + (1-\beta)g_t^2$

  2. Update the parameters: $\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{E[g^2]_t + \epsilon}} \cdot g_t$

     This allows the optimizer to “forget” gradients from the distant past and maintain a stable rate of descent.

---

5. Adam (Adaptive Moment Estimation)

Adam is currently the most common optimizer for LLM training. It combines the advantages of Momentum (the first moment, which controls direction) and RMSProp (the second moment, which controls step size).

• Mathematical principle:

  1. First moment (momentum): $m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t$

  2. Second moment (scaling): $v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2$

  3. Bias correction: because $m_t$ and $v_t$ are initialized to zero, they are biased toward zero early in training and must be corrected:

     $\hat{m}_t = \frac{m_t}{1-\beta_1^t}, \quad \hat{v}_t = \frac{v_t}{1-\beta_2^t}$

  4. Parameter update: $\theta_{t+1} = \theta_t - \eta \cdot \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$

---

6. AdamW (Adam with Weight Decay)

This is an important correction to Adam. In Adam, L2 regularization (weight decay) is improperly scaled by the second moment $v_t$, weakening the intended regularization effect.

• Core improvement: AdamW decouples weight decay from the gradient update and applies it directly to the parameters $\theta$.
• Current practice: AdamW is the default for Transformer models and mainstream large-model pretraining and fine-tuning.