deepseek-v4

I have been too busy lately; I will complete this technical post about DeepSeek V4 when I have time.

Based on the technical report, its main new features appear to be:

1. Hybrid attention: CSA combined with HCA, a dense-attention mechanism after low-rank projection.
2. mHC (Manifold-Constrained Hyper-Connections).
3. Muon Optimizer

It also inherits MoE, MTP, and other features from V3.

1. hybrid attn

CSA

Before reading the report, I expected DeepSeek to retain DSA and combine it with linear attention or a small, relatively dense attention pattern. Instead, it introduces a new attention mechanism.

DSA already reduces cache pressure substantially, changing its size from $N \times n_{head} \times h$ to $N \times h_c$. The purpose of the new CSA is to reduce $N$ even further.

Compressed Sparse Attention combines compression and sparsity. Intuitively, several KV entries are compressed into one, while sparse selection chooses indices during attention computation.

2. mHC

Q: Why Introduce a New Residual Connection?

The paper says the goal is to strengthen the conventional residual connections between adjacent layers. As model depth increases, standard residual connections become increasingly sensitive: noise accumulates through the layers and may cause exploding gradients.

Pre-norm mitigates this to some extent by preserving gradient flow after normalization. But it introduces a new problem, an output deficiency: the contribution of $f(x)$ becomes diluted, and in deep networks the residual path approaches an identity transformation.

For a vivid, if imperfect, analogy, residual connections are roads joined end to end. An accident on one road creates congestion that gradually accumulates. Normalization is like placing a toll booth at every junction to limit traffic: congestion becomes less likely, but total throughput also falls sharply.

If one broad road does not work, could several smaller roads provide better residual connections?

Transformer blocks