Positional Encoding

Visualizing the sinusoidal waves that give Transformers a sense of order.

PE_{(pos, 2i)} = \sin(pos / 10000^{2i/d_{model}})

PE_{(pos, 2i+1)} = \cos(pos / 10000^{2i/d_{model}})

The wavelength increases geometrically from $2\pi$ to $10000 \cdot 2\pi$ . This allows the model to easily learn to attend by relative positions.

Sequence Length (L)100

10500

Embedding Dimension (d_model)256

161024

Loading Visualization...

Hover over the heatmap to see exact values.