Understanding positional encodings

“Interestingly”, standard dot-product attention mechanism¹ is unaware of element’s positions within a sequence: Outputs are a weighted average of inputs according to their inter-similarity. Unless encoding positional information somehow, we could forward the input sequence in a permuted order and the output would be the same. Order, however, is a key piece of information for sequential data².

¹ The heart 💙 of the transformer architecture

² An unordered sentence, video or audio makes no sense!

Today, we look at the two most influential ways of encoding positional information into the transformer architecture:

• Sinusoidal embeddings [2017]: The original approach presented in Attention is all you need.

• Rotary Position Embedding (ROPE) [2021]: Presented in the RoFormer paper and now widely adopted by most modern architectures: DeepSeek-v3, Qwen, Llama-3…

Before we start, let’s review a couple of obvious approaches that might seem to do the trick good-enough, but don’t work well in practice. This is a nice exercise to surface some desirable properties of positional encodings.

Tip 1: Naive approaches and why they don’t work

At a first glance, “appending positional information” seems like a trivial task. One might ask:

🤔 How about appending each element’s sequence index to the input as an extra feature?

• Large numbers: The sequence length can be very long, making these indexes become very large. This value disparity with other inputs can make learning harder, usually one wants more-or-less normalized inputs.
• Harder generalization: At inference-time, the model might see sentences with different lengths (larger values) than on the training set, making predictions unstable.

🤔 How about instead of appending the index, linearly splitting the 0-1 interval and appending each corresponding value to each input?

• The main issue with this approach is time-step inconsistency across different input length: An input of 10 elements has a step-size of $0.1$, while an input of 20 elements has $0.05$.

Ok, so it seems it won’t be so easy-peasy lemon-squeezy, let’s see how the pros do it!

Sinusoidal positional encodings

This was the approach taken by the original transformer architecture (2017). Given a list of $T$ tokens: First, they project each token to its corresponding embedding, resulting into a matrix of shape $T\times d$.³ Later, they simply sum a “temporal information” matrix (of shape $T\times d$) on top of the input embeddings. This happens at the very beginning of the transformer architecture:

³ $d$ being the dimensionality of the embedding.

Where the positional matrix is summed with the input embeddings (red circles)

After the addition, each token embedding contains information on which position it holds in the sequence.

Intuitively explained

Each row of the matrix can be seen as a “timestamp”. A bit similar to what a number of $d$ digits would look like (one digit for each coordinate). However, instead of using base-10 digits⁴ the cycles are given by $sin$ and $cos$ functions. In particular, they pair contiguous coordinates of the row: the first element of each pair being the sine and the second the cosine of a function with a given frequency. First coordinates have higher frequencies than later coordinates.

⁴ For reasons similar to what I explained in Tip 1.

Representation of what a given sinusoids embedding might look like.

I give further interpretability of this in Tip 2 and explain advantages of this design choice in @Relative-position-awareness. But before that, let’s first formalize everything.

Mathematical definition

Each row of this matrix is a vector defined by:

$$\begin{array}{r}{p}_t^{(i)}:=\{\begin{array}{ll}\sin ({\omega }_k\cdot t),& \text{if}i=2k\\ \cos ({\omega }_k\cdot t),& \text{if}i=2k+1\end{array}\end{array}$$

Where:

$${\omega }_k=\frac{1}{{10000}^{2k/d}}$$

They call $k$ “offset” in the paper (referring to the embedding coordinates), and I think the $10000$ is quite arbitrary, as long as you have a large number, that should work.

How does this matrix actually look like?

This $T\times d$ matrix looks like this:

Sinusoidal positional embeddings for a sequence of 50 elements (rows) and a 16-dimensional embedding (columns). Notice the high frequency in the first two components (k=1), and how each subsequent component has a lower frequency.

Tip 2: Interpretability

There are many ways of thinking about this, here I put a couple I like.

The clocks analogy

Given a $d$-dimensional positional encoding, take the vector at row $t$:

$$[\sin \left(\frac{t}{1}\right),\cos \left(\frac{t}{1}\right),\sin \left(\frac{t}{{10000}^{\frac{2}{d}}}\right),\cos \left(\frac{t}{{10000}^{\frac{2}{d}}}\right),\sin \left(\frac{t}{{10000}^{\frac{4}{d}}}\right),\cos \left(\frac{t}{{10000}^{\frac{4}{d}}}\right),\dots ,\sin \left(\frac{t}{{10000}^1}\right),\cos \left(\frac{t}{{10000}^1}\right),]$$

As $t$ grows, every component evolves slower: The denominator grows, which makes the frequency lower. I like to pack this array in subsequent pairs of coordinates, and imagine it as an array of $\frac{d}{2}$ clocks⁵:

$$[\underset{🕐}{\underbrace{\sin \left(\frac{t}{1}\right),\cos \left(\frac{t}{1}\right)}},\underset{🕑}{\underbrace{\sin \left(\frac{t}{{10000}^{\frac{2}{d}}}\right),\cos \left(\frac{t}{{10000}^{\frac{2}{d}}}\right)}},\underset{🕒}{\underbrace{\sin \left(\frac{t}{{10000}^{\frac{4}{d}}}\right),\cos \left(\frac{t}{{10000}^{\frac{4}{d}}}\right)}},\dots ,\underset{🕧}{\underbrace{\sin \left(\frac{t}{{10000}^1}\right),\cos \left(\frac{t}{{10000}^1}\right)}},]$$

As we move through time (advance down the rows of the matrix): The first clocks move very fast (like seconds), and then each one moves progressively slower (minutes, hours, …) as we go to the right. Notice that now the “offset” value $k$ is analogous to the clock “index” in this vector:

$$[🕐,🕑,🕒,\dots ,🕧]$$

This is exactly the same as we are doing with the definition of the positional embedding! But since clocks are not numbers⁶, we store $y$ and $x$ coordinates of each clock’s hand - a way to uniquely identify each clock position. These $y$ and $x$ coordinates are the $sin$ and $cos$ of the hand’s angle 🤯

Binary encoding analogy

On a different note, we can also see a similar pattern when writing numbers in binary.

Consider 4-bit binary numbers:

$$\begin{array}{rl}& {0000}_2=0_{10}\\ & {0001}_2=1_{10}\\ & {0010}_2=2_{10}\\ & {0011}_2=3_{10}\\ & {0100}_2=4_{10}\\ & {0101}_2=5_{10}\\ & {0110}_2=6_{10}\\ & {0111}_2=7_{10}\\ & {1000}_2=8_{10}\\ & {1001}_2=9_{10}\end{array}$$

We have that the least significant bit (rightmost) changes every time, the next one every 2 times, then every 4, then every 8, etc. In essence, we have the same pattern as before, where one coordinate changes very frequently, and the next one less so, and so on.

This is how a binary-positional-embedding matrix would look like. Also for 50 elements and 16 dimensions.

🤔 Why don’t we do this for our positional encodings?

In summary, binary embeddings don’t have some of the nice properties of sinusoidal embeddings (which we’ll see shortly). In addition, there is no big reason to use something as discrete as 0’s and 1’s to encode our positional information when everything else are floating point numbers. In machine learning, we usually prefer to work with smooth, continuous, and differentiable stuff, to make things easier for the optimizer.

⁶ Because they are clocks, which is a totally different thing.

⁵ Imagine the clocks only have one hand

Sinusoidal positional encodings present some very nice properties: uniqueness, boundedness, absolute-position-awareness, relative-position-awareness… I’d like to develop more on relative position awareness property: there exists a time-independent linear function transformation mapping the embedding from time $t$ to $t+n$.

Relative position awareness

Given the coordinates $k$ and $k+1$ of the positional encoding at times $n$ and $m$, we have that there exists an absolute-time-independent matrix ${\mathbf{M}}_k$ such that:

$${\mathbf{M}}_k(m-n)\cdot \left[\begin{array}{c}\sin ({\omega }_k\cdot n)\\ \cos ({\omega }_k\cdot n)\end{array}\right]=\left[\begin{array}{c}\sin ({\omega }_k\cdot m)\\ \cos ({\omega }_k\cdot m)\end{array}\right]$$

In particular the matrix ${\mathbf{M}}_k$ is:

$${\mathbf{M}}_k(m-n)=\left[\begin{array}{cc}\cos ({\omega }_k(m-n))& \sin ({\omega }_k(m-n))\\ -\sin ({\omega }_k(m-n))& \cos ({\omega }_k(m-n))\end{array}\right].$$

Interpretation of this linear mapping between two different time-steps ($n$ and $m$), and a given offset $k$ (in this case $k=1$). Of course, this happens for any pair of sin-cos (any $k$), but with a different matrix ${\mathbf{M}}_k$ for each pair.

Tip 3: Rambling: Rotation matrices

In linear algebra, we refer to a matrix of type:

$$\left[\begin{array}{cc}\cos (\varphi )& -\sin (\varphi )\\ \sin (\varphi )& \cos (\varphi )\end{array}\right]$$

as a rotation matrix. It performs a rotation operation in the euclidean space: It is a transformation which rotates the $xy$ plane counterclock-wise through an angle $\theta$ around the origin.

In this case, however, we have a matrix which looks like:

$$\left[\begin{array}{cc}\cos (\varphi )& \sin (\varphi )\\ -\sin (\varphi )& \cos (\varphi )\end{array}\right]$$

It is still a rotation matrix, but it applies the rotation clock-wise instead. It is easy to see since: $\sin (-x)=-\sin (x)$ and $\cos (-x)=\cos (x)$.

As the authors put it: “We chose this function because we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset $k$, $P(m)$ can be represented as a linear function of $P(n)$.”

The hypothesis is that this linearity makes it easier for the model to understand relative distances between positions: The same function maps the difference between position $50$ and $60$ to the difference between position $1000$ and $1010$.

Tip 4: Rambling: Why addition instead of concatenation?

This all looks very cool and all, but something that always bothered me was:

🤔 Why summing this matrix to the input embedding? Wouldn’t it be less destructive to just concatenate it?

It’d be less destructive, but we’d need double the embedding dimension, wasting a lot of memory in the process.

The counter-argument is that this addition is taken into account in the training process. Most likely, the model will not include as much information in the first dimensions as it does in the later ones. Given that the first dimensions’ values oscillate much faster with input position.

RoPE: Rotary Position Embedding

While sinusoidal positional encodings work well, they present some limitations when it comes to implementing more advanced/efficient versions of the transformer⁷. The paper RoFormer addresses them by introducing RoPE.

⁷ For instance: they aren’t as meaningful when compressing subsequences in a single context, or when breaking up sequences across contexts, or when using kernelized variants of the attn mechanism.

RoPE happens directly at the attention mechanism instead of the positional embedding at the beginning.

RoPE is applied in the attention operations (red circles). No positional encoding on the inputs is applied.

Intuitively explained

RoPE’s core idea is actually very close to the sinusoids. However, instead of adding a matrix of “rotations” as before, we are directly modify the query and key embeddings. We do this modification right before the dot-product operation. Given a query embedding⁸, they also do the split-in-pairs thingy of the sinusoids vectors, treat each pair of elements as coordinates in the 2D plane, and apply a rotation to these coordinates.

⁸ They do exactly the same for key embeddings, just focussing on queries for readability.

The angle of the rotation is given by both: the element’s position in the sequence, and the coordinate offset $k$ (same as before).

Steps followed by the RoPE function of a given embedding of the queries at position $n$ and for coordinates $k$. The process is exacly the same for keys.

Mathematical definition

When dealing with 2D stuff, it is often helpful to express things in terms of complex numbers $\mathbb{C}$. Mainly because there are a lot of simple formulas that ease computation⁹. In Tip 5 I explain the connection between both.

⁹ Of course, we could do everything in ${\mathbb{R}}^2$, but it’d be a bit more tedious.

Tip 5: Complex numbers and rotations

Expressing complex numbers

Remember that a complex number $z\in \mathbb{C}$ can be both expressed by ts real and imaginary parts:

$$z=a+ib$$

Where $i=\sqrt{-1}$ is the imaginary unit and $a=\text{Re}(z)$ is the real part, and $b=\text{Im}(z)$ is the imaginary part of $z$.

Similarly, we can express it by its modulus $m$ and angle $\theta$ (aka phase):

$$z=m\cdot e^{i\cdot \theta }$$

We can find the equivalence of both expressions through basic trigonometry:

$$\begin{array}{rl}a& =m\cos \theta \\ b& =m\sin \theta \end{array}$$

Or:

$$\begin{array}{rl}m& =\sqrt{a^2+b^2}\\ \theta & =\arctan \frac{b}{a}\end{array}$$

Applying rotations

Imagine we want to “rotate” a complex value $z=a+i\cdot b=m\cdot e^{i\cdot \theta }$ by an angle of $\varphi$. We basically want a new value $z^{\prime }$ such that its phase is: $\theta +\varphi$.

Here it is very nice to simply use its exponential form and multiply by $e^{i\cdot \varphi }$:

$$z^{\prime }=z\cdot e^{i\cdot \varphi }=m\cdot e^{i\cdot (\theta +\varphi )}$$

This can also be seen through rotation matrices in ${\mathbb{R}}^2$:

$$\begin{array}{rl}{a}^{\prime }& =m\cos (\theta +\varphi )=m\cos (\theta )\cos (\varphi )-m\sin (\theta )\sin (\varphi )\\ b^{\prime }& =m\sin (\theta +\varphi )=m\sin (\theta )\cos (\varphi )+m\cos (\theta )\sin (\varphi )\end{array}$$

Where we usd the sine and cosine sum formulas:

$$\begin{array}{rl}\cos (\alpha +\beta )& =\cos (\alpha )\cos (\beta )-\sin (\alpha )\sin (\beta )\\ \sin (\alpha +\beta )& =\sin (\alpha )\cos (\beta )+\cos (\alpha )\sin (\beta )\end{array}$$

We can further simplify the previous formula using $a=m\cos \theta$, and $b=m\sin \theta$:

$$\begin{array}{rl}{a}^{\prime }& =m\cos (\theta +\varphi )=a\cos (\varphi )-b\sin (\varphi )\\ b^{\prime }& =m\sin (\theta +\varphi )=b\cos (\varphi )+a\sin (\varphi )\end{array}$$

Which, if we express in matrix formulation:

$$\left[\begin{array}{c}{a}^{\prime }\\ b^{\prime }\end{array}\right]=\left[\begin{array}{cc}\cos (\varphi )& -\sin (\varphi )\\ \sin (\varphi )& \cos (\varphi )\end{array}\right]\left[\begin{array}{c}a\\ b\end{array}\right]$$

Which, again (see Tip 3) is a rotation matrix. This time it applies a counterclock-wise rotation.

More formally, given an embedding $x$ at position $n$: $$x=[x_0,x_1,x_2,x_3,\dots ,x_{d-2},x_{d-1}]$$

We do the “pairing” of contiguous dimensions, this time expressing the values as complex numbers:

$$x=(x_0+i\cdot x_1),(x_2+i\cdot x_3),\dots ,(x_{d-2}+i\cdot x_{d-1})$$

We have $k\in {\mathbb{Z}}_{0:\frac{d}{2}}$ pairs of complex values. $k$ is also referred as the “offset” (same as before).

The RoPE operation is defined by applying a rotation of $n{\theta }_k$ at each corresponding complex number:

$$x^{\prime }=(x_0+i\cdot x_1)\cdot e^{i\cdot n\cdot {\theta }_0},(x_2+i\cdot x_3)\cdot e^{i\cdot n\cdot {\theta }_1},\dots ,(x_{d-2}+i\cdot x_{d-1})\cdot e^{i\cdot n\cdot {\theta }_{\frac{d}{2}}}$$

Where ${\theta }_k$ is the same as before:

$${\theta }_k:=\frac{1}{{10000}^{\frac{2k}{d}}}$$

Notice we can equivalently express the RoPE operation as a linear function without using complex numbers. Given the query (or key) embedding $x$ at position $n$:

$$\left[\begin{array}{c}{x}_0^{\prime }\\ x_1^{\prime }\\ x_2^{\prime }\\ x_3^{\prime }\\ ⋮\\ x_{d-2}^{\prime }\\ x_{d-1}^{\prime }\end{array}\right]=\left[\begin{array}{ccccccc}\cos (n\cdot {\theta }_0)& -\sin (n\cdot {\theta }_0)& 0& 0& \dots & 0& 0\\ \sin (n\cdot {\theta }_0)& \cos (n\cdot {\theta }_0)& 0& 0& \dots & 0& 0\\ 0& 0& \cos (n\cdot {\theta }_1)& -\sin (n\cdot {\theta }_1)& \dots & 0& 0\\ 0& 0& \sin (n\cdot {\theta }_1)& \cos (n\cdot {\theta }_1)& \dots & 0& 0\\ ⋮& ⋮& ⋮& ⋮& \ddots & ⋮& ⋮\\ 0& 0& 0& 0& \dots & \cos (n\cdot {\theta }_{\frac{d}{2}})& -\sin (n\cdot {\theta }_{\frac{d}{2}})\\ 0& 0& 0& 0& \dots & \sin (n\cdot {\theta }_{\frac{d}{2}})& \cos (n\cdot {\theta }_{\frac{d}{2}})\end{array}\right]\left[\begin{array}{c}{x}_0\\ x_1\\ x_2\\ x_3\\ ⋮\\ x_{d-2}\\ x_{d-1}\end{array}\right]$$