2026-01-19
Do RoPE rotations mess with semantics?
How do RoPE rotations interact with semantic meaning that vectors represent?
If RoPE is rotating dimension pairs, doesn't it influence the semantic meaning as vector dimensions determine semantic meaning?
Discuss this on discord or DMs. I'm open for collaboration on this.