# twistedcubic/attention-rank-collapse

<https://repo.masters-masters.com/en/repos/twistedcubic/attention-rank-collapse>

[Home](https://repo.masters-masters.com/en) › [twistedcubic](https://repo.masters-masters.com/en/owners/twistedcubic) › [attention-rank-collapse](https://repo.masters-masters.com/en/repos/twistedcubic/attention-rank-collapse)

twistedcubic/attention-rank-collapse has 173 stars on GitHub. Its RepoMasters card: Prototype tier. \[ICML 2021 Oral\] We show pure attention suffers rank…

| Field | Value |
| --- | --- |
| Owner | [twistedcubic](https://repo.masters-masters.com/en/owners/twistedcubic) |
| Stars | 173 |
| Forks | 11 |
| Language | [Jupyter Notebook](https://repo.masters-masters.com/en/languages/jupyter-notebook) |
| License | Apache-2.0 |
| Created | 2021 |
| Tier | [Prototype (100 to 249 ★)](https://repo.masters-masters.com/en/ranks/rush) |
| View on GitHub | [twistedcubic/attention-rank-collapse](https://github.com/twistedcubic/attention-rank-collapse) |

## Repository

\[ICML 2021 Oral\] We show pure attention suffers rank collapse, and how different mechanisms combat it.

The tier for repos with 100 to 249 ★. Published odds per card pulled: 70%. Rarity: Common.

## Nearby in Jupyter Notebook

- [VizuaraAILabs/DeepSeek-From-Scratch · ★ 173 · Jupyter Notebook · Prototype](https://repo.masters-masters.com/en/repos/VizuaraAILabs/DeepSeek-From-Scratch)
- [Brandon-Collins7/sketchvlm · ★ 173 · Jupyter Notebook · Prototype](https://repo.masters-masters.com/en/repos/Brandon-Collins7/sketchvlm)
- [wow-world-model/wow-world-model · ★ 173 · Jupyter Notebook · Prototype](https://repo.masters-masters.com/en/repos/wow-world-model/wow-world-model)
- [ContinuumIO/numbapro-examples · ★ 173 · Jupyter Notebook · Prototype](https://repo.masters-masters.com/en/repos/ContinuumIO/numbapro-examples)
- [microsoft/Cognitive-Vision-Python · ★ 173 · Jupyter Notebook · Prototype](https://repo.masters-masters.com/en/repos/microsoft/Cognitive-Vision-Python)
- [eleanorlutz/AnimatedPythonPatterns · ★ 173 · Jupyter Notebook · Prototype](https://repo.masters-masters.com/en/repos/eleanorlutz/AnimatedPythonPatterns)

---
Data: GitHub · Illustration drawn by RepoMasters from the repo’s ID and its language color.