PrototypeCommon
twistedcubic/attention-rank-collapse
[ICML 2021 Oral] We show pure attention suffers rank collapse, and how different mechanisms combat it.
- Owner
- twistedcubic (personal account)
- Stars
- Stars: 173
- Forks
- Forks: 11
- Language
- Jupyter Notebook
- License
- Apache-2.0
- Created
- 2021
- Tier
- Prototype (100 to 249 ★) What this tier means
Nearby in Jupyter Notebook
All Jupyter Notebook repos- VizuaraAILabs/DeepSeek-From-ScratchStars: 173Jupyter NotebookPrototype
- Brandon-Collins7/sketchvlmStars: 173Jupyter NotebookPrototype
- wow-world-model/wow-world-modelStars: 173Jupyter NotebookPrototype
- ContinuumIO/numbapro-examplesStars: 173Jupyter NotebookPrototype
- microsoft/Cognitive-Vision-PythonStars: 173Jupyter NotebookPrototype
- eleanorlutz/AnimatedPythonPatternsStars: 173Jupyter NotebookPrototype