BetaRare
microsoft/MInference
[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.
- Owner
- microsoft (organization)
- Stars
- Stars: 1,231
- Forks
- Forks: 84
- Language
- Python
- License
- MIT
- Created
- 2024
- Tier
- Beta (700 to 1,999 ★) What this tier means
More repos by microsoft
All 1,923 repos by microsoft- microsoft/vscodeStars: 194,428TypeScriptLTS
- microsoft/markitdownStars: 193,282PythonLTS
- microsoft/PowerToysStars: 140,362CLTS
- microsoft/generative-ai-for-beginnersStars: 122,672Jupyter NotebookLTS
- microsoft/TypeScriptStars: 111,706GoLTS
- microsoft/terminalStars: 105,300C++LTS
Nearby in Python
All Python repos- taranis-ai/taranis-aiStars: 1,231PythonBeta
- LifeArchiveProject/HistoryHubStars: 1,231PythonBeta
- refreshdotdev/web-eval-agentStars: 1,231PythonBeta
- asciidisco/plugin.video.netflixStars: 1,231PythonBeta
- python-adaptive/adaptiveStars: 1,231PythonBeta
- qubvel/classification_modelsStars: 1,231PythonBeta