PrototypeCommon
IBM/ModuleFormer
ModuleFormer is a MoE-based architecture that includes two different types of experts: stick-breaking attention heads and feedforward experts. We released a collection of ModuleFormer-based Language Models (MoLM) ranging in scale from 4 billion to 8 billion parameters.
- Owner
- IBM (organization)
- Stars
- Stars: 223
- Forks
- Forks: 11
- Language
- Python
- License
- Apache-2.0
- Created
- 2023
- Topics
- lmmoe
- Tier
- Prototype (100 to 249 ★) What this tier means
More repos by IBM
All 172 repos by IBM- IBM/saramaStars: 12,525GoStable
- IBM/plexStars: 11,736CSSStable
- IBM/mcp-context-forgeStars: 4,733PythonRC
- IBM/AssetOpsBenchStars: 2,409PythonRC
- IBM/css-gridishStars: 2,216CSSRC
- IBM/fp-goStars: 2,034GoRC
Nearby in Python
All Python repos- moskoo/xxg-portrait-rebuild-lightStars: 223PythonPrototype
- M1n-n9/academic-figures-drawerStars: 223PythonPrototype
- LiQiiiii/JevaStars: 223PythonPrototype
- qgriffith-zz/OpenEatsStars: 223PythonPrototype
- aewallin/openvoronoiStars: 223PythonPrototype
- janosgyerik/bashonelinersStars: 223PythonPrototype