# IBM/ModuleFormer

<https://repo.masters-masters.com/en/repos/IBM/ModuleFormer>

[Home](https://repo.masters-masters.com/en) › [IBM](https://repo.masters-masters.com/en/owners/IBM) › [ModuleFormer](https://repo.masters-masters.com/en/repos/IBM/ModuleFormer)

IBM/ModuleFormer has 223 stars on GitHub. Its RepoMasters card: Prototype tier. ModuleFormer is a MoE-based architecture that includes two different types of…

| Field | Value |
| --- | --- |
| Owner | [IBM](https://repo.masters-masters.com/en/owners/IBM) |
| Stars | 223 |
| Forks | 11 |
| Language | [Python](https://repo.masters-masters.com/en/languages/python) |
| License | Apache-2.0 |
| Created | 2023 |
| Topics | lm, moe |
| Tier | [Prototype (100 to 249 ★)](https://repo.masters-masters.com/en/ranks/rush) |
| View on GitHub | [IBM/ModuleFormer](https://github.com/IBM/ModuleFormer) |

## Repository

ModuleFormer is a MoE-based architecture that includes two different types of experts: stick-breaking attention heads and feedforward experts. We released a collection of ModuleFormer-based Language Models (MoLM) ranging in scale from 4 billion to 8 billion parameters.

The tier for repos with 100 to 249 ★. Published odds per card pulled: 70%. Rarity: Common.

## More repos by IBM

- [IBM/sarama · ★ 12,525 · Go · Stable](https://repo.masters-masters.com/en/repos/IBM/sarama)
- [IBM/plex · ★ 11,736 · CSS · Stable](https://repo.masters-masters.com/en/repos/IBM/plex)
- [IBM/mcp-context-forge · ★ 4,733 · Python · RC](https://repo.masters-masters.com/en/repos/IBM/mcp-context-forge)
- [IBM/AssetOpsBench · ★ 2,409 · Python · RC](https://repo.masters-masters.com/en/repos/IBM/AssetOpsBench)
- [IBM/css-gridish · ★ 2,216 · CSS · RC](https://repo.masters-masters.com/en/repos/IBM/css-gridish)
- [IBM/fp-go · ★ 2,034 · Go · RC](https://repo.masters-masters.com/en/repos/IBM/fp-go)

## Nearby in Python

- [moskoo/xxg-portrait-rebuild-light · ★ 223 · Python · Prototype](https://repo.masters-masters.com/en/repos/moskoo/xxg-portrait-rebuild-light)
- [M1n-n9/academic-figures-drawer · ★ 223 · Python · Prototype](https://repo.masters-masters.com/en/repos/M1n-n9/academic-figures-drawer)
- [LiQiiiii/Jeva · ★ 223 · Python · Prototype](https://repo.masters-masters.com/en/repos/LiQiiiii/Jeva)
- [qgriffith-zz/OpenEats · ★ 223 · Python · Prototype](https://repo.masters-masters.com/en/repos/qgriffith-zz/OpenEats)
- [aewallin/openvoronoi · ★ 223 · Python · Prototype](https://repo.masters-masters.com/en/repos/aewallin/openvoronoi)
- [janosgyerik/bashoneliners · ★ 223 · Python · Prototype](https://repo.masters-masters.com/en/repos/janosgyerik/bashoneliners)

---
Data: GitHub · Illustration drawn by RepoMasters from the repo’s ID and its language color.