# marcelroed/gigatoken

<https://repo.masters-masters.com/en/repos/marcelroed/gigatoken>

[Home](https://repo.masters-masters.com/en) › [marcelroed](https://repo.masters-masters.com/en/owners/marcelroed) › [gigatoken](https://repo.masters-masters.com/en/repos/marcelroed/gigatoken)

marcelroed/gigatoken has 4,150 stars on GitHub. Its RepoMasters card: RC tier. Language model tokenization at GB/s Language: Rust.

| Field | Value |
| --- | --- |
| Owner | [marcelroed](https://repo.masters-masters.com/en/owners/marcelroed) |
| Stars | 4,150 |
| Forks | 224 |
| Language | [Rust](https://repo.masters-masters.com/en/languages/rust) |
| License | MIT |
| Created | 2025 |
| Topics | llm, nlp, tokenization, tokenizer |
| Tier | [RC (2,000 to 4,999 ★)](https://repo.masters-masters.com/en/ranks/blockbuster) |
| View on GitHub | [marcelroed/gigatoken](https://github.com/marcelroed/gigatoken) |

## Repository

Language model tokenization at GB/s

The tier for repos with 2,000 to 4,999 ★. Published odds per card pulled: 3%. Rarity: Epic.

## Nearby in Rust

- [linebender/resvg · ★ 4,173 · Rust · RC](https://repo.masters-masters.com/en/repos/linebender/resvg)
- [zhongbai2333/Tomato-Novel-Downloader · ★ 4,163 · Rust · RC](https://repo.masters-masters.com/en/repos/zhongbai2333/Tomato-Novel-Downloader)
- [lumina-ai-inc/chunkr · ★ 4,153 · Rust · RC](https://repo.masters-masters.com/en/repos/lumina-ai-inc/chunkr)
- [juhaku/utoipa · ★ 4,141 · Rust · RC](https://repo.masters-masters.com/en/repos/juhaku/utoipa)
- [Fredolx/open-tv · ★ 4,141 · Rust · RC](https://repo.masters-masters.com/en/repos/Fredolx/open-tv)
- [arxanas/git-branchless · ★ 4,140 · Rust · RC](https://repo.masters-masters.com/en/repos/arxanas/git-branchless)

---
Data: GitHub · Illustration: the project’s public image, pixelated by RepoMasters.