RCEpic
marcelroed/gigatoken
Language model tokenization at GB/s
- Owner
- marcelroed (personal account)
- Stars
- Stars: 4,150
- Forks
- Forks: 223
- Language
- Rust
- License
- MIT
- Created
- 2025
- Topics
- llmnlptokenizationtokenizer
- Tier
- RC (2,000 to 4,999 ★) What this tier means
Nearby in Rust
All Rust repos- zhongbai2333/Tomato-Novel-DownloaderStars: 4,158RustRC
- lumina-ai-inc/chunkrStars: 4,153RustRC
- anomalyco/hexStars: 4,152RustRC
- juhaku/utoipaStars: 4,141RustRC
- Fredolx/open-tvStars: 4,141RustRC
- arxanas/git-branchlessStars: 4,139RustRC