# guillaume-be/rust-tokenizers

<https://repo.masters-masters.com/en/repos/guillaume-be/rust-tokenizers>

[Home](https://repo.masters-masters.com/en) › [guillaume-be](https://repo.masters-masters.com/en/owners/guillaume-be) › [rust-tokenizers](https://repo.masters-masters.com/en/repos/guillaume-be/rust-tokenizers)

guillaume-be/rust-tokenizers has 345 stars on GitHub. Its RepoMasters card: Alpha tier. Rust-tokenizer offers high-performance tokenizers for modern language…

| Field | Value |
| --- | --- |
| Owner | [guillaume-be](https://repo.masters-masters.com/en/owners/guillaume-be) |
| Stars | 345 |
| Forks | 33 |
| Language | [Rust](https://repo.masters-masters.com/en/languages/rust) |
| License | Apache-2.0 |
| Created | 2019 |
| Topics | tokenizer, rust-lang, transformer, deep-learning |
| Tier | [Alpha (250 to 699 ★)](https://repo.masters-masters.com/en/ranks/b_movie) |
| View on GitHub | [guillaume-be/rust-tokenizers](https://github.com/guillaume-be/rust-tokenizers) |

## Repository

Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models

The tier for repos with 250 to 699 ★. Published odds per card pulled: 20%. Rarity: Uncommon.

## More repos by guillaume-be

- [guillaume-be/rust-bert · ★ 3,091 · Rust · RC](https://repo.masters-masters.com/en/repos/guillaume-be/rust-bert)

## Nearby in Rust

- [guoqingbao/xinfer · ★ 345 · Rust · Alpha](https://repo.masters-masters.com/en/repos/guoqingbao/xinfer)
- [nsaintot/cdj3k-emu · ★ 345 · Rust · Alpha](https://repo.masters-masters.com/en/repos/nsaintot/cdj3k-emu)
- [Dicklesworthstone/franken\_ocr · ★ 345 · Rust · Alpha](https://repo.masters-masters.com/en/repos/Dicklesworthstone/franken_ocr)
- [zonyitoo/rust-ini · ★ 345 · Rust · Alpha](https://repo.masters-masters.com/en/repos/zonyitoo/rust-ini)
- [reem/rust-ordered-float · ★ 345 · Rust · Alpha](https://repo.masters-masters.com/en/repos/reem/rust-ordered-float)
- [mattgathu/cute · ★ 345 · Rust · Alpha](https://repo.masters-masters.com/en/repos/mattgathu/cute)

---
Data: GitHub · Illustration drawn by RepoMasters from the repo’s ID and its language color.