# OpenNMT/Tokenizer

<https://repo.masters-masters.com/en/repos/OpenNMT/Tokenizer>

[Home](https://repo.masters-masters.com/en) › [OpenNMT](https://repo.masters-masters.com/en/owners/OpenNMT) › [Tokenizer](https://repo.masters-masters.com/en/repos/OpenNMT/Tokenizer)

OpenNMT/Tokenizer has 343 stars on GitHub. Its RepoMasters card: Alpha tier. Fast and customizable text tokenization library with BPE and SentencePiece support

| Field | Value |
| --- | --- |
| Owner | [OpenNMT](https://repo.masters-masters.com/en/owners/OpenNMT) |
| Stars | 343 |
| Forks | 83 |
| Language | [C++](https://repo.masters-masters.com/en/languages/c-plus-plus) |
| License | MIT |
| Created | 2017 |
| Topics | tokenizer, sentencepiece, natural-language-processing, machine-translation, bpe, unicode, tokenization, icu, python, cpp |
| Tier | [Alpha (250 to 699 ★)](https://repo.masters-masters.com/en/ranks/b_movie) |
| View on GitHub | [OpenNMT/Tokenizer](https://github.com/OpenNMT/Tokenizer) |

## Repository

Fast and customizable text tokenization library with BPE and SentencePiece support

The tier for repos with 250 to 699 ★. Published odds per card pulled: 20%. Rarity: Uncommon.

## More repos by OpenNMT

- [OpenNMT/OpenNMT-py · ★ 7,021 · Python · Stable](https://repo.masters-masters.com/en/repos/OpenNMT/OpenNMT-py)
- [OpenNMT/CTranslate2 · ★ 4,730 · C++ · RC](https://repo.masters-masters.com/en/repos/OpenNMT/CTranslate2)
- [OpenNMT/OpenNMT · ★ 2,387 · Lua · RC](https://repo.masters-masters.com/en/repos/OpenNMT/OpenNMT)
- [OpenNMT/OpenNMT-tf · ★ 1,485 · Python · Beta](https://repo.masters-masters.com/en/repos/OpenNMT/OpenNMT-tf)
- [OpenNMT/Im2Text · ★ 137 · Lua · Prototype](https://repo.masters-masters.com/en/repos/OpenNMT/Im2Text)

## Nearby in C++

- [hkr04/cpp-mcp · ★ 343 · C++ · Alpha](https://repo.masters-masters.com/en/repos/hkr04/cpp-mcp)
- [MarioGalindoQ/Modern-CPP-Design-Patterns · ★ 343 · C++ · Alpha](https://repo.masters-masters.com/en/repos/MarioGalindoQ/Modern-CPP-Design-Patterns)
- [nathanshankar/terminal\_rviz · ★ 343 · C++ · Alpha](https://repo.masters-masters.com/en/repos/nathanshankar/terminal_rviz)
- [philogb/v8-gl · ★ 343 · C++ · Alpha](https://repo.masters-masters.com/en/repos/philogb/v8-gl)
- [amcewen/HttpClient · ★ 343 · C++ · Alpha](https://repo.masters-masters.com/en/repos/amcewen/HttpClient)
- [nektra/Deviare-InProc · ★ 343 · C++ · Alpha](https://repo.masters-masters.com/en/repos/nektra/Deviare-InProc)

---
Data: GitHub · Illustration drawn by RepoMasters from the repo’s ID and its language color.