Illustration drawn by RepoMasters from the repo’s ID and its language color. Credits

PrototypeCommon

actual-computer/toks

exact tokenizers ids, as fast as the hardware allows

Owner: actual-computer (organization)
Stars: Stars: 119
Forks: Forks: 10
Language: C
License: Apache-2.0
Created: 2026
Topics: assemblyavx2bpechuggingfaceneonsentencepiecesimdtokenizerwordpiece
Tier: Prototype (100 to 249 ★) What this tier means

Nearby in C

Eruditi/CE-MCP-PluginStars: 119CPrototype
X-3306/Project-OnyxStars: 119CPrototype
staturnzz/momentariusStars: 119CPrototype
kupferlauncher/keybinderStars: 119CPrototype
dwelch67/thumbulatorStars: 119CPrototype
pprett/nutStars: 119CPrototype