PrototypeCommon
psmarter/mini-infer
LLM inference engine from scratch — paged KV cache, continuous batching, chunked prefill, prefix caching, speculative decoding, CUDA graph, tensor parallelism, OpenAI-compatible serving
- Owner
- psmarter (personal account)
- Stars
- Stars: 237
- Forks
- Forks: 12
- Language
- Python
- License
- MIT
- Created
- 2025
- Topics
- cudainferencellmmachine-learningpagedattentionpytorchtransformertritonlanguage-modelcontinuous-batching
- Tier
- Prototype (100 to 249 ★) What this tier means
More repos by psmarter
All 3 repos by psmarter- psmarter/PMPP-LearningStars: 197C++Prototype
- psmarter/CUDA-PracticeStars: 127CudaPrototype
Nearby in Python
All Python repos- jmedina21/deck-as-controllerStars: 237PythonPrototype
- 10291124jcjc-dotcom/ai-character-swapStars: 237PythonPrototype
- RabbittuX/purrgeStars: 237PythonPrototype
- jeetsukumaran/DendroPyStars: 237PythonPrototype
- wordnik/wordnik-pythonStars: 237PythonPrototype
- GDSSecurity/GWT-Penetration-Testing-ToolsetStars: 237PythonPrototype