DeepInfer: DeepSeek's open-source inference framework
DeepInfer is the framework DeepSeek built to run its own models in production — and on October 20, 2026, it became open source under Apache-2.0. Here is what the release actually contains, what it deliberately leaves out, and why the machinery behind cheap tokens matters if you pay for inference by the million tokens.
Facts verified against the official repository on 2026-10-23 · sources listed at the end
What is DeepInfer?
DeepInfer is DeepSeek's execution framework for large-scale inference of its own models — the layer that turns model weights into served tokens. The repo describes the release as providing offline inference with the core features needed for serving at scale, and it is the same stack of kernels, parallelism, and scheduling that powers DeepSeek's production DeepSeek-V4.1-Flash services. It is not a new model: it is the engine the model runs on.
- Developer
- DeepSeek
- License
- Apache-2.0
- Public release
- October 20, 2026 (20.4k stars three days later)
- Model support
- DeepSeek-V4.1-Flash only (checkpoint DeepSeek-V4.1-Flash-1017)
- Hardware
- NVIDIA SM100 GPUs; Huawei Ascend 950 NPUs
- Built on
- vLLM (inspired by SGLang and TensorRT-LLM)
- Requires
- Linux, GCC 13+, Python 3.12, CUDA 13 or CANN 9.2
- Release scope
- Offline inference; serving features partial (see below)
What the release actually ships
- Production kernels and runtime — FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels, several of which DeepSeek had already open-sourced individually.
- Parallelism — attention data parallelism / balanced context parallelism, and MoE expert parallelism with expert-parallel load balancing (EPLB).
- Speculative decoding — DSpark with confidence-scheduled verification of multi-token prediction (MTP) draft tokens.
- Throughput features — SWA bounded replay (decoder prefill reduction) and asynchronous scheduling, enabled by default.
- Reliability features (NVIDIA only) — batch-invariant inference and deterministic sampling.
- Structured outputs — constrained decoding against regex, JSON Schema, or EBNF grammars.
What it does not include yet
The README is explicit that this is an offline-inference release, not a turnkey serving stack: the KV cache system is absent, prefill/decode disaggregation is only partially implemented, and load balancing across data-parallel replicas is not included. If you are evaluating DeepInfer for production serving, those gaps — not the kernels — are the current limiting factor.
Why an AI cost site covers an inference framework
Because inference is where model prices come from. Every per-million-token rate on this site is set downstream of a serving stack: how many tokens a given accelerator pushes per hour, and how much of the drafted output survives verification, decides the floor of what any provider — or your own GPUs — can afford to charge. DeepSeek's per-token pricing is widely regarded as the reference point for inexpensive frontier inference, and DeepInfer is the first public look at the machinery behind it: MLA attention kernels, expert-parallel communication, and speculative decoding are exactly the levers that make cheap tokens possible. If you self-host DeepSeek models, tokens-per-GPU-hour — not list price — is your real unit cost. To compare list prices across API providers for the models we track, use the model cost calculator or read our methodology.
DeepInfer and vLLM
DeepInfer is not a vLLM competitor in the usual sense. The README states the framework is built on vLLM and takes inspiration from SGLang and TensorRT-LLM; what DeepSeek open-sourced is the production layer it added on top. Community reaction — the r/LocalLLaMA thread and 20k+ stars in the first days — reads it as a reference implementation other frameworks will learn from, not a drop-in replacement.
Frequently asked questions
What is DeepInfer?
DeepInfer is the high-performance inference framework DeepSeek uses to run its own models at scale — including the DeepSeek-V4.1-Flash production services. DeepSeek released it as open source under Apache-2.0 on October 20, 2026. It is built on vLLM and adds DeepSeek's production kernels, parallelism, and speculative decoding on top.
Is DeepInfer free and open source?
Yes. The repository is licensed under Apache-2.0, which permits commercial use, modification, and redistribution.
Which models can DeepInfer run?
The first public release supports DeepSeek-V4.1-Flash only (default checkpoint DeepSeek-V4.1-Flash-1017). Other model families are not included yet.
Does DeepInfer run on Huawei Ascend NPUs?
Yes. The release targets NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs (CUDA 13 or CANN 9.2 respectively). Batch-invariant inference and deterministic sampling are NVIDIA-only features.
Can DeepInfer replace vLLM or SGLang for serving?
Not as-is. The release focuses on offline inference: the KV cache system is absent, prefill/decode disaggregation is only partially implemented, and load balancing across data-parallel replicas is not included. DeepInfer is also built on top of vLLM rather than being an independent competitor to it.
What is DSpark?
DSpark is DeepInfer's speculative-decoding path: it drafts tokens with multi-token prediction (MTP) and accepts them via confidence-scheduled verification. Speculative decoding is one of the main levers for decode throughput — and therefore for tokens per dollar.
Sources
- DeepSeek (GitHub) — DeepInfer: high-performance, large-scale inference framework for DeepSeek modelshttps://github.com/deepseek-ai/DeepInfer · accessed 2026-10-23
- MarkTechPost — DeepSeek open-sources DeepInfer: the inference engine behind its production AI systemshttps://www.marktechpost.com/2026/10/20/deepseek-deepinfer-open-source-inference-framework · published 2026-10-20 · accessed 2026-10-23
This page is an independent explainer; TaskPriced is unaffiliated with DeepSeek. DeepInfer evolves fast — check the repository before acting on specifics. See also the latest model updates feed.