Short answer: On Oct 20, DeepSeek open-sourced DeepInfer, the inference framework it says runs DeepSeek-V4.1-Flash in production. It is licensed Apache 2.0 and supports only that one model, on NVIDIA SM100 GPUs (Blackwell-class data center chips) and Huawei Ascend 950 NPUs. This first release is offline inference only and leaves out a KV cache system, load balancing across replicas and full prefill/decode splitting. If you do not own that hardware, treat it as code to study, not something to install. Throughput claims are DeepSeek’s and have not been independently tested.
What is DeepInfer?
According to the project README, it is a high-performance, large-scale inference framework for DeepSeek models, built on projects such as FlashMLA, DeepGEMM and DeepEP. The New Stack reports it began as an early version of vLLM before being heavily customized, and that DeepSeek said in April 2025 its engine was too tightly tied to internal infrastructure to release. DeepSeek has not explained how it overcame that. At the time I read it, the repository showed about 12.4k stars.
What is included, and what is missing?
Included, per the README: optimized kernels and a model and sampler runtime, attention data parallelism, balanced context parallelism, expert parallelism with load balancing, speculative decoding with confidence-based verification, asynchronous scheduling on by default, and structured outputs (regex, JSON schema, grammar). Batch-invariant inference and deterministic sampling exist, but only on NVIDIA.
Missing: a KV cache system, load balancing across data-parallel replicas, and full prefill/decode disaggregation. A full production service needs those, so you would have to build or add them.
Can I run it?
| Requirement | Detail |
|---|---|
| Model | DeepSeek-V4.1-Flash only |
| NVIDIA | SM100 architecture with CUDA 13. Hopper and consumer cards are not listed |
| Huawei | Ascend 950 with CANN 9.2 |
| OS and tools | Linux, GCC 13+, Python 3.12 |
| License | Apache 2.0 |
| Mode | Offline (batch) inference |
The README’s performance charts were measured on 4 SM100 GPUs for prefill and 64 for decode, with a script to reproduce them. The page shows charts rather than raw numbers. The New Stack says performance on Huawei chips is unknown and that developers without SM100 or Ascend hardware may mainly benefit from studying the code.
What is the model it serves?
DeepSeek-V4.1-Flash-1017 was announced on Oct 17 under the MIT license, with a 1 million token context window, according to mind-verse, which summarizes the Hugging Face card. The article lists benchmark gains over the earlier V4.1-Flash, all reported by DeepSeek.
Scores are from DeepSeek’s model card, via mind-verse. Not independently reproduced.
The largest reported jumps are Terminal-Bench 4.0 (31.2 to 42.3) and ProgramBench (20.3 to 29.1). The same source reports Search and Fetch APIs at $0.75 per 1,000 calls per endpoint, with server-side tool calls free for API users until billing starts on Oct 31, 16:00 UTC. That pricing is for search, not model inference.
Who should care?
- If you run a Blackwell data center cluster: this is the first look at how DeepSeek serves its model, and it is usable now for offline runs.
- If you build inference engines: the kernels, scheduling and speculative decoding code are worth reading.
- If you have a single consumer GPU: keep using vLLM, SGLang or llama.cpp, and see our hardware guide to open-weight coding models for what fits. DeepInfer does not list your hardware.
- If you follow the China chip story: official Ascend 950 support is a signal. The New Stack notes Bloomberg, The Information and Reuters have reported DeepSeek plans around Huawei chips, but a first-version port has caveats.
What I am unsure about
- How fast it really is. The speed claims come from DeepSeek’s own charts, and I did not run it.
- The model’s parameter count. A Hugging Face card summary lists 763B total, another source lists a different figure for the base model, and I could not settle it. Check the card.
- Whether the earlier-to-1017 comparison table uses the version I assume. The source calls it a smaller update to V4.1-Flash.
- I read the README through a fetch tool and press coverage, and did not build the project.
FAQ
Is DeepInfer the same as DeepInfra?
No. DeepInfra is a separate hosting company. DeepInfer is DeepSeek’s engine.
Does it run other models, like Llama or Qwen?
The README lists only DeepSeek-V4.1-Flash.
Is it better than vLLM or SGLang?
The sources do not compare them. DeepSeek’s work began on vLLM and draws ideas from SGLang and TensorRT-LLM.
Sources
- deepseek-ai/DeepInfer — GitHub
- DeepSeek open-sources the inference engine behind its API — The New Stack
- DeepInfer: What’s actually inside — DEV Community
- DeepSeek V4.1-Flash-1017 with open weights — mind-verse
Written with AI assistance and checked against the sources above. See About for how ossbeat works.
