The DeepSeek Trial: I rented two Blackwells. One 180K request filled them.
DeepSeek V4 Flash on two 96 GB RTX PRO 6000 Blackwell cards looked like a genuinely promising serving path. The custom stack worked, write-heavy throughput looked good, and then long context hit a hard memory ceiling.
The plan was genuinely promising
I rented a Vast.ai machine with two NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs. Each card had 96 GB of VRAM, for roughly 192 GB total. A comparable machine is about $25,000 worth of hardware.
The target was DeepSeek V4 Flash, a 284B mixture-of-experts model that activates about 13B parameters per token and advertises a 1,048,576-token maximum context. The NVIDIA hybrid NVFP4 checkpoint is about 156.75 GiB, so TP2 looked tight but possible.
I started from a deployment playbook that pinned the checkpoint, SGLang requirements, storage layout, TP2 settings, and the correct flashinfer_trtllm_routed backend for NVIDIA's hybrid export. That avoided the obvious mistakes, but stock SGLang still was not enough for this SM120 TP2 setup.
The hacked-together branch worked
The final known-good setup used the regular deepseek-ai/DeepSeek-V4-Flash checkpoint with a custom SGLang build combining PR #30272, experimental PR #29927, and DeepGEMM PR #324. It used TP2, FP8 KV cache, Marlin for the output-heavy MoE path, and EAGLE/MTP.
Do not copy the Marlin choice back to NVIDIA's hybrid NVFP4 export. These are different checkpoint paths. The NVIDIA export still needs flashinfer_trtllm_routed.
The core launch flags ended up like this:
sglang serve \ --model-path deepseek-ai/DeepSeek-V4-Flash \ --revision 60d8d70770c6776ff598c94bb586a859a38244f1 \ --trust-remote-code \ --attention-backend dsv4 \ --kv-cache-dtype fp8_e4m3 \ --tp 2 \ --moe-runner-backend marlin \ --fp8-gemm-backend cutlass \ --mem-fraction-static 0.90 \ --chunked-prefill-size 8192 \ --max-prefill-tokens 8192 \ --cuda-graph-max-bs-decode 32 \ --speculative-algorithm EAGLE \ --speculative-eagle-topk 1 \ --speculative-num-steps 2 \ --speculative-num-draft-tokens 3 \ --reasoning-parser deepseek-v4 \ --tool-call-parser deepseekv4
The optimized runtime loaded about 74.97 GB of main weights and 2.70 GB of MTP weights per GPU. After graphs and runtime allocations, each 96 GB card settled around 89.5 GB used. SGLang reported a total runtime token budget of 186,624.
The write-heavy result was promising
The same setup looked much better when the workload shifted to short inputs and long generations. This run used 20 prompts configured for 500 random input tokens and up to 5,000 output tokens, with client concurrency 20 and no warmup.
python -m sglang.benchmark.serving \ --backend sglang-oai \ --base-url http://127.0.0.1:18000 \ --model deepseek-ai/DeepSeek-V4-Flash \ --dataset-name random \ --num-prompts 20 \ --random-input-len 500 \ --random-output-len 5000 \ --random-range-ratio 0.0 \ --max-concurrency 20 \ --warmup-requests 0 \ --output-file /workspace/deepseek-v4-c10-i1k-o5k.json
All 20 requests completed. The run processed 4,449 actual input tokens and generated 45,903 output tokens in 68.78 seconds. Aggregate output throughput reached 667.34 tokens per second, with a mean time per output token of 18.04 ms.
| Successful requests | 20 / 20 |
|---|---|
| Duration | 68.78 seconds |
| Generated tokens | 45,903 |
| Output throughput | 667.34 tokens/second |
| Total throughput | 732.03 tokens/second |
| Mean end-to-end latency | 42.90 seconds |
| Mean time to first token | 3.88 seconds |
| Mean time per output token | 18.04 ms |
| MTP accept length | 2.60 |
Once the short prompt was out of the way and the GPUs stayed decode-heavy, the machine could move output. This was genuinely promising.
Full write-heavy benchmark output
============ Serving Benchmark Result ============ Backend: sglang-oai Traffic request rate: inf Max request concurrency: 20 Successful requests: 20 Benchmark duration (s): 68.78 Total input tokens: 4449 Total input text tokens: 4449 Total generated tokens: 45903 Total generated tokens (retokenized): 45899 Request throughput (req/s): 0.29 Input token throughput (tok/s): 64.68 Output token throughput (tok/s): 667.34 Peak output token throughput (tok/s): 380.00 Peak concurrent requests: 20 Total token throughput (tok/s): 732.03 Concurrency: 12.47 Accept length: 2.60 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 42899.91 Median E2E Latency (ms): 47091.68 P90 E2E Latency (ms): 63943.69 P95 E2E Latency (ms): 67044.47 P99 E2E Latency (ms): 68407.32 ---------------Time to First Token---------------- Mean TTFT (ms): 3884.63 Median TTFT (ms): 3911.45 P90 TTFT (ms): 3912.78 P95 TTFT (ms): 3913.35 P99 TTFT (ms): 3913.52 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 18.04 Median TPOT (ms): 18.03 P90 TPOT (ms): 21.22 P95 TPOT (ms): 21.65 P99 TPOT (ms): 22.83 ---------------Inter-Token Latency---------------- Mean ITL (ms): 17.01 Median ITL (ms): 16.28 P90 ITL (ms): 24.90 P95 ITL (ms): 26.15 P99 ITL (ms): 49.02 Max ITL (ms): 567.29 ==================================================
Then the memory ceiling showed up at 64K
The path stayed promising until I shifted from write-heavy generation to long inputs. This capacity run used 30 prompts, exactly 64,000 input tokens and 500 generated tokens each, a maximum client concurrency of 20, and no warmup.
python -m sglang.benchmark.serving \ --backend sglang-oai \ --base-url http://127.0.0.1:18000 \ --model deepseek-ai/DeepSeek-V4-Flash \ --dataset-name random \ --num-prompts 30 \ --random-input-len 64000 \ --random-output-len 500 \ --random-range-ratio 1.0 \ --max-concurrency 20 \ --warmup-requests 0 \ --output-file /workspace/dsv4-c1-i180k-o200.jsonl
All 30 requests completed, processing 1.92 million input tokens and generating 15,000 output tokens. The completion count stayed clean, but the latency exposed the memory ceiling.
| Successful requests | 30 / 30 |
|---|---|
| Duration | 280.32 seconds |
| Input throughput | 6,849.41 tokens/second |
| Output throughput | 53.51 tokens/second |
| Reported average concurrency | 14.23 |
| Mean end-to-end latency | 132.94 seconds |
| Median end-to-end latency | 153.20 seconds |
| Mean time to first token | 121.92 seconds |
| Median time to first token | 141.51 seconds |
| Mean time per output token | 22.09 ms |
| MTP accept length | 2.54 |
Full benchmark output
============ Serving Benchmark Result ============ Backend: sglang-oai Traffic request rate: inf Max request concurrency: 20 Successful requests: 30 Benchmark duration (s): 280.32 Total input tokens: 1920000 Total input text tokens: 1920000 Total generated tokens: 15000 Total generated tokens (retokenized): 15000 Request throughput (req/s): 0.11 Input token throughput (tok/s): 6849.41 Output token throughput (tok/s): 53.51 Peak output token throughput (tok/s): 98.00 Peak concurrent requests: 21 Total token throughput (tok/s): 6902.92 Concurrency: 14.23 Accept length: 2.54 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 132942.08 Median E2E Latency (ms): 153198.71 P90 E2E Latency (ms): 187495.98 P95 E2E Latency (ms): 191277.56 P99 E2E Latency (ms): 194769.66 ---------------Time to First Token---------------- Mean TTFT (ms): 121919.26 Median TTFT (ms): 141510.63 P90 TTFT (ms): 176319.39 P95 TTFT (ms): 176798.78 P99 TTFT (ms): 181536.81 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 22.09 Median TPOT (ms): 21.79 P90 TPOT (ms): 38.18 P95 TPOT (ms): 38.77 P99 TPOT (ms): 40.28 ---------------Inter-Token Latency---------------- Mean ITL (ms): 22.09 Median ITL (ms): 7.05 P90 ITL (ms): 10.61 P95 ITL (ms): 10.77 P99 ITL (ms): 24.27 Max ITL (ms): 3776.63 ==================================================
The path ended at the memory cap
One 180K request was enough to consume basically the entire useful setup. At 64K, the server logs showed about two requests materially active on the GPUs at a time. The model may support a one-million-token request, but this machine did not have the runtime memory to make long context useful as a service.
The benchmark's reported concurrency of 14.23 counts requests across their full end-to-end lifetime, including requests waiting through chunked prefill. It does not mean 14 separate 64K contexts were resident and decoding together. The 121.92-second mean time to first token is the giveaway: most of that apparent concurrency was a queue.
The reason is simple. The model weights already occupy most of the 192 GB. What remains has to cover the KV cache, MTP weights, CUDA graphs, workspaces, and runtime buffers. A reported token pool of 186,624 maps almost perfectly to what I observed: one 180K job with almost no headroom, or two 64K jobs.
Maximum context is not serving capacity. A model card tells you what one request can address on a sufficiently large system. It does not tell you how many customers can use that context at once after the weights and runtime take their share.
The memory cap changed the economics
That is when these RTX PRO 6000 Blackwells started to feel like 5090s with extra VRAM. That is not a universal statement about Blackwell. It is what the RTX PRO 6000 value proposition looked like for this model. The cards had plenty of compute for a model activating 13B parameters per token. The promising path stopped on memory capacity and memory bandwidth. RTX PRO 6000 Blackwell cards do not have NVLink, so tensor-parallel traffic also had to use PCIe.
The extra VRAM only becomes comfortable around TP4. Four 96 GB cards give the model room to breathe, but now the machine is in roughly $50,000 territory. At that point I am buying a lot of extra compute just to solve a memory problem.
At the prices I was seeing, I would rather move to H200-class hardware. H200 systems offer HBM, much higher memory bandwidth, more memory per GPU, and platforms designed for this kind of tensor-parallel serving. Even with current AI-bubble pricing, that looked like better value than forcing four PCIe RTX PRO Blackwells into the job.
The verdict
This was genuinely promising, not a failed experiment. DeepSeek V4 Flash ran on two RTX PRO 6000 Blackwells with clean responses, working reasoning and tool parsing, MTP decode, and strong write-heavy throughput.
With short inputs and long outputs, the setup delivered 667.34 output tokens per second and an 18.04 ms mean TPOT. The software path worked. The compute was there. Memory was the cap.
At 64K, the average first token took 121.92 seconds. At 180K, one request filled the useful setup. For long-context agent workloads, $25,000 of hardware buys one or two useful GPU-resident slots and a long queue behind them.
TP2 showed that the idea could work. TP4 would give the model room to breathe, but it moves the machine into roughly $50,000 territory. That is where H200-class hardware starts looking like the better value, even at current AI-bubble pricing.