钽TANDIU钽铥
SELF-HOSTING

Kimi K3: Self-Hosting Estimate

Kimi K3 (Moonshot AI) on different GPUs: configuration, estimated performance and use cases.

Data updated: 2026-09-16

← All self-hosting estimates

Total parameters2,780BSets weight VRAM
Active parameters104BSets decode compute and speed
Public benchmarksAA — · LiveBench 69.4
Vendor API price$3.00 / $15.00input / output per 1M tokens · cached $0.300 · Token prices →
Vendor API output speed—measured by Artificial Analysis; compare with local estimates below

B300 288GB

288 GB · 8,000 GB/s · Top-tier Blackwell card (no official channels to China)
Suggested precision
INT4 (AWQ/GPTQ)
GPUs needed
8priced as 1 × 8-GPU HGX node
Est. decode speed
1,231 tok/ssingle-stream theoretical ceiling
Est. batched throughput
out 2,092 tok/sin 11,250 tok/s · many concurrent requests
Hardware list price
$400,084–$440,093Cloud rental (live) ×8: $79.52–$100.01/hr (2026-11-20)
Hardware market price
$2,545,253–$2,994,415
Typical use
Enterprise-grade ultra-large-scale training and high-concurrency inference clusters

GB200 192GB (single card equivalent)

192 GB · 8,000 GB/s · Top-tier Blackwell card (no official channels to China)
Suggested precision
INT4 (AWQ/GPTQ)
GPUs needed
16 (exceeds one node: multi-node needed)
Est. decode speed
2,462 tok/ssingle-stream theoretical ceiling
Est. batched throughput
out 4,185 tok/sin 25,000 tok/s · many concurrent requests
Hardware list price
$479,490–$719,235
Hardware market price
$1,437,319–$2,874,639
Typical use
Ultra-large-scale trillion-parameter model training and inference clusters (networked with Grace CPU)

B200 180GB

180 GB · 8,000 GB/s · Top-tier Blackwell card (no official channels to China)
Suggested precision
INT4 (AWQ/GPTQ)
GPUs needed
16 (exceeds one node: multi-node needed)priced as 2 × 8-GPU HGX node
Est. decode speed
2,462 tok/ssingle-stream theoretical ceiling
Est. batched throughput
out 4,185 tok/sin 22,500 tok/s · many concurrent requests
Hardware list price
$720,150–$880,186Cloud rental (live) ×16: $132.43–$140.02/hr (2026-11-20)
Hardware market price
$898,325–$1,048,045
Typical use
Enterprise-grade large-scale online inference and training

H200 141GB

141 GB · 4,800 GB/s · Top-tier export-controlled card (channels to China are volatile)
Suggested precision
INT4 (AWQ/GPTQ)
GPUs needed
16 (exceeds one node: multi-node needed)priced as 2 × 8-GPU HGX node
Est. decode speed
1,477 tok/ssingle-stream theoretical ceiling
Est. batched throughput
out 2,511 tok/sin 9,890 tok/s · many concurrent requests
Hardware list price
$496,105–$640,134Cloud rental (live) ×16: $59.81–$114.11/hr (2026-11-20)
Hardware market price
$2,006,258–$2,066,147
Typical use
Enterprise-grade production environment high-concurrency inference

H100 SXM5 80GB

80 GB · 3,350 GB/s · Top-tier export-controlled card (no official channels to China)
Suggested precision
INT4 (AWQ/GPTQ)
GPUs needed
32 (exceeds one node: multi-node needed)priced as 4 × 8-GPU HGX node
Est. decode speed
2,062 tok/ssingle-stream theoretical ceiling
Est. batched throughput
out 3,505 tok/sin 19,780 tok/s · many concurrent requests
Hardware list price
$1,184,246–$1,236,258Cloud rental (live) ×32: $55.07–$139.46/hr (2026-11-20)
Hardware market price
$2,006,258
Typical use
Enterprise-grade production environment high-concurrency inference/training

H800 SXM 80GB

80 GB · 3,350 GB/s · Top-tier export-controlled card (discontinued)
Suggested precision
INT4 (AWQ/GPTQ)
GPUs needed
32 (exceeds one node: multi-node needed)priced as 4 × 8-GPU HGX node
Est. decode speed
2,062 tok/ssingle-stream theoretical ceiling
Est. batched throughput
out 3,505 tok/sin 19,780 tok/s · many concurrent requests
Hardware list price
$1,173,811–$1,245,677
Hardware market price
$1,616,984–$1,676,873
Typical use
Enterprise-grade production deployment (original China-compliant model, now discontinued)

H20 96GB

96 GB · 4,000 GB/s · Export-compliant special supply card (significantly reduced compute power)
Suggested precision
INT4 (AWQ/GPTQ)
GPUs needed
32 (exceeds one node: multi-node needed)
Est. decode speed
2,462 tok/ssingle-stream theoretical ceiling
Est. batched throughput
out 4,185 tok/sin 2,960 tok/s · many concurrent requests
Hardware list price
n/a
Hardware market price
$838,436
Typical use
Suboptimal choice in export-compliant environments, medium-to-large scale inference deployment (FP16 compute power is only about 15% of H100, but higher memory bandwidth offers an advantage in decode-intensive scenarios)

A800 PCIe 80GB

80 GB · 1,935 GB/s · Compliant special supply card (export compliance status adjusted multiple times)
Suggested precision
INT4 (AWQ/GPTQ)
GPUs needed
32 (exceeds one node: multi-node needed)
Est. decode speed
1,191 tok/ssingle-stream theoretical ceiling
Est. batched throughput
out 1,215 tok/sin 3,744 tok/s · many concurrent requests · PCIe multi-GPU penalty
Hardware list price
$479,106–$498,271
Hardware market price
$632,421
Typical use
Medium-scale inference/training deployment in export-compliant environments

Ascend 950DT 144GB

144 GB · 4,000 GB/s · Huawei's new-generation SuperNode card (domestic, parameters and prices for reference only)
Suggested precision
INT4 (AWQ/GPTQ)
GPUs needed
16 (exceeds one node: multi-node needed)
Est. decode speed
1,231 tok/ssingle-stream theoretical ceiling
Est. batched throughput
out 2,092 tok/sin 5,000 tok/s · many concurrent requests
Hardware list price
n/a
Hardware market price
$1,858,933
Typical use
Domestic large model inference SuperNode (64-card SuperPoD), report states inference performance is approx. 0.86x that of H200 at the same latency

Ascend 910C 128GB

128 GB · 3,200 GB/s · Huawei dual-chip data center card (domestically produced)
Suggested precision
INT4 (AWQ/GPTQ)
GPUs needed
16 (exceeds one node: multi-node needed)
Est. decode speed
985 tok/ssingle-stream theoretical ceiling
Est. batched throughput
out 1,674 tok/sin 7,520 tok/s · many concurrent requests
Hardware list price
n/a
Hardware market price
$191,643–$479,106
Typical use
Domestically produced large model training and inference clusters, CloudMatrix and other super-node solutions

Ascend 910B 64GB

64 GB · 400 GB/s · Domestic alternative (not subject to US export controls)
Suggested precision
INT4 (AWQ/GPTQ)
GPUs needed
32 (exceeds one node: multi-node needed)
Est. decode speed
246 tok/ssingle-stream theoretical ceiling
Est. batched throughput
out 418 tok/sin 6,400 tok/s · many concurrent requests
Hardware list price
n/a
Hardware market price
$527,017–$574,928
Typical use
Domestic alternative, not subject to US export controls, medium-to-large scale inference/training deployment, memory bandwidth significantly lower than NVIDIA cards of similar VRAM capacity

RTX PRO 6000 Blackwell 96GB

96 GB · 1,792 GB/s · Professional workstation/server card (China-specific version is 6000D 84GB)
Suggested precision
INT4 (AWQ/GPTQ)
GPUs needed
32 (exceeds one node: multi-node needed)
Est. decode speed
1,103 tok/ssingle-stream theoretical ceiling
Est. batched throughput
out 1,125 tok/sin 6,000 tok/s · many concurrent requests · PCIe multi-GPU penalty
Hardware list price
$513,228Cloud rental (live) ×32: $42.30–$64.22/hr (2026-11-20)
Hardware market price
$814,481
Typical use
Single-card large VRAM inference, local deployment of medium MoE models, private all-in-one machine

RTX PRO 5500 Blackwell 84GB

84 GB · 1,398 GB/s · Professional workstation card (specs are NVIDIA preliminary data)
Suggested precision
INT4 (AWQ/GPTQ)
GPUs needed
32 (exceeds one node: multi-node needed)
Est. decode speed
860 tok/ssingle-stream theoretical ceiling
Est. batched throughput
out 878 tok/sin 5,424 tok/s · many concurrent requests · PCIe multi-GPU penalty
Hardware list price
$364,892
Hardware market price
$407,240–$431,196
Typical use
Workstation large VRAM inference, departmental private deployment

RTX PRO 5000 Blackwell 48GB

48 GB · 1,344 GB/s · Professional workstation card
Suggested precision
INT4 (AWQ/GPTQ)
GPUs needed
64 (exceeds one node: multi-node needed)
Est. decode speed
1,654 tok/ssingle-stream theoretical ceiling
Est. batched throughput
out 1,687 tok/sin 6,192 tok/s · many concurrent requests · PCIe multi-GPU penalty
Hardware list price
n/aCloud rental (live) ×64: $38.53–$72.70/hr (2026-11-20)
Hardware market price
$275,735
Typical use
Small to medium model inference, development testing, single-machine multi-card private deployment

L40S 48GB

48 GB · 864 GB/s · Compliant inference card
Suggested precision
INT4 (AWQ/GPTQ)
GPUs needed
64 (exceeds one node: multi-node needed)
Est. decode speed
1,063 tok/ssingle-stream theoretical ceiling
Est. batched throughput
out 1,085 tok/sin 8,688 tok/s · many concurrent requests · PCIe multi-GPU penalty
Hardware list price
$431,196–$622,838Cloud rental (live) ×64: $29.89–$51.33/hr (2026-11-20)
Hardware market price
$431,196–$622,838
Typical use
SME teams building inference services, medium-low concurrency

L20 48GB

48 GB · 864 GB/s · Compliant inference card
Suggested precision
INT4 (AWQ/GPTQ)
GPUs needed
64 (exceeds one node: multi-node needed)
Est. decode speed
1,063 tok/ssingle-stream theoretical ceiling
Est. batched throughput
out 1,085 tok/sin 2,868 tok/s · many concurrent requests · PCIe multi-GPU penalty
Hardware list price
$223,714–$224,998
Hardware market price
$335,375
Typical use
SME teams building inference services, medium-low concurrency

RTX 5090 32GB

32 GB · 1,792 GB/s · Consumer flagship (not sold in China, 5090D for domestic market)
Suggested precision
INT4 (AWQ/GPTQ)
GPUs needed
64 (exceeds one node: multi-node needed)
Est. decode speed
2,206 tok/ssingle-stream theoretical ceiling
Est. batched throughput
out 2,250 tok/sin 10,056 tok/s · many concurrent requests · PCIe multi-GPU penalty
Hardware list price
$128,017Cloud rental (live) ×64: $29.18–$38.46/hr (2026-11-20)
Hardware market price
$459,942–$498,271
Typical use
Local inference and development for individuals/small teams, multi-card VRAM pooling for quantized MoE

RTX 5090D 24GB

24 GB · 1,344 GB/s · Consumer-grade/small-scale self-built
Suggested precision
INT4 (AWQ/GPTQ)
GPUs needed
70 (exceeds one node: multi-node needed)
Est. decode speed
1,809 tok/ssingle-stream theoretical ceiling
Est. batched throughput
out 1,845 tok/sin 7,796 tok/s · many concurrent requests · PCIe multi-GPU penalty
Hardware list price
$172,917
Hardware market price
$293,453–$419,218
Typical use
Personal development/small-scale verification/POC, not recommended for production-level high concurrency

RTX 4090D 24GB

24 GB · 1,008 GB/s · Consumer-grade/small-scale self-built
Suggested precision
INT4 (AWQ/GPTQ)
GPUs needed
70 (exceeds one node: multi-node needed)
Est. decode speed
1,357 tok/ssingle-stream theoretical ceiling
Est. batched throughput
out 1,384 tok/sin 7,718 tok/s · many concurrent requests · PCIe multi-GPU penalty
Hardware list price
$136,235
Hardware market price
$183,397
Typical use
Personal development/small-scale verification/POC, not recommended for production-level high concurrency
FAQ

FAQ

How is the estimated decode speed calculated?

Estimated speed = (memory bandwidth × GPU count) ÷ (active parameters × bytes per parameter): the memory-bound upper limit for one request (roofline). It scales linearly with GPU count and ignores interconnect overhead, batching and framework losses, so real results are usually well below it.

Can I trust the speed numbers for consumer GPUs?

The linear assumption is reasonable for NVLink data-center cards (H/B series) but too optimistic for consumer PCIe cards without NVLink (RTX 5090D/4090D); multi-GPU consumer setups lose far more to communication, so equal-looking numbers do not mean equal performance.

How is “Suggested precision” chosen?

It is the highest precision that fits within 8 GPUs; beyond that it is flagged as multi-node.

Where do the prices come from?

USD converted at ¥6.68 per USD. Cloud rental is the international vast.ai market rate, not a China purchase price.