SELF-HOSTING # Qwen3.8 2.4T A95B: Self-Hosting Estimate Qwen3.8 2.4T A95B (Alibaba (Qwen)) on different GPUs: configuration, estimated performance and use cases. Data updated: 2026-09-16 [← All self-hosting estimates](https://www.tandiu.cn/en/local-deploy) Total parameters**2,446B**Sets weight VRAM Active parameters**95B**Sets decode compute and speed License**Qwen custom open-source license agreement (HF labeled license:other, non-standard OSI protocol, specific terms must be reviewed before use)**[Official repo →](https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B) Public benchmarks**No public benchmark yet** Vendor API price**$2.00 / $6.00**input / output per 1M tokens · [Token prices →](https://www.tandiu.cn/en/token-market) Vendor API output speed**—**measured by Artificial Analysis; compare with local estimates below ## B300 288GB 288 GB · 8,000 GB/s · Top-tier Blackwell card (no official channels to China) Suggested precision **INT4 (AWQ/GPTQ)** GPUs needed **8priced as 1 × 8-GPU HGX node** Est. decode speed **1,347 tok/s**single-stream theoretical ceiling Est. batched throughput **out 2,291 tok/s**in 12,316 tok/s · many concurrent requests Hardware list price **$299,058–$498,429**Cloud rental (live) ×8: $63.08–$63.08/hr (2026-11-04) Hardware market price **$1,047,167** Typical use Enterprise-grade ultra-large-scale training and high-concurrency inference clusters ## GB200 192GB (single card equivalent) 192 GB · 8,000 GB/s · Top-tier Blackwell card (no official channels to China) Suggested precision **INT4 (AWQ/GPTQ)** GPUs needed **8** Est. decode speed **1,347 tok/s**single-stream theoretical ceiling Est. batched throughput **out 2,291 tok/s**in 13,684 tok/s · many concurrent requests Hardware list price **$239,544–$359,316** Hardware market price **$718,058–$1,436,115** Typical use Ultra-large-scale trillion-parameter model training and inference clusters (networked with Grace CPU) ## B200 180GB 180 GB · 8,000 GB/s · Top-tier Blackwell card (no official channels to China) Suggested precision **INT4 (AWQ/GPTQ)** GPUs needed **16 (exceeds one node: multi-node needed)priced as 2 × 8-GPU HGX node** Est. decode speed **2,695 tok/s**single-stream theoretical ceiling Est. batched throughput **out 4,581 tok/s**in 24,632 tok/s · many concurrent requests Hardware list price **$598,116–$996,858**Cloud rental (live) ×16: $119.17–$200.16/hr (2026-11-05) Hardware market price **$897,572–$1,047,167** Typical use Enterprise-grade large-scale online inference and training ## H200 141GB 141 GB · 4,800 GB/s · Top-tier export-controlled card (channels to China are volatile) Suggested precision **INT4 (AWQ/GPTQ)** GPUs needed **16 (exceeds one node: multi-node needed)priced as 2 × 8-GPU HGX node** Est. decode speed **1,617 tok/s**single-stream theoretical ceiling Est. batched throughput **out 2,749 tok/s**in 10,827 tok/s · many concurrent requests Hardware list price **$559,673–$999,417**Cloud rental (live) ×16: $47.86–$73.71/hr (2026-11-05) Hardware market price **$448,786–$466,737** Typical use Enterprise-grade production environment high-concurrency inference ## H100 SXM5 80GB 80 GB · 3,350 GB/s · Top-tier export-controlled card (no official channels to China) Suggested precision **INT4 (AWQ/GPTQ)** GPUs needed **32 (exceeds one node: multi-node needed)priced as 4 × 8-GPU HGX node** Est. decode speed **2,257 tok/s**single-stream theoretical ceiling Est. batched throughput **out 3,837 tok/s**in 21,654 tok/s · many concurrent requests Hardware list price **$959,440–$3,326,057**Cloud rental (live) ×32: $61.89–$131.62/hr (2026-11-05) Hardware market price **$1,282,930** Typical use Enterprise-grade production environment high-concurrency inference/training ## H800 SXM 80GB 80 GB · 3,350 GB/s · Top-tier export-controlled card (discontinued) Suggested precision **INT4 (AWQ/GPTQ)** GPUs needed **32 (exceeds one node: multi-node needed)priced as 4 × 8-GPU HGX node** Est. decode speed **2,257 tok/s**single-stream theoretical ceiling Est. batched throughput **out 3,837 tok/s**in 21,654 tok/s · many concurrent requests Hardware list price **$1,172,828–$1,244,633** Hardware market price **$1,615,630–$1,675,468** Typical use Enterprise-grade production deployment (original China-compliant model, now discontinued) ## H20 96GB 96 GB · 4,000 GB/s · Export-compliant special supply card (significantly reduced compute power) Suggested precision **INT4 (AWQ/GPTQ)** GPUs needed **16 (exceeds one node: multi-node needed)** Est. decode speed **1,347 tok/s**single-stream theoretical ceiling Est. batched throughput **out 2,291 tok/s**in 1,620 tok/s · many concurrent requests Hardware list price **n/a** Hardware market price **$191,482–$263,288** Typical use Suboptimal choice in export-compliant environments, medium-to-large scale inference deployment (FP16 compute power is only about 15% of H100, but higher memory bandwidth offers an advantage in decode-intensive scenarios) ## A800 PCIe 80GB 80 GB · 1,935 GB/s · Compliant special supply card (export compliance status adjusted multiple times) Suggested precision **INT4 (AWQ/GPTQ)** GPUs needed **32 (exceeds one node: multi-node needed)** Est. decode speed **1,304 tok/s**single-stream theoretical ceiling Est. batched throughput **out 1,330 tok/s**in 4,099 tok/s · many concurrent requests · PCIe multi-GPU penalty Hardware list price **$478,705–$497,853** Hardware market price **$517,002–$574,446** Typical use Medium-scale inference/training deployment in export-compliant environments ## Ascend 910C 128GB 128 GB · 3,200 GB/s · Huawei dual-chip data center card (domestically produced) Suggested precision **INT4 (AWQ/GPTQ)** GPUs needed **16 (exceeds one node: multi-node needed)** Est. decode speed **1,078 tok/s**single-stream theoretical ceiling Est. batched throughput **out 1,832 tok/s**in 8,232 tok/s · many concurrent requests Hardware list price **n/a** Hardware market price **$191,482–$478,705** Typical use Domestically produced large model training and inference clusters, CloudMatrix and other super-node solutions ## Ascend 910B 64GB 64 GB · 400 GB/s · Domestic alternative (not subject to US export controls) Suggested precision **INT4 (AWQ/GPTQ)** GPUs needed **32 (exceeds one node: multi-node needed)** Est. decode speed **269 tok/s**single-stream theoretical ceiling Est. batched throughput **out 458 tok/s**in 7,006 tok/s · many concurrent requests Hardware list price **n/a** Hardware market price **$526,576–$574,446** Typical use Domestic alternative, not subject to US export controls, medium-to-large scale inference/training deployment, memory bandwidth significantly lower than NVIDIA cards of similar VRAM capacity ## RTX PRO 6000 Blackwell 96GB 96 GB · 1,792 GB/s · Professional workstation/server card (China-specific version is 6000D 84GB) Suggested precision **INT4 (AWQ/GPTQ)** GPUs needed **16 (exceeds one node: multi-node needed)** Est. decode speed **604 tok/s**single-stream theoretical ceiling Est. batched throughput **out 616 tok/s**in 3,284 tok/s · many concurrent requests · PCIe multi-GPU penalty Hardware list price **$256,399**Cloud rental (live) ×16: $19.25–$36.30/hr (2026-11-05) Hardware market price **$281,383** Typical use Single-card large VRAM inference, local deployment of medium MoE models, private all-in-one machine ## RTX PRO 5500 Blackwell 84GB 84 GB · 1,398 GB/s · Professional workstation card (specs are NVIDIA preliminary data) Suggested precision **INT4 (AWQ/GPTQ)** GPUs needed **32 (exceeds one node: multi-node needed)** Est. decode speed **942 tok/s**single-stream theoretical ceiling Est. batched throughput **out 961 tok/s**in 5,938 tok/s · many concurrent requests · PCIe multi-GPU penalty Hardware list price **$364,587** Hardware market price **$406,899–$430,835** Typical use Workstation large VRAM inference, departmental private deployment ## RTX PRO 5000 Blackwell 48GB 48 GB · 1,344 GB/s · Professional workstation card Suggested precision **INT4 (AWQ/GPTQ)** GPUs needed **32 (exceeds one node: multi-node needed)** Est. decode speed **905 tok/s**single-stream theoretical ceiling Est. batched throughput **out 924 tok/s**in 3,389 tok/s · many concurrent requests · PCIe multi-GPU penalty Hardware list price **n/a**Cloud rental (live) ×32: $17.95–$34.21/hr (2026-11-05) Hardware market price **$258,501** Typical use Small to medium model inference, development testing, single-machine multi-card private deployment ## L40S 48GB 48 GB · 864 GB/s · Compliant inference card Suggested precision **INT4 (AWQ/GPTQ)** GPUs needed **32 (exceeds one node: multi-node needed)** Est. decode speed **582 tok/s**single-stream theoretical ceiling Est. batched throughput **out 594 tok/s**in 4,756 tok/s · many concurrent requests · PCIe multi-GPU penalty Hardware list price **$215,417–$311,158**Cloud rental (live) ×32: $19.23–$29.18/hr (2026-11-05) Hardware market price **$215,417–$311,158** Typical use SME teams building inference services, medium-low concurrency ## L20 48GB 48 GB · 864 GB/s · Compliant inference card Suggested precision **INT4 (AWQ/GPTQ)** GPUs needed **32 (exceeds one node: multi-node needed)** Est. decode speed **582 tok/s**single-stream theoretical ceiling Est. batched throughput **out 594 tok/s**in 1,570 tok/s · many concurrent requests · PCIe multi-GPU penalty Hardware list price **$111,763–$112,405** Hardware market price **$167,547** Typical use SME teams building inference services, medium-low concurrency ## RTX 5090 32GB 32 GB · 1,792 GB/s · Consumer flagship (not sold in China, 5090D for domestic market) Suggested precision **INT4 (AWQ/GPTQ)** GPUs needed **64 (exceeds one node: multi-node needed)** Est. decode speed **2,414 tok/s**single-stream theoretical ceiling Est. batched throughput **out 2,463 tok/s**in 11,009 tok/s · many concurrent requests · PCIe multi-GPU penalty Hardware list price **n/a**Cloud rental (live) ×64: $28.29–$34.05/hr (2026-11-05) Hardware market price **$430,825** Typical use Local inference and development for individuals/small teams, multi-card VRAM pooling for quantized MoE ## RTX 5090D 24GB 24 GB · 1,344 GB/s · Consumer-grade/small-scale self-built Suggested precision **INT4 (AWQ/GPTQ)** GPUs needed **64 (exceeds one node: multi-node needed)** Est. decode speed **1,811 tok/s**single-stream theoretical ceiling Est. batched throughput **out 1,847 tok/s**in 7,803 tok/s · many concurrent requests · PCIe multi-GPU penalty Hardware list price **$157,963**Cloud rental (live) ×64: $28.29–$33.92/hr (2026-11-05) Hardware market price **$268,075–$382,964** Typical use Personal development/small-scale verification/POC, not recommended for production-level high concurrency ## RTX 4090D 24GB 24 GB · 1,008 GB/s · Consumer-grade/small-scale self-built Suggested precision **INT4 (AWQ/GPTQ)** GPUs needed **64 (exceeds one node: multi-node needed)** Est. decode speed **1,358 tok/s**single-stream theoretical ceiling Est. batched throughput **out 1,385 tok/s**in 7,724 tok/s · many concurrent requests · PCIe multi-GPU penalty Hardware list price **$124,454–$181,898**Cloud rental (live) ×64: $21.63–$27.90/hr (2026-11-05) Hardware market price **$248,927–$287,223** Typical use Personal development/small-scale verification/POC, not recommended for production-level high concurrency FAQ ## FAQ **How is the estimated decode speed calculated?** Estimated speed = (memory bandwidth × GPU count) ÷ (active parameters × bytes per parameter): the memory-bound upper limit for one request (roofline). It scales linearly with GPU count and ignores interconnect overhead, batching and framework losses, so real results are usually well below it. **Can I trust the speed numbers for consumer GPUs?** The linear assumption is reasonable for NVLink data-center cards (H/B series) but too optimistic for consumer PCIe cards without NVLink (RTX 5090D/4090D); multi-GPU consumer setups lose far more to communication, so equal-looking numbers do not mean equal performance. **How is “Suggested precision” chosen?** It is the highest precision that fits within 8 GPUs; beyond that it is flagged as multi-node. **Where do the prices come from?** USD converted at ¥6.68 per USD. Cloud rental is the international vast.ai market rate, not a China purchase price.