LIVE

Serving Massive Mixture-of-Experts AI Models Locally Across Two Nvidia DGX Stations

🔌 Chips & Hardware 1 min read 👁 7

Engineers demonstrate how to run the 753-billion parameter GLM-5.3 AI model across two Nvidia DGX stations using vLLM and tensor parallelism.

Serving Massive Mixture-of-Experts AI Models Locally Across Two Nvidia DGX Stations🔌Chips & Hardware

Photo: Hacker News Front Page

Key points

  • GLM-5.3 features 753 billion parameters with 40 billion activated per token.
  • The 464 GB checkpoint exceeds single GB300 HBM3e memory, requiring a two-station setup.
  • Tensor parallelism splits tensors across two Nvidia DGX Stations using vLLM to achieve 607.8 tokens per second.

Running frontier-scale artificial intelligence models locally typically requires keeping all weights within GPU memory, but the largest architectures have outgrown single-machine capacity. A newly published engineering guide details how to serve Z.ai's GLM-5.3 NVFP4 model—a massive 753-billion parameter mixture-of-experts system with 40 billion active parameters per token—across two Nvidia DGX Stations.

Because the model checkpoint spans approximately 464 GB, it exceeds the 252 GB of HBM3e available on a single GB300 GPU. To overcome this memory constraint, the setup connects two DGX Stations using ConnectX-8 RDMA and applies tensor parallelism via vLLM. This configuration splits the weights and computational workload across both machines, using Station A as the primary API host and Station B as a dedicated worker rank.

By establishing direct network rails and synchronizing the massive checkpoint across both nodes, the deployment achieves a throughput of 607.8 tokens per second under concurrent load. The approach provides a practical blueprint for running heavy sparse-activation models locally without relying entirely on external cloud providers.

What to watch next: Watch for further optimizations in multi-node local serving frameworks as larger mixture-of-experts models continue to push hardware limits.

Source Hacker News Front Page — developer.nvidia.com/blog/when-one-gpu-isnt-enough-serving-glm-5-3-acro… ↗

Contact the newsroom

Got a tip, a proposal or a partnership question? Write to us and we will reply shortly.