Serving Massive Mixture-of-Experts AI Models Locally Across Two Nvidia DGX Stations
Engineers demonstrate how to run the 753-billion parameter GLM-5.3 AI model across two Nvidia DGX stations using vLLM and tensor parallelism.
🔌Chips & HardwarePhoto: Hacker News Front Page
Key points
- GLM-5.3 features 753 billion parameters with 40 billion activated per token.
- The 464 GB checkpoint exceeds single GB300 HBM3e memory, requiring a two-station setup.
- Tensor parallelism splits tensors across two Nvidia DGX Stations using vLLM to achieve 607.8 tokens per second.
Running frontier-scale artificial intelligence models locally typically requires keeping all weights within GPU memory, but the largest architectures have outgrown single-machine capacity. A newly published engineering guide details how to serve Z.ai's GLM-5.3 NVFP4 model—a massive 753-billion parameter mixture-of-experts system with 40 billion active parameters per token—across two Nvidia DGX Stations.
Because the model checkpoint spans approximately 464 GB, it exceeds the 252 GB of HBM3e available on a single GB300 GPU. To overcome this memory constraint, the setup connects two DGX Stations using ConnectX-8 RDMA and applies tensor parallelism via vLLM. This configuration splits the weights and computational workload across both machines, using Station A as the primary API host and Station B as a dedicated worker rank.
By establishing direct network rails and synchronizing the massive checkpoint across both nodes, the deployment achieves a throughput of 607.8 tokens per second under concurrent load. The approach provides a practical blueprint for running heavy sparse-activation models locally without relying entirely on external cloud providers.
What to watch next: Watch for further optimizations in multi-node local serving frameworks as larger mixture-of-experts models continue to push hardware limits.
Source Hacker News Front Page — developer.nvidia.com/blog/when-one-gpu-isnt-enough-serving-glm-5-3-acro… ↗
Read next
🚀Startups & FundingHarvard Study Reveals US Hyperscale Data Centers Consume Power Rivaling Entire States
A new Harvard study reveals that America's 403 hyperscale data centers consumed between 68 and 99 terawatt-hours of electricity during the 2024-25 study period, rivaling …
🚀Startups & FundingCalifornia Files Lawsuit Against OpenAI Over Autonomous AI Incidents
California Attorney General Rob Bonta has filed a lawsuit against OpenAI, alleging the company failed to control its autonomous AI agents and misled the public about safe…
🧠AI AnalysisPedestal AI Secures Seed Funding to Automate Logistics With Intelligent Agents
Pedestal AI has launched publicly with $4.7 million in seed backing to deploy agentic artificial intelligence aimed at streamlining logistics and supply chain operations.
🏢Big TechUnited Launch Alliance Investigates Vulcan Rocket Failure
United Launch Alliance is investigating a mission failure of its Vulcan rocket, which suffered from low booster performance shortly after liftoff. The incident raises que…
🌐Work & SocietyUS Southwest Hydropower Output Hits Record Lows Amid Persistent Drought
Hydropower generation in the US Southwest dropped 28% below the 10-year average this summer as record-low water levels in the Colorado River Basin threaten regional energ…
🏢Big TechYouTube Updates View Metrics to Include TV Co-Viewing Estimates
YouTube is incorporating estimated co-viewing data into its public view counts to better reflect group consumption on television sets.