LIVE
Pedestal AI Secures Seed Funding to Automate Logistics With Intelligent Agents United Launch Alliance Investigates Vulcan Rocket Failure US Southwest Hydropower Output Hits Record Lows Amid Persistent Drought YouTube Updates View Metrics to Include TV Co-Viewing Estimates OpenAI Partners with Chilean Ministry to Advance Public Sector AI Ambience Healthcare Expands Clinical AI Suite for Hospital Operations AI Models Vulnerable to Manipulation in Simulated Warfare Scenarios, Research Shows AWS Highlights Enterprise Governance and AI Workspace Quick Ahead of re:Invent 2026 LoanPro Integrates Amazon Bedrock to Automate Loan Servicing Workflows Google Brings Wallet App to iOS Users Across the United States Pedestal AI Secures Seed Funding to Automate Logistics With Intelligent Agents United Launch Alliance Investigates Vulcan Rocket Failure US Southwest Hydropower Output Hits Record Lows Amid Persistent Drought YouTube Updates View Metrics to Include TV Co-Viewing Estimates OpenAI Partners with Chilean Ministry to Advance Public Sector AI Ambience Healthcare Expands Clinical AI Suite for Hospital Operations AI Models Vulnerable to Manipulation in Simulated Warfare Scenarios, Research Shows AWS Highlights Enterprise Governance and AI Workspace Quick Ahead of re:Invent 2026 LoanPro Integrates Amazon Bedrock to Automate Loan Servicing Workflows Google Brings Wallet App to iOS Users Across the United States

Google Open-Sources TPU Sync to Tackle Memory Bottlenecks in Large Model Serving

🧠 AI Analysis 2 min read 👁 8

Google Cloud has released TPU Sync, an open-source C++ library designed to streamline KV cache transfers and lower inference latency on Cloud TPUs.

🧠AI Analysis

Key points

  • Google Cloud open-sourced TPU Sync, a C++ library aimed at speeding up memory transfers on Cloud TPUs.
  • Internal testing demonstrated up to 90 percent reductions in transfer latency for models up to 480 billion parameters.
  • The software supports PyTorch and JAX on TPU v7x and v6e, though Google cautions it is not yet intended for general production use.

Google Cloud has released TPU Sync, an open-source C++ library designed to accelerate key-value (KV) cache transfers and model weight distribution across Cloud TPUs. Released under an Apache-2.0 license, the software optimizes memory transfers between hardware high-bandwidth memory, host RAM, and data center networks, addressing a growing latency bottleneck in running large artificial intelligence workloads.

The library integrates directly with JAX and PyTorch frameworks through native Python bindings, bypassing XLA compilation in its data path. In Google's internal laboratory tests on Cloud TPU v7x (Ironwood) and v6e hardware, the tool achieved up to a 90 percent reduction in time to first token (TTFT) for Mixture-of-Experts models ranging from 235 billion to 480 billion parameters. In one benchmark using a 235-billion parameter model, KV transfer latency dropped from several seconds to roughly 230 milliseconds.

By treating onboard TPU memory, host system memory, and network storage as a single memory hierarchy, TPU Sync allows asynchronous data transfers to run simultaneously alongside active compute operations. This design minimizes delays during disaggregated prefill and decode inference as well as reinforcement learning training cycles. Google cautioned in the project repository that TPU Sync remains in an early stage and is not yet recommended for general production deployments.

Source MarkTechPost — marktechpost.com/2026/11/13/google-tpu-sync-kv-cache-transfer-libra… ↗

Contact the newsroom

Got a tip, a proposal or a partnership question? Write to us and we will reply shortly.