LIVE

Salesforce Optimizes Global AI Model Inference with Amazon SageMaker

🧠 AI Analysis 2 min read 👁 0

Salesforce has implemented cross-Region inference routing in Amazon SageMaker to manage AI model capacity across 30+ global regions more efficiently.

Salesforce Optimizes Global AI Model Inference with Amazon SageMaker🧠AI Analysis

Photo: AWS Machine Learning Blog

Key points

  • Salesforce deployed cross-Region inference routing to resolve capacity fragmentation across 30+ AWS regions.
  • The new system uses SageMaker-managed routers to direct traffic based on real-time endpoint health and availability.
  • The architecture supports data residency compliance while enabling rapid scaling of the Agentforce Foundation's model fleet.

Salesforce’s Agentforce Foundation team has integrated cross-Region inference routing within Amazon SageMaker to address the challenges of managing AI model capacity across more than 30 global regions. Previously, Salesforce relied on a custom-built internal gateway to handle traffic spikes, which suffered from a lack of real-time capacity signals and required traffic to traverse public networking, increasing the risk of cascading failures.

The new SageMaker-managed routing system allows Salesforce to consolidate its model serving infrastructure. By using a centralized router, the company can now direct requests to available endpoints across different AWS accounts and regions. This setup ensures that if a local endpoint reaches capacity or experiences errors, the system automatically shifts traffic to healthy, pre-deployed endpoints without requiring manual intervention.

Key operational improvements include:

  • Capacity Awareness: The router evaluates the health of destination endpoints based on recent request outcomes, including timeouts and 5xx errors.
  • Secure Networking: By utilizing SageMaker’s native routing, Salesforce avoids the need for traffic to leave the virtual private cloud via public gateways.
  • Scalability: The system allows for the grouping of model endpoints under logical targets, preventing localized issues from impacting unrelated AI services.

This implementation enables Salesforce to meet strict data residency requirements while maintaining the flexibility to scale its AI fleet rapidly as new business units are onboarded.

Source AWS Machine Learning Blog — aws.amazon.com/blogs/machine-learning/how-salesforce-is-advancing-globa… ↗

Contact the newsroom

Got a tip, a proposal or a partnership question? Write to us and we will reply shortly.