AWS Introduces Usage-Based Metering for SageMaker AI Bidirectional Streaming

Amazon Web Services has rolled out a new usage-based metering feature for Amazon SageMaker AI bidirectional (bidi) streaming inference. The update addresses a long-standing software billing challenge for real-time speech artificial intelligence, allowing model providers on the AWS Marketplace to bill customers based on exact consumption during live streaming sessions rather than relying on flat hourly fees or endpoint uptime.

While infrastructure costs for hosting models on SageMaker AI endpoints remain billed separately, the new framework allows software charges to scale directly with usage. Standard request-response APIs typically rely on HTTP response headers to report billing quantities, but live bidirectional streams—which can span anywhere from half a minute to over half an hour via HTTP/2 and WebSockets—accumulate metrics continuously while active.

To solve this, the infrastructure opens a companion WebSocket connection alongside the inference stream to transmit JSON-formatted metering records. Real-time speech AI provider Deepgram acts as the launch partner for the integration, bringing consumption-based billing for Speech-to-Text and Text-to-Speech services to the marketplace. Model servers send incremental usage metrics during active sessions, which AWS Marketplace then maps to predefined pricing tiers.

What to watch next: Adoption of usage-based streaming metrics by additional enterprise speech AI vendors on AWS Marketplace.