LIVE

AssemblyAI Optimizes GPU Inference Costs on AWS Using Spot Instances

🔌 Chips & Hardware 2 min read 👁 10

AssemblyAI has successfully scaled its speech-to-text AI infrastructure on Amazon Web Services by utilizing Amazon EC2 Spot Instances and a queue-based retry architecture.

AssemblyAI Optimizes GPU Inference Costs on AWS Using Spot Instances🔌Chips & Hardware

Photo: AWS Machine Learning Blog

Key points

  • AssemblyAI scales speech-to-text inference using Amazon EC2 Spot Instances.
  • An SQS-based queue and Lambda handler manage interruptions gracefully without dropping jobs.
  • The architecture complies with European data residency requirements for regulated industries.

Speech-to-text processing requires substantial computational power, relying heavily on NVIDIA GPUs to run deep learning pipelines for acoustic and language models. For high-volume platforms processing thousands of audio files, running entirely on standard On-Demand infrastructure can become financially unsustainable. To address this scaling challenge, AssemblyAI reconfigured its speech recognition and analysis platform to leverage a combination of Amazon Elastic Compute Cloud (EC2) Spot Instances, Amazon Elastic Container Service (ECS), and Amazon Simple Queue Service (SQS) across European AWS regions.

Because cloud providers can reclaim Spot capacity with a brief two-minute warning, building a resilient error-handling system was critical to protect customer service level agreements. The newly implemented architecture uses an Amazon SQS queue-based retry mechanism alongside an AWS Lambda function triggered by Amazon EventBridge. When a GPU instance receives a termination notice, active transcription jobs are safely returned to the queue to be picked up by another healthy task, while maintaining strict data residency rules required by regulated sectors like healthcare and finance.

To balance reliability with cost efficiency, the system pairs an always-on baseline of On-Demand instances with a diversified pool of Spot instances across multiple GPU families. Amazon ECS managed scaling automatically adjusts capacity based on incoming demand. What to watch next: How other AI infrastructure providers adopt similar hybrid cloud strategies to mitigate rising GPU compute expenses.

Source AWS Machine Learning Blog — aws.amazon.com/blogs/machine-learning/cost-optimizing-assemblyai-speech… ↗

Contact the newsroom

Got a tip, a proposal or a partnership question? Write to us and we will reply shortly.