Simple Queue Service

Amazon Simple Queue Service provides a managed queueing service that allows decoupling services and serverless applications from one another. Messages can be sent by publishers, stored in the queue and received by consumers for processing. Any volume, no worry of message loss.

Benefits

  • Message storage redundant across multiple availability zones
  • Sensitive data is secured with KMS
  • Elastic scaling -- costs scale with demand, infinite scalability of standard queues
  • Resilience, because we can defer processing to a better time

Common use cases

  • Handling spiky workloads, e.g. voting systems
  • Deferring background processing, e.g. video transcoding

Queue types

  • Standard queues at least once, maybe multiple; best-effort ordering. This is the type that most AWS services support publishing to.
  • FIFO queues deliver messages in-order, exactly once. Limited number of transactions; 3,000 per second if using batching, 300 per second otherwise. No SNS. Messages are grouped per consumer, allowing multiple consumers to be added.

Polling

  • Short polling returns messages immediately; increased costs, subset of servers polled, so may be necessary to retry multiple times to hit the right server. weighted random distribution; 1,000 or less in two requests
  • Long polling fixes cost issues by waiting up to "Receive message wait time" seconds (up to 20s) when the queue is empty.

Durable delivery

Two things catch messages that couldn't be handled:

  • The Dead letter queue catches messages that couldn't be processed after a set number of attempts, allowing for debugging of failures.
  • The Visibility timeout allows controlling how long a message should remain invisible upon delivery to a consumer before becoming visible to other consumers. It must allow enough time to process and delete the message, else it'll be processed again.

To facilitate delivery, each Message gets an identifier in the SendMessageResponse. Messages get a new Receipt Handle for each read; the last one can be used by the consumer to delete the message. If it fails to delete it within the visibility timeout window, the message will become visible to other consumers.