SQS dead-letter queues with Lambda: debugging and redrive

Messages are piling up in your dead-letter queue (DLQ) and the Lambda logs don’t obviously say why. Here is how messages get there when Lambda consumes an SQS queue, how to find the real cause, and how to send them back safely.

How messages reach the DLQ

A DLQ is attached to the source queue through its redrive policy. Every time a message is received, SQS increments its receive count. When the count exceeds maxReceiveCount, SQS moves the message to the DLQ instead of delivering it again.

With Lambda, a message is received each time the event source mapping hands it to your function. If the invocation fails — or if the message isn’t deleted for any other reason — it becomes visible again after the visibility timeout and is received again. So anything that causes repeated receives eventually lands a message in the DLQ, including problems that aren’t errors in your code.

resource "aws_sqs_queue" "orders_dlq" {
  name                      = "orders-dlq"
  message_retention_seconds = 1209600 # 14 days
}

resource "aws_sqs_queue" "orders" {
  name                       = "orders"
  visibility_timeout_seconds = 180 # at least 6x the function timeout
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.orders_dlq.arn
    maxReceiveCount     = 5
  })
}

Visibility timeout: the usual suspect

If the queue’s visibility timeout is shorter than your function’s run time, a message becomes visible again while the first invocation is still processing it. It gets received again, the receive count climbs, and healthy messages end up in the DLQ. AWS recommends setting the source queue’s visibility timeout to at least six times the function timeout, which leaves room for retries and throttling.

Partial batch failures

Lambda reads SQS in batches. By default, if your function throws for one message, the whole batch is returned to the queue — so the nine good messages are reprocessed along with the bad one, and all ten move toward the DLQ. Turn on partial batch responses and report only the failed message IDs:

public class OrderHandler implements RequestHandler<SQSEvent, SQSBatchResponse> {
  @Override
  public SQSBatchResponse handleRequest(SQSEvent event, Context context) {
    List<SQSBatchResponse.BatchItemFailure> failures = new ArrayList<>();
    for (SQSEvent.SQSMessage msg : event.getRecords()) {
      try {
        process(msg.getBody());
      } catch (Exception e) {
        context.getLogger().log("Failed " + msg.getMessageId() + ": " + e);
        failures.add(new SQSBatchResponse.BatchItemFailure(msg.getMessageId()));
      }
    }
    return new SQSBatchResponse(failures);
  }
}

resource "aws_lambda_event_source_mapping" "orders" {
  event_source_arn        = aws_sqs_queue.orders.arn
  function_name           = aws_lambda_function.orders.arn
  batch_size              = 10
  function_response_types = ["ReportBatchItemFailures"]
}

Both halves are required: the handler must return the failures, and the mapping must enable the feature.

Throttling and concurrency

If your function is throttled, the event source mapping can’t deliver messages and they return to the queue, still counting as receives. A low reserved concurrency on the function is a common cause. To limit how hard the queue hits a downstream system, set maximum concurrency on the SQS event source mapping instead — it caps concurrent invocations without throttling them.

Finding out why messages failed

SQS does not store why a message failed. You need your own breadcrumbs:

• Log the message ID and the exception for every failure (as in the handler above). Then search CloudWatch Logs for the IDs of messages sitting in the DLQ.
• Inspect a few DLQ messages — body and attributes — in the console or with the CLI. Patterns (one customer, one schema version, one size) point at the cause quickly.
• Check timing: if there are no errors logged for a message, suspect the visibility timeout or throttling rather than your code.
• Alarm on the DLQ with a CloudWatch alarm on ApproximateNumberOfMessagesVisible above zero, so you find out in minutes, not weeks.

Redriving safely

1. Fix and deploy the root cause first.
2. Make sure processing is idempotent — some DLQ messages may have partly succeeded the first time. Powertools for AWS Lambda includes an idempotency utility backed by DynamoDB.
3. Use SQS DLQ redrive (in the console, or the StartMessageMoveTask API) to move messages back to the source queue, ideally at a controlled rate.
4. Watch the source queue, function errors and the DLQ while the redrive runs.

Common pitfalls

• Lambda’s own DLQ setting doesn’t apply here. The function-level dead-letter queue and on-failure destinations are for asynchronous invocations; for SQS sources, the DLQ belongs on the queue.
• FIFO queues need a FIFO DLQ, and a failing message blocks its message group until it succeeds or moves to the DLQ.
• Set DLQ retention longer than the source queue’s. For standard queues, a message’s expiry is based on its original enqueue time, so it can expire soon after arriving in the DLQ.
• maxReceiveCount of 1 sends a message to the DLQ on its first hiccup; a value of three to five usually balances resilience and speed.