AWS Lambda with SQS trigger failure handling using DLQ

What is a Dead letter queue ?

A Dead letter queue is an Amazon SQS queue which can hold all the messages that failed to process. Messages that can’t be delivered due to client errors or server errors are held in the Dead letter queue for future analysis or reprocessing of the message.

Dead Letter Queue setup:

One primary thing that you need to note is that the Dead letter queue should be the same type of your primary Queue.

• At first setup SQS trigger to your Lambda function. To know how to setup check our previous article: AWS Lambda with SQS — Setup SQS trigger to Lambda

Create a simple queue of same type as primary queue:

• I have named the queue as ‘dead letter queue’ for easy understanding purpose.
• Keep other configuration as default and click create queue. If you want to know more about these configuration, check our previous article: [AWS] Tutorial SQS (Simple Queue Service).
• Add the created queue as a Dead letter queue to the primary queue.

• Go to your primary queue configuration and enable the Dead letter queue and select the new queue we created.
• Maximum receives — This denotes the number of retries for a particular message, for example: if the maximum receives is 2, and the message fails to process at the first time, then it will retry to process the message another time and if it fails again then it will be added to the Dead letter queue. I have set maximum receive as 1.
• Now go to the trigger configuration in AWS Lambda that we have already created.

• Make sure that the report batch item failures checkbox is checked in.

• This feature helps us to push only the failed items to the Dead letter queue. For example, when a batch of messages are sent to lambda to process and only one message fails the whole batch will be added to the Dead letter queue if this checkbox was not checked.
• Click save and proceed.
• In lambda we have to return a object in a particular format.

{
  batchItemFailures: [{
                        itemIdentifier: "messageid1",    
                       },
                       {
                        itemIdentifier: "messageid1",    
                       }]
}

• It should return an object which contains a batchItemFailures array with objects which has itemIdentifier mapped to message Id.

• In the lambda function, I have made our action as promise and based on the promise result status we are creating a response object in the format that I have mentioned above.

• I have added a new message which will not pass our condition (message body is not ‘hi’ ). Thus the promise will give the status as true so that the message will not be added to our Dead letter queue and it will be deleted from our main queue automatically (Note: This is the behaviour of the lambda with SQS Architecture. If the lambda responds with empty batchItemFailure array then the messages processed by that lambda is considered as successful process and the messages will be deleted from primary queue).

• As you can see that there are no messages in our primary queue and in the Dead letter queue. Let us check the CloudWatch logs to confirm this.

• From the logs you can see that the record has arrived and the response object is with empty batchItemfailure array.
• Lets add new message as ‘hi’. Which will pass our condition and promise status will be false thus the message has to be added to the Dead letter queue.

• Message has been added. Let us check the logs:

• You can see that the message passed the condition and the response object has batchItemFailure with an object with message id as the itemIdentifier. Thus it should have added the message to the Dead letter queue.
• Lets check the number of messages in our Dead letter queue.

• The message has been successfully added to the Dead letter queue. Let us poll the messages from the Dead letter queue and check whether it is the same message.

• As you can see that the message id matches and the message body is ‘hi’.

And with that we have learned how to setup a Dead Letter Queue to our SQS trigger with lambda architecture.

About the Author

Sangar Naarayanan R is a Software Development Engineer in CodeStax.Ai. With a hands-on experience and good knowledge in various technologies and looking forward how to upgrade his own skills by researching every day and provide the best possible solutions.

About CodeStax.Ai

At CodeStax.Ai, we stand at the nexus of innovation and enterprise solutions, offering technology partnerships that empower businesses to drive efficiency, innovation, and growth, harnessing the transformative power of no-code platforms and advanced AI integrations.

But the real magic? It’s our tech tribe behind the scenes. If you’ve got a knack for innovation and a passion for redefining the norm, we’ve got the perfect tech playground for you. CodeStax.Ai offers more than a job — it’s a journey into the very heart of what’s next. Join us, and be part of the revolution that’s redefining the enterprise tech landscape.