Guaranteed Delivery using Dead Letter Queue

When an enterprise implements messaging, Guaranteed Delivery (Messages are persistent and are not lost even when the system crashes) becomes an imminent requirement.

When implementing messaging, ensuring Guaranteed Delivery means answers to the following:

• Where does the message get sent when a condition is not met?
• Can each individual message be re-delivered?
• What happens to the message when an exception is thrown?

Before we answer these questions, let’s crack into “Dead Letter Queue”

What is “Dead Letter Queue”?

Dead Letter Queue is a service to store messages when:

• The message that is sent to a queue doesn’t exist
• Message length exceeds the limit set on Queue.
• Message is rejected, by another queue or exchange.

Consider a simple example of accepting an http Inbound request, storing the message and then processing the request. 

In the Publisher Flow (above diagram), Http Inbound connector delivers the message to the AMQP Outbound Connector, which connects to RabbitMQ.

<amqp:outbound-endpoint

          exchangeName="${rabbitmq.exchange.name}"

          queueName="${rabbitmq.sendemail.message.queue}"

          routingKey="${rabbitmq.sendemail.message.routingkey}"

          exchangeType="direct"

          exchangeDurable="true"

          queueDurable="true"

          responseTimeout="${rabbitmq.response.timeout}"

          connector-ref="AMQP_0_9_Connector"

          doc:name="Publisher Flow">

          <reconnect frequency="${rabbitmq.reconnect.frequency}" count="${rabbitmq.reconnect.count}" />          

</amqp:outbound-endpoint>

The “reconnect” element ensures that the delivery of messages is re-delivered in case of connection errors to RabbitMQ.

The Subscriber Flow is configured with an AMQP Inbound Connector, listening to any messages received and then delivers the message to Application Flow – to perform business logic.

<amqp:inbound-endpoint

          exchangeName="${rabbitmq.exchange.name}"

          queueName="${rabbitmq.sendemail.message.queue}"

          routingKey="${rabbitmq.sendemail.message.routingkey}"

          exchangeDurable="true"

          exchangeType="direct"

          queueDurable="true"

          responseTimeout="${rabbitmq.response.timeout}"

          connector-ref="AMQP_0_9_Connector"

          doc:name="Subscriber Flow">

          <reconnect frequency="${rabbitmq.reconnect.frequency}" count="${rabbitmq.reconnect.count}" />  

          <properties>

                    <spring:entry key="amqp-queue.x-dead-letter-exchange" value="${rabbitmq.exchange.name}" />

                    <spring:entry key="amqp-queue.x-dead-letter-routing-key" value="${rabbitmq.sendemail.message.dead.letter.routingkey}" />

          </properties>

</amqp:inbound-endpoint>

In the Subscriber flow, we have configured 2 additional properties to configure the dead letter queue. Messages will be sent to “Dead Letter Queue”, when the given condition is not met.

Until, this point we have answered 2 major questions:

Where does the message get sent when a condition is not met?

How to configure re-delivery of messages?

Let’s look at what happens when there is an exception in the application flow (in the above diagram) and where does the message get sent.

Typically, on successful processing of message, a positive “Acknowledgement” is sent to the queue, which then safely deletes the message from the queue.

<amqp:acknowledge-message  doc:name="AMQP-0-9 Acknowledge Message" />

But, when an exception occurs a “Reject” acknowledgement needs to be sent to the queue so that the message is rolled back into the queue for further use.

<exception-strategy ref="global-choice-rollback-exception-strategy" doc:name="Reference Exception Strategy" />

A rollback exception strategy is configured as:

<choice-exception-strategy name="global-choice-rollback-exception-strategy">

          <rollback-exception-strategy maxRedeliveryAttempts="${exception.strategy.rollback.attempts}">

          <choice>

                    <when expression="#[message.inboundProperties['http.status'] != null &amp;&amp; Integer.parseInt(message.inboundProperties['http.status']) != 200]">

                    <flow-ref name="perform-api-error-event-logging" doc:name="perform-api-error-event-logging" />

                    <set-property propertyName="Content-Type" value="application/json" doc:name="Property" />

                    <set-payload value="#[flowVars.payloadJson]" doc:name="Set Response" />

                    <json:object-to-json-transformer doc:name="Object to JSON" />

                    </when>

          <otherwise>

                    <logger message="No other exception handled" level="INFO" doc:name="Payload Logger" />

          </otherwise>

          </choice>

          <!-- re-queue message for re-tries -->

<amqp:reject-message requeue="true" doc:name="AMQP-0-9 Reject Message" />

             <!-- reject message, this will deliver message to DLQ -->

                    <on-redelivery-attempts-exceeded>

                              <amqp:reject-message doc:name="AMQP-0-9

                                        Reject Message" />

                    </on-redelivery-attempts-exceeded>

</rollback-exception-strategy>

</choice-exception-strategy>

The rejected messages in a queue can be viewed in the queue browser (like RabbitMQ Admin Console). The rejected messages can then be processed manually after taking corrective actions or the recovery process can be automated.

In conclusion, we have looked at how to implement Guaranteed Delivery of messages and answered the core questions of Guaranteed Delivery.