When Dead Letter Queue becomes a zombie

    Dead Letter Queue (DLQ) defined within JMS provider is known to be a basket for messages that cannot be delivered to a particular destination. A typical scenario is that a DLQ is defined, a consumer of dead messages is attached and everything seems alright. It gives some kind of mental comfort. But is it real?

I have used JBoss 7.1.1 with HornetQ 2.2.13 JMS provider (bundled with JBoss) and Message Driven Bean (MDB) listener to present my observation.

The most important part of the configuration is defined as follows (standalone.xml):

<address-settings>
  <address-setting match="#"><
    <dead-letter-address>jms.queue.DLQ</dead-letter-address>
    <expiry-address>jms.queue.ExpiryQueue</expiry-address>
    <redelivery-delay>0</redelivery-delay>
    <max-delivery-attempts>3</max-delivery-attempts>
    <max-size-bytes>10485760</max-size-bytes>
    <address-full-policy>BLOCK</address-full-policy>
    <message-counter-history-day-limit>10</message-counter-history-day-limit>
  </address-setting>
</address-settings>
...
<pooled-connection-factory name="hornetq-ra">
  <transaction mode="xa"/>
    <connectors>
      <connector-ref connector-name="in-vm"/>
    </connectors>
    <entries>
      <entry name="java:/JmsXA"/>
    </entries>
</pooled-connection-factory>
... 
<jms-destinations>
 <jms-queue name="DLQ">
    <entry name="queue/dlq"/>
    <entry name="java:jboss/exported/jms/queue/dlq"/>
  </jms-queue>
  <jms-queue name="testQueue">
    <entry name="queue/test"/>
    <entry name="java:jboss/exported/jms/queue/test"/>
  </jms-queue>
</jms-destinations> 

The code:

Test queue consumer:

@MessageDriven(name = "TestMessageMDB", activationConfig = {
@ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue"),
@ActivationConfigProperty(propertyName = "destination", propertyValue = "java:/queue/test"),
@ActivationConfigProperty(propertyName = "acknowledgeMode", propertyValue = "Auto-acknowledge") })
public class TestMessageConsumer implements MessageListener {

    @Override
    public void onMessage(Message message) {
        throw new RuntimeException(); // OK, we are prepared for that (max delivery attempts and DLQ)
    }
}

DLQ consumer:

@MessageDriven(name = "DeadLetterMDB", activationConfig = {
@ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue"),
@ActivationConfigProperty(propertyName = "destination", propertyValue = "java:/queue/dlq"),
@ActivationConfigProperty(propertyName = "acknowledgeMode", propertyValue = "Auto-acknowledge") })
public class DeadLetterConsumer implements MessageListener {

    @Override
    public void onMessage(Message message) {
        throw new RuntimeException(); // are we prepared for that?!
    }
}

After sending a message to java:/queue/test we have... BOOM! Evil dead is awoken from its grave. The JMS provider tries to redeliver the message to DLQ listener indefinitely - it is a good livelock example. It consumes CPU cycles and it occupies the server's disk space. Within minutes our log file grows to the size of GBs. The root of all evil is bad configuration. DLQ is nothing special - it is an ordinary queue which is under the law of the configuration. # sign means that the configuration is applied to all of the destinations. It implicates that DLQ has the same DLQ which has the same DLQ and so on. After changing the configuration to:

<address-settings>
  <address-setting match="jms.queue.testQueue">
  <dead-letter-address>jms.queue.DLQ</dead-letter-address>
  <expiry-address>jms.queue.ExpiryQueue</expiry-address>
  <redelivery-delay>0</redelivery-delay>
  <max-delivery-attempts>3</max-delivery-attempts>
  <max-size-bytes>10485760</max-size-bytes>
  <address-full-policy>BLOCK</address-full-policy>
  <message-counter-history-day-limit>10</message-counter-history-day-limit>
</address-setting>
<address-setting match="jms.queue.DLQ"> // no DLQ for DLQ!
  <max-delivery-attempts>3</max-delivery-attempts>
  <max-size-bytes>10485760</max-size-bytes>
  <address-full-policy>BLOCK</address-full-policy>
  <message-counter-history-day-limit>10</message-counter-history-day-limit>
  </address-setting>
</address-settings>

HornetQ behaves reasonably:

Message has reached maximum delivery attempts, 

sending it to Dead Letter Address jms.queue.DLQ from jms.queue.testQueue.

and:

Message has exceeded max delivery attempts. 

No Dead Letter Address configured for queue jms.queue.DLQ so dropping it.

Dealing with JMS can be dangerous. Two important things should be remembered:

1. Message delivery to DLQ consumer can be unsuccessful - we have to deal with such situations.

2. Messages which are routed to DLQ should be consumed in order to reclaim memory. They cannot stay in a queue forever - OutOfMemoryError is our enemy.