ActiveMQ: How to handle broker failovers while using temporary queues

Question

On my JMS applications we use temporary queues on Producers to be able to receive replies back from Consumer applications.

I am facing exactly same issue on my end as mentioned in this thread: http://activemq.2283324.n4.nabble.com/jira-Created-AMQ-3336-Temporary-Destination-errors-on-H-A-failover-in-broker-network-with-Failover-tt-td3551034.html#a3612738

Whenever I restarted an arbitrary broker in my network, I was getting many errors like this in my Consumer application log while trying to send reply to a temporary queue:

javax.jms.InvalidDestinationException:
  Cannot publish to a deleted Destination: temp-queue://ID:...

Then I saw Gary's response there suggesting to use

jms.watchTopicAdvisories=false

as a url param on the client brokerURL. I promptly changed my client broker URLs with this additional parameter. However now I am seeing errors like this when I restart my brokers in network for this failover testing:

javax.jms.JMSException: 
  The destination temp-queue:
    //ID:client.host-65070-1308610734958-2:1:1 does not exist.

I am using ActiveMQ 5.5 version. And my client broker URL looks like this:

failover:(tcp://amq-host1:61616,tcp://amq-host2.tred.aol.com:61616,tcp://amq-host3:61616,tcp://amq-host4:61616)?jms.useAsyncSend=true&timeout=5000&jms.watchTopicAdvisories=false
 

Additionally here is my activemq config XML for one of the 4 brokers: amq1.xml

Can someone here please look into this problem and suggest me what mistake I am making in this setup.

Update:

To clarify further on how I am doing request-response in my code:

1. I already use a per producer destination (i.e. temporary queue) and set this in reply-to header of every message.
2. I am already sending a per message unique correlation identifier in JMSCorrelationID header.
3. As far as I know even Camel and Spring are also using temporary queue for request-response mechanism. Only difference is that Spring JMS implementation creates and destroys temporary queue for every message whereas I create temporary queue for the lifetime of the producer. This temporary queue is destroyed when client (producer) app shutsdown or by the AMQ broker when it realizes there are no active producer attached with this temporary queue.
4. I am already setting a message expiry on each message on Producer side so that message is not held up in a queue for too long (60 sec).

Marlite

Is the new JMSException just logged or thrown into your client code? Also, is the exception thrown on every message the client sends to the broker, or does the exception stop when the failover completes? (I.e. Is the exception only thrown during the time when the client is not connected?) – Johnniejohnny

There seems to be a couple of things you need to do in addition to the jms.watchTopicAdvisories=false, i.e. <broker advisorySupport="false"> in your XML configuration, and statically configuring your network. (your amq1.xml file gives me a 404 Not Found) – Stoplight

@Bringer128: Thanks for your comment. That JMS exception is thrown on the other AMQ broker where producer connects after reconnect. And once this happens JMS producer just stops receiving any responses from consumer since AMQ broker just cannot send the reply back to producer with the above JMS Exception. – Marlite

@opyate: Thanks for your suggestion. However just to let you know with advisorySupport="false" temporary destinations don't work in the current AMQ5.5 release even with the staticallyIncludedDestinations. However this feature has been added in upcoming 5.6 release as per my request. Pls see my conversation with Gary Tully here: activemq.2283324.n4.nabble.com/… for more details – Marlite

Answer 1

There is a broker attribute, org.apache.activemq.broker.BrokerService#cacheTempDestinations that should help in the failover: case. Set that to true in xml configuration, and a temp destination will not be removed immediately when a client disconnects. A fast failover: reconnect will be able to producer and/or consume from the temp queue again.

There is a timer task based on timeBeforePurgeTempDestinations (default 5 seconds) that handles cache removal.

One caveat though, I don't see any tests in activemq-core that make use of that attribute so I can't give you any guarantee on this one.

Cannonade

If you notice I wrote in my question: Whenever I restarted an arbitrary broker in my network. So I'm wondering if broker itself is being restarted will this broker attribute cacheTempDestinations have any effect? – Marlite

When you restart a broker, any temp destinations will along with any pending messages will be lost. There is no persistent state maintained by the broker for temporary destinations. – Cannonade

The best we can do is: 1) support failover: reconnect to an existing broker. 2) in the case of a network partition between brokers or a broker failure before a reply to a temp destination is sent, allow the temp destination to be auto created. – Cannonade

I am already using failover: in my JMS client's URL to connect to AMQ cluster. When you wrote allow the temp destination to be auto created: Did you mean allow the temp destination to be auto created on one of the live brokers in the event of some random broker dying? My question is how do I get this temp destination auto creation? – Marlite

The broker attribute allowTempAutoCreationOnSend was added in support of AMQ-3253. – Cannonade

Looking at this JIRA it seems this attribute allowTempAutoCreationOnSend has been added in AMQ 5.6.0 not in current 5.5.0? If that's the case I will have to wait until 5.6.0 is finally released since our Operations policy doesn't allow us to use beta, RC or snapshot releases. – Marlite

Answer 2

Temporary queues are created on the broker to which the requestor (producer) in your request-reply scenario connects. They are created from a javax.jms.Session, so on that session disconnecting, either because of client disconnect or broker failure/failover, those queues are permanently gone. None of the other brokers will understand what is meant when one of your consumers attempts to reply to those queues; hence your exception.

This requires an architectural shift in mindset assuming that you want to deal with failover and persist all your messages. Here is a general way that you could attack the problem:

1. Your reply-to headers should refer to a queue specific to the requestor process: e.g. queue:response.<client id>. The client id might be a standard name if you have a limited number of clients, or a UUID if you have a large number of these.
2. The outbound message should set a correlation identifier (simply a sting that lets you associate a request with a response - requestors after all might make more than one request at the same time). This is set in the JMSCorrelationID header, and ought to be copied from the request to the response message.
3. The requestor needs to set up a listener on that queue that will return the message body to the requesting thread based on that correllation id. There is some multithreading code that needs to be written for this, as you'll need to manually manage something like a map of correlation ids to originating threads (via Futures perhaps).

This is a similar approach to that taken by Apache Camel for request-response over messaging.

One thing to be mindful of is that the queue will not go away when the client does, so you should set a time to live on the response message such that it gets deleted from the broker if it has not been consumed, otherwise you will get a backlog of unconsumed messages. You will also need to set up a dead letter queue strategy to automatically discard expired messages.

Kono

Thanks for your detailed answer. I have added an update section in my question to respond to all of your points. Are you in by any chance suggesting me to not to use temporary queue per producer? – Marlite

That's right, temporary queues are not designed for failover - they are a contract between the client and broker via the session; the session sits on an individual connection. Camel does do request-replies via temporary queues by default, but that is often impractical for the reasons outlined, and as such supports static queues as a fallback to meet that quality of service (though it uses JMS selectors over queues for correlation, which should be avoided for performance reasons). – Kono

IMO it is incorrect to assume that temporary queues are not designed for failover. Also if a temporary queue is not used then how can you have a queue specific to the requestor process or in other words per Producer when you don't know all the producers in advance. – Marlite

If you do go down this path, requestors should name the queues they listen to in such a way that it does not clash. You might use a combination of uuid and actual user ids if your requestors are tied to something like a GUI instance, or a host name if your requestors are servers (I've seen both of these approaches used) - the trick is to ensure the queue name is unique. – Kono

If you notice all these are characteristics of a temporary queue. And moreover if every producer starts creating uniquely named queues, how will they be cleaned up if producer dies abruptly? – Marlite

You define a destination policy for these queues, using gcInactiveDestinations=true (false by default) and setting inactiveTimoutBeforeGC to an sensible timeout. Check out activemq.apache.org/per-destination-policies.html – Kono

I have actually experimented with both of these gc options in destinationPolicy and found that in a network of broker it doesn't work as it should (may be because of advisory subscriptions). Once I run these options without network of broker then gc happens smoothly. However I am willing to test it again when AMQ 5.6.0 is released. – Marlite