jms producer performance with spring

Question

i created a simple producer consumer simulation based on spring, jms and activemq, i'm trying to reach high performance from both sides, producers and consumers,

Connection settings :

<tx:annotation-driven />
<bean id="transactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
     <property name="connectionFactory"  ref="connectionFactory" />
</bean>

<amq:connectionFactory id="amqConnectionFactory" brokerURL="failover:(tcp://${broker.url}:61616)"  />

<bean id="connectionFactory"
    class="org.springframework.jms.connection.CachingConnectionFactory">
    <property name="targetConnectionFactory" ref="amqConnectionFactory" />
</bean>

<amq:queue id="queue" physicalName="queue" />

<beans:bean id="jsonMessageConverter" class="XXXXX.converter.JsonMessageConverter" />

Consumer settings :

<jms:listener-container concurrency="10"
    acknowledge="auto" prefetch="1" message-converter="jsonMessageConverter" transaction-manager="transactionManager"

    >
    <jms:listener id="queueListener_1" destination="ooIntegrationQueue"
        ref="myMessageListenerAdapter" />
</jms:listener-container>


<beans:bean id="myMessageListenerAdapter"
    class="org.springframework.jms.listener.adapter.MessageListenerAdapter" >
    <beans:property name="delegate" ref="consumer"/>
</beans:bean>


<beans:bean id="consumer" class="XXX.ConsumerImpl"/>

Producer settings :

<beans:bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate"
    p:connectionFactory-ref="connectionFactory" p:messageConverter-ref="jsonMessageConverter"
    p:defaultDestination-ref="ooIntegrationQueue" p:sessionTransacted="true" />

starting with the consumer, i managed to consume about 25 messages per second, which is extremely slow, i discovered the bottleneck to be the fact that i am using transactions, after googling for a bit, and playing with the configs, i found out that after autowiring the DefaultMessageListenerContainer and changing the cachelevel to

listenerContainer.setCacheLevelName("CACHE_SESSION") 

my performance increases to about 1500 messages per second while still having transactions.

my problem is now with the producer which is still stuck at about 25 operations per sec, my producer test is simple :

int numOfMessages = getNumberOfMessages();


double startTime = System.currentTimeMillis();

for (int i = 1; i <= numOfMessages; i++) {
    jmsTemplate.convertAndSend("HelloWorld" + i);
}

double endTime = System.currentTimeMillis();

double totalTime=(endTime-startTime)/1000;
System.out.println("Time - "+totalTime+" seconds");
System.out.println("EPS - "+numOfMessages/totalTime);

i'm wondering how to reach similiar performances with the producer, since it now bottlenecks the entire system.

Tavares

Answer 1

Sorry if this answer comes to late to help the original poster. I recently investigated JmsTemplate performance. Even with the same delivery and acknowledgment modes, native JMS code seemed much faster than JmsTemplate. The problem turned out to be that ActiveMQ normally defaults to async sending, but when you use JmsTemplate it instead uses sync sending. This dramatically reduces performance. You can set ActiveMQConnectionFactory's useAsyncSend property to true to force async sending. More details here: JmsTemplate is not evil

Stormy

Answer 2

JMSTemplate does a walk of ConnectionFactiory -> Connection -> Session -> MessageProducer, closing each object after each send. To get around this, wrap your amqConnectionFactory bean with a org.apache.activemq.pool.PooledConnectionFactory, and use that under the template rather than a CachingConnectionFactory.

Typhon

already tried that, CachingConnectionFactory defaults to a pool size of 1, changing that value doesn't change much, also using PooledConnectionFactory (which defaults to a pool size of 500) also gives similar performance – Tavares

There's definitely something running there - stick a profiler over the test itself and see if you can identify the hotspot. Out of curiosity, what's the performance like when you take the converter out of the equation? I.e. send simple test messages. The other thing is, you're testing linear performance in a single thread; that's probably not the way it will be used live. A more useful test would be to saturate it with horizontal load, by wrapping the call to convertAndSend in a Runnable and passing it to an ExecutorService. – Typhon

Down vote, really? I didn't see your response to Matt's answer, but I think was a legitimate question. I presume your producer code is in a JUnit test, although there's not enough info there. With a similar config I'm seeing 1000 messages per second. – Typhon

+1'ed since this is, in general, a correct answer although it doesn't seem to help James T's specific case. We don't yet know how expensive the JsonMessageConverter is and this test is setup to do things in a synchronous fashion on the producer side so the JsonMessageConverter might actually be the bottleneck. – Marchak

Answer 3

Try to change acknowledge method to from AUTO to CLIENT_ACKNOWLEDGE. For more information look to the Specification.

Burthen

Answer 4

What is the default delivery mode for ActiveMQ? is it a persistent queue? if so, how is it configured? how remote is the broker? These answers will determine the basic cost of sending to queue by answering how long it takes the server to ack the send (i.e. network RTT + potential cost of persisting the message to disk synchronously).

The other possibility is that you're actually creating a new connection, session and messageproducer on every send. This is pretty costly to say the least. It will be worth confirming whether this is happening (e.g. add debug logging to spring, check amq admin console for connection churn) or not as a basic sanity check. By the looks of it CachingConnectionFactory should cache a single session and messageproducer by default and convertAndSend should close the session it obtains after sending which results in a returning that cached session to the pool. This should mean it is relatively quick (spring jms goes through an awful lot of code just to send a message) to get the cached session on the next send.

Whitmore

well currently the messages aren't persistent, i debugged the jmsTemplate and the sessions and producers are cached, i decided to try a basic example without spring, and as it turns out (which makes sense) committing the session per message, results in the same performance, only bulking a certain amount of messages and then committing the transaction gives me the desired results. – Tavares

if a plain vanilla test and a spring based test have the same performance then you can be pretty certain the bottleneck is on the broker. 25/s equates to 40ms per message. This might be quite reasonable depending on the way the broker acks (and whether send is async or not) & depending on how it handles the message and what hardware it runs on. What is the default delivery mode? My guess is PERSISTENT, if so try switching to NON_PERSISTENT and you should see throughput go up. You should also look into using async sends if it has not defaulted to that already. – Whitmore