Run @Scheduled task only on one WebLogic cluster node?

Question

We are running a Spring 3.0.x web application (.war) with a nightly @Scheduled job in a clustered WebLogic 10.3.4 environment. However, as the application is deployed to each node (using the deployment wizard in the AdminServer's web console), the job is started on each node every night thus running multiple times concurrently.

How can we prevent this from happening?

I know that libraries like Quartz allow coordinating jobs inside clustered environment by means of a database lock table or I could even implement something like this myself. But since this seems to be a fairly common scenario I wonder if Spring does not already come with an option how to easily circumvent this problem without having to add new libraries to my project or putting in manual workarounds.

• We are not able to upgrade to Spring 3.1 with configuration profiles, as mentioned here

Please let me know if there are any open questions. I also asked this question on the Spring Community forums. Thanks a lot for your help.

Churchly

Do you want to achieve some kind of failover? If so configuration profiles won't be really helpful here and you need some synchronization anyway. We had a similar requirement and ended up with a lock table which also provides some quick execution summary. – Evoke

@Evoke No, we are (currently) not looking for failover, since this would complicate things quite a lot. – Churchly

Answer 1

We are implementing our own synchronization logic using a shared lock table inside the application database. This allows all cluster nodes to check if a job is already running before actually starting it itself.

Churchly

What isolation level are you using here? Read_committed or serializable? – Dairy

Answer 2

We only have one task that send a daily summary email. To avoid extra dependencies, we simply check whether the hostname of each node corresponds with a configured system property.

private boolean isTriggerNode() {
   String triggerHostmame = System.getProperty("trigger.hostname");;
   String hostName = InetAddress.getLocalHost().getHostName();
   return hostName.equals(triggerHostmame);
}

public void execute() {
   if (isTriggerNode()) {
      //send email
   }
}

Lavina

Answer 3

Be careful, since in the solution of implementing your own synchronization logic using a shared lock table, you always have the concurrency issue where the two cluster nodes are reading/writing from the table at the same time.

Best is to perform the following steps in one db transaction: - read the value in the shared lock table - if no other node is having the lock, take the lock - update the table indicating you take the lock

Afire

Hi Mathias. Even though it's been a while, thank you for your helpful suggestion. We actually do the check and the writing of the lock entry inside a single transaction. Good point and a common pitfall. – Churchly

Answer 4

I solved this problem by making one of the box as master. basically set an environment variable on one of the box like master=true.

and read it in your java code through system.getenv("master"). if its present and its true then run your code.

basic snippet

@schedule()
void process(){
boolean master=Boolean.parseBoolean(system.getenv("master"));
if(master)
{
   //your logic
}

}

Ovular

I think this is not the best solution, lets imagine you have a cluster of two nodes, you made the first node as a master and the second is NOT. After that for any reason the first node went down, then your job will not be executed because the Node that should handle the job down, so you will lose the most important feature in the cluster environment "High Availability" – Astronaut

Answer 5

you can try using TimerManager (Job Scheduler in a clustered environment) from WebLogic as TaskScheduler implementation (TimerManagerTaskScheduler). It should work in a clustered environment.

Andrea

Homemaking

Answer 6

You don't neeed to synchronize your job start using a DB. On a weblogic application you can get the instanze name where the application is running:

String serverName = System.getProperty("weblogic.Name");

Simply put a condition two execute the job:

if (serverName.equals(".....")) {
  execute my job;
}

If you want to bounce your job from one machine to the other, you can get the current day in the year, and if it is odd you execute on a machine, if it is even you execute the job on the other one. This way you load a different machine every day.

Acculturize

Answer 7

We can make other machines on cluster not run the batch job by using the following cron string. It will not run till 2099.

0 0 0 1 1 ? 2099

Byrle