I have 2 active servers behind a load balancer. Lets say load balancer is A and nodes are A1 and A2.
Now, A1 and A2 has got quartz schedulers and A1 and A2 has got spring batch job instances.
At 1:00 PM lets say a job is supposed to be kicked and to be run on either A1 or A2.
But sometimes what happens is A1 and A2 both kicks the jobs and somehow they run twice. It runs either on different nodes or on the same node.
Issue is at both level quartz somehow kicks two instances of the same job and somehow spring batch also runs the two instances.
Any suggestions on fixing the issue or best practices from industry is welcomed.
You should simply check the clock of servers A1 and A2.
As clearly pointed out in Quartz guide,
Never run clustering on separate machines, unless their clocks are synchronized using some form of time-sync service (daemon) that runs very regularly (the clocks must be within a second of each other). See http://www.boulder.nist.gov/timefreq/service/its.htm if you are unfamiliar with how to do this.
Anyway, is a risky business to run multiple instance on different machines. Probably is better to use A2 as failover-cluster of A1 insted of running it in parallel.