Quartz Scheduler + EJB Singleton = sure-fire (?) deadlock

On our project we have used an EJB Singleton as a facade to the Quartz Scheduler. Something like:

@Singleton
@Startup
public class MyScheduler {

  private Scheduler scheduler;

  @PostConstruct
  private void init() {
    SchedulerFactory sf = new StdSchedulerFactory(props);
    scheduler = sf.getScheduler();
    scheduler.start();
  }

  public void scheduleJob(Class jobClass, Date executeAt) {
    //build Trigger and JobDetail
    //...
    //and finally
    scheduler.scheduleJob(jobDetail, trigger);
  }

  @PreDestroy
  private void destructor() {
    scheduler.shutdown();
  }
}

While this design has many benefits:

• Automatic startup and shutdown of the scheduler with deployment and undeployment of the application
• Serializing multithreaded access to Quartz scheduler (although some developer on terracota forum claims the scheduler to be thread-safe there is no mention of it in the Scheduler javadoc and we prefer not to debug potential multi-threading issues in a 3rd party library
• Using the singleton pattern we have a single Quartz Scheduler instance in the application

Unfortunately, such an approach is deadlock-prone if we want to schedule multiple jobs within a single transaction. Consider scenario where you have 2 EJBs, with methods respectively someMethod() and someOtherMethod(). Let's assume both methods run in their own transactions (e.g. use TransactionAttributeType.REQUIRED and are being invoked without transactional context):

1. invoke ejbA.someMethod()
2. ejbA.someMethod() starts a new transaction
3. within ejbA.someMethod you schedule some job via MyScheduler.scheduleJob(myJobClass, myJobExecuteAt). Quartz Scheduler acquirers its internal lock on trigger access in the DB (something like SELECT * FOR UPDATE FROM QUARTZ_LOCKS WHERE LOCK_NAME = 'TRIGGER_ACCESS' AND SCHED_NAME = 'yourSchedulerName')
4. in parallel, somewhere in the same application ejbB.someOtherMethod() gets invoked
5. within ejbB.someOtherMethod() you want to schedule some other job via MyScheduler.scheduleJob(myOtherJobClass, otherJobExecuteAt). As Quartz is again trying to get its lock on trigger access in the DB and the ejbA.someMethod() transaction has neither rolled back nor commited, it is stuck waiting for the DB lock.
6. Now, within ejbA.someMethod() you want to schedule another job via MyScheduler.scheduleJob(yetAnotherJobClass, yetAnotherJobExecuteAt). Unfortunately, in accordance with the EJB Singleton multithreading semantics the ejbB.someOtherMethod() acquired the lock to our singleton MyScheduler and ejbA.someMethod() has to wait until ejbB.someOtherMethod() exits from MyScheduler.scheduleJob(myOtherJobClass, otherJobExecuteAt) invocation.

And now we have a classic deadlock - ejbA.someMethod() is holding lock to trigger access while waiting for lock to MyScheduler and ejbB.someOtherMethod() is holding lock to MyScheduler and is wating for lock to trigger access.

In fact, a similar deadlock scenario can happen in any EJB Singleton with serialized access which creates DB locks (not only explicit locks but also e.g. implicit locks for DML updates or inserts)

Solution

The solution is much less complicated than the deadlock scenario:

• either schedule each Job in separate transaction (e.g. annotate the MyScheduler.scheduleJob() method with @TransactionAttribute(REQUIRES_NEW); this has the disadvantage that you lose the atomicity of the business logic that you are executing and scheduling of the job)
• or schedule all the jobs you need in single invocation of MyScheduler via method like MyScheduler.scheduleJobs(List jobsToSchedule>) where MyJobData class contains all the information you need to schedule a single job (i.e. job class, job execution date time, possibly also job params)

Docker & Gentoo Linux & su could not open session

kamil@kamil-elitebook:~$ docker run -ti oraclelinux:6 /bin/bash
[root@bac7e6cd1aad /]# su - oracle
could not open session

2 hours of googling later:

kamil@kamil-elitebook:~$ docker run --privileged -ti oraclelinux:6 /bin/bash
[root@2f676eeb3b7a /]# su - oracle
[oracle@2f676eeb3b7a ~]$