poniedziałek, 29 czerwca 2015

Quartz Scheduler + EJB Singleton = sure-fire (?) deadlock

On our project we have used an EJB Singleton as a facade to the Quartz Scheduler. Something like:
@Singleton
@Startup
public class MyScheduler {

  private Scheduler scheduler;

  @PostConstruct
  private void init() {
    SchedulerFactory sf = new StdSchedulerFactory(props);
    scheduler = sf.getScheduler();
    scheduler.start();
  }

  public void scheduleJob(Class jobClass, Date executeAt) {
    //build Trigger and JobDetail
    //...
    //and finally
    scheduler.scheduleJob(jobDetail, trigger);
  }

  @PreDestroy
  private void destructor() {
    scheduler.shutdown();
  }
}

While this design has many benefits: Unfortunately, such an approach is deadlock-prone if we want to schedule multiple jobs within a single transaction. Consider scenario where you have 2 EJBs, with methods respectively someMethod() and someOtherMethod(). Let's assume both methods run in their own transactions (e.g. use TransactionAttributeType.REQUIRED and are being invoked without transactional context):
  1. invoke ejbA.someMethod()
  2. ejbA.someMethod() starts a new transaction
  3. within ejbA.someMethod you schedule some job via MyScheduler.scheduleJob(myJobClass, myJobExecuteAt). Quartz Scheduler acquirers its internal lock on trigger access in the DB (something like SELECT * FOR UPDATE FROM QUARTZ_LOCKS WHERE LOCK_NAME = 'TRIGGER_ACCESS' AND SCHED_NAME = 'yourSchedulerName')
  4. in parallel, somewhere in the same application ejbB.someOtherMethod() gets invoked
  5. within ejbB.someOtherMethod() you want to schedule some other job via MyScheduler.scheduleJob(myOtherJobClass, otherJobExecuteAt). As Quartz is again trying to get its lock on trigger access in the DB and the ejbA.someMethod() transaction has neither rolled back nor commited, it is stuck waiting for the DB lock.
  6. Now, within ejbA.someMethod() you want to schedule another job via MyScheduler.scheduleJob(yetAnotherJobClass, yetAnotherJobExecuteAt). Unfortunately, in accordance with the EJB Singleton multithreading semantics the ejbB.someOtherMethod() acquired the lock to our singleton MyScheduler and ejbA.someMethod() has to wait until ejbB.someOtherMethod() exits from MyScheduler.scheduleJob(myOtherJobClass, otherJobExecuteAt) invocation.

And now we have a classic deadlock - ejbA.someMethod() is holding lock to trigger access while waiting for lock to MyScheduler and ejbB.someOtherMethod() is holding lock to MyScheduler and is wating for lock to trigger access.

In fact, a similar deadlock scenario can happen in any EJB Singleton with serialized access which creates DB locks (not only explicit locks but also e.g. implicit locks for DML updates or inserts)

Solution
The solution is much less complicated than the deadlock scenario:
  • either schedule each Job in separate transaction (e.g. annotate the MyScheduler.scheduleJob() method with @TransactionAttribute(REQUIRES_NEW); this has the disadvantage that you lose the atomicity of the business logic that you are executing and scheduling of the job)
  • or schedule all the jobs you need in single invocation of MyScheduler via method like MyScheduler.scheduleJobs(List jobsToSchedule>) where MyJobData class contains all the information you need to schedule a single job (i.e. job class, job execution date time, possibly also job params)

piątek, 26 czerwca 2015

Docker & Gentoo Linux & su could not open session

kamil@kamil-elitebook:~$ docker run -ti oraclelinux:6 /bin/bash
[root@bac7e6cd1aad /]# su - oracle
could not open session
2 hours of googling later:
kamil@kamil-elitebook:~$ docker run --privileged -ti oraclelinux:6 /bin/bash
[root@2f676eeb3b7a /]# su - oracle
[oracle@2f676eeb3b7a ~]$ 

sobota, 21 lutego 2015

JBoss dummy login module with predefined role

In case you need to test authorization of your JBoss-deployed application and you don't want to connect to the remote system or can't configure it to return proper roles for you, you may use JBoss Identity login module:


<security-domain name="mydomain" cache-type="default">
    <authentication>
      <login-module code="Identity" flag="required">
	<module-option name="principal" value="myprincipal"/>
	<module-option name="roles" value="ROLE_TO_BE_TESTED1,ROLE_TO_BE_TESTED2"/>
      </login-module>
    </authentication>
</security-domain>

where:
  • mydomain is the security domain name specified in the application
  • myprincipal - principal (i.e. login) name of the authenticated user. Default is guest.
  • ROLE_TO_BE_TESTED1,ROLE_TO_BE_TESTED2 - comma-delimited list of roles that will be assigned to the authenticated user.

sobota, 24 stycznia 2015

Authorizing JBoss EAP 6.1 Alpha web app users with Active Directory

Delivering an intranet web application hosted on JBoss EAP 6.1.Alpha to my customer I came to the point where some authentication and authorization mechanism needed to be introduced. Customer had Active Directory infrastructure already in place, so the obvious choice was to integrate with it.

After some research it turned out 2 options were available:
  1. Kerberos/SPNEGO-based SSO.
  2. Authentication using LDAP protocol.
The former option seemed more tempting at first - it would provide passwordless (yes, no login form!) integration for Windows domain users, but unfortunately it required changes in web browser configuration for each user. As no automatic provision of such configuration was possible I headed for the latter option. As the local admin wanted to "opt out as much as possible" of the project the provision of application roles was to be done in a local database (will hopefully be described in another post)

Now, what I needed to do was to configure the JBoss security domain and the ldap authentication module in standalone.xml:
<security-domain name="LDAPRealm">
    <authentication>
        <login-module code="Ldap" flag="required">
            <module-option name="java.naming.provider.url" value="ldap://hostname:389/"/>
            <module-option name="java.naming.security.authentication" value="DIGEST-MD5"/>
            <module-option name="javax.security.sasl.qop" value="auth-conf"/>
            <module-option name="password-stacking" value="useFirstPass"/>
        </login-module>
<!-- Database module to retrieve user roles - will be described in another post -->
    </authentication>
</security-domain>
As for the meaning of each option:

java.naming.provider.url - the URL of ldap one wants to connect to. It is important - at least using DIGEST-MD5 - that it is set to hostname, which the Active Directory server recognizes. In particular the IP address shouldn't be used. Otherwise one would get:
javax.naming.AuthenticationException: [LDAP: error code 49 - 80090303: LdapErr: DSID-0C090420, comment: The digest-uri does not match any LDAP SPN's registered for this server., data 0, vece
java.naming.security.authentication - type of authentication to use:
  • none - no authentication. That's not what I want - I do want to authenticate the user
  • simple - plain text authentication using Distinguished Name as username, e.g. in my case this would be CN=Kamil Roman,CN=Users,DC=mydomain,DC=com. Please note that the domain username is not used in the login string.
  • one (or more) SASL authentication mechanism. For most LDAPs including Active Directory you can check the supported mechanisms by issuing
kamil@localhost:~$ ldapsearch -H ldap://192.168.1.100 -x -b "" -s base -LLL supportedSASLMechanisms
dn:
supportedSASLMechanisms: GSSAPI
supportedSASLMechanisms: GSS-SPNEGO
supportedSASLMechanisms: EXTERNAL
supportedSASLMechanisms: DIGEST-MD5
as long as DIGEST-MD5 is returned on the list it is fine

In case of SASL authentication the domain login (e.g. kroman in my case) is used as username.

javax.security.sasl.qop - security level of the protocol, valid options:
  • auth - Authentication only (default)
  • auth-int - Authentication with integrity protection
  • auth-conf - Authentication with integrity and privacy protection
In my case using the default option resulted in error message from the AD server:
0002028: LdapErr: DSID-0C090169, comment: The server requires binds to turn on integrity checking if SSL\TLS are not already active on the connection, data 0, vece 
Anyway, using auth-conf seems the most secure option and thus the best as long as it is supported by the LDAP server.

password-stacking - setting this to useFirstPass will result in retrieving roles from the next login-module (will be described in another post)

That's it on the app server side! Now we need to specify to use the newly configured security domain in the WEB-INF/jboss-web.xml file:


<?xml version="1.0" encoding="UTF-8"?>
<jboss-web>
    <security-domain>java:/jaas/LDAPRealm</security-domain>
</jboss-web>

Troubleshooting

  • set logging level on org.jboss.security and org.jboss.as.web.security to TRACE or ALL
  • JBoss swallows the root cause authentication exception and in the log file you will always get only
    DEBUG [org.jboss.security] (http-/0.0.0.0:8080-5) PBOX000206: Login failure: javax.security.auth.login.FailedLoginException: PBOX000070: Password invalid/Password required
    and a
    javax.servlet.ServletException: JBWEB000053: Failed to authenticate a principal thrown from the HttpServletRequest.login() if you use this method for authentication.
    In order to know the root cause and/or message received from the AD server you need to debug the LdapLoginModule.validatePassword method and especially its catch(Throwable e){} block