Quartz Resource Adapter as an alternative to EJB Timers

Anyone developing a J2EE application probably faced the challenge of setting some fixed interval timers that need to be up upon application startup. This, in fact involves two challenges:

First one: J2EE provides no means of startup notification on EJB-Jars.

Second: EJB timers are persistent (they survive server restarts) and if you don’t cancel them upon application shutdown you’ll end up having lots of those timers.

So, wouldn’t it be fine if we had some mechanism of having a notification sent down from AppServer to our application in a timed manner? It’d be even better if we had some mechanism to specify the intervals using Unix Cron format.

Happily there is a simple solution: Quartz Resource Adapter from JBoss licensed as LGPL (as stated on its ra.xml) that makes it safely usable on commercial products. It comes packaged as a rar file on JBoss deploy directory as it seems to be used for JBoss internal purposes but nothing (or almost nothing) prevents you from using it into another AppServer.

By almost nothing I meant that as it is packaged on JBoss there is one quirk:

QuartzResourceAdapter.endpointActivation method does one trick to detect whether the endpoint configured is a stateless or stateful endpoint that renders it unusable on WebSphere for example (WebSphere prevents you from calling MessageEndpointFactory.createEndpoint from the Thread that invoked endpointActivation) and I am sincerely not sure whether this is something the spec enforces. I’ll paste the trick below:

 // allocate instance of endpoint to figure out its endpoint interface
      Class clazz = QuartzJob.class;
      MessageEndpoint tmpMe = endpointFactory.createEndpoint(null);
      if (tmpMe instanceof StatefulJob) clazz = StatefulQuartzJob.class;
      tmpMe.release();

And the best thing is that this trick can be safely replaced by two markup ActivationSpec classes, so we end up having the following code:

      // figure out endpoint interface through activationspec (WAS denies endpoint creation on main thread)
      Class clazz = null;
      if (spec instanceof QuartzStatelessActivationSpec) {
          clazz = QuartzJob.class;
      }
      else {
          clazz = StatefulQuartzJob.class;
      }

And the two markup classes:

package org.jboss.resource.adapter.quartz.inflow;

public class QuartzStatefulActivationSpec extends QuartzActivationSpec {

}

package org.jboss.resource.adapter.quartz.inflow;

public class QuartzStatelessActivationSpec extends QuartzActivationSpec {

}

And the ra.xml file for the RAR package gets changed from:

 <messagelistener>
               <messagelistener-type>org.quartz.Job</messagelistener-type>
               <activationspec>
                  <activationspec-class>org.jboss.resource.adapter.quartz.inflow.QuartzActivationSpec</activationspec-class>
                  <required-config-property>
                      <config-property-name>cronTrigger</config-property-name>
                  </required-config-property>
               </activationspec>
            </messagelistener>
            <messagelistener>
               <messagelistener-type>org.quartz.StatefulJob</messagelistener-type>
               <activationspec>
                  <activationspec-class>org.jboss.resource.adapter.quartz.inflow.QuartzActivationSpec</activationspec-class>
                  <required-config-property>
                      <config-property-name>cronTrigger</config-property-name>
                  </required-config-property>
               </activationspec>
            </messagelistener>

To:

 <messagelistener>
               <messagelistener-type>org.quartz.Job</messagelistener-type>
               <activationspec>
                  <activationspec-class>org.jboss.resource.adapter.quartz.inflow.QuartzStatelessActivationSpec</activationspec-class>
                  <required-config-property>
                      <config-property-name>cronTrigger</config-property-name>
                  </required-config-property>
               </activationspec>
            </messagelistener>
            <messagelistener>
               <messagelistener-type>org.quartz.StatefulJob</messagelistener-type>
               <activationspec>
                  <activationspec-class>org.jboss.resource.adapter.quartz.inflow.QuartzStatefulActivationSpec</activationspec-class>
                  <required-config-property>
                      <config-property-name>cronTrigger</config-property-name>
                  </required-config-property>
               </activationspec>
            </messagelistener>

Apart from this you’ll have to include jboss-common.jar (found on $JBOSS_HOME\lib) into the quartz-ra.rar file in the same level as quartz-ra.jar and quartz.jar.
Now you are done to install it into your application server and finally create your Activation Specs.
I am providing a PDF with Patched Quartz Resource Adapter for anyone willing to test on your preferred J2EE server and I also posted a message on JBoss AS Development forum to check whether they are interested in incorporating these changes.
Updates: It seems like this is going to get integrated into JBoss AS main code. I’ve submitted the patch as requested.

Anatomy of an inbound jca connector

Continuing with the posts about JCA lets now have a look on the specifics of an Inbound JCA connector. As already mentioned in a previous post the term inbound is related to the flow of the call on your application.
The endpointActivation method plays a key role in an Inbound adapter. This is the method that is triggered by the container on each of the Activation Specs you have configured on your Application Server instance. This is also the point where you start your monitoring thread (FooMonitor in our example). Remember that this monitoring object has to have access to a number of key container classes such as MessageEndpointFactory implementation and also WorkManager.
Apart from container objects you’ll certainly need to have the ActivationSpec object passed on the endpointActivation method since it carries on the configuration provided for this activation spec (taking JMS as an example it’d be host, queue, username, …).
Another point to remember is that you’ll need to have a reference to the monitor threads that were activated since on container shutdown it’ll invoke the endpointDeactivation method and you’ll need to retrieve the related monitor thread and stop its execution.

Inbound Connector classes

Too much text lets now have a look on some diagrams:

As you may have already guessed, FooMonitor is the thread that will poll for events and schedule their notifications through the WorkManager. You should never use the monitor thread for the notification since it’ll be blocked during notification denying the sending of other notifications. That’s the reason you should schedule an asynchronous notification using WorkManager.

FooNotification implementation

Skipping to the FooNotification class, its implementation will be something like the following code:

public void run() {
	MessageEndpoint endpoint = null;
	try {
		endpoint =  messageEndpointFactory.createEndpoint(null);
		((FooListener) endpoint).onMessage(msg);
	} catch (Exception ex) {
		logger.error("Error on foo notification" ,ex);
	} finally {
		if (endpoint != null)
			endpoint.release();
	}
}

As already mentioned, that’s the point where the MessageEndpointFactory is used. It provides us proxies for FooMDB classes. Remember to ALWAYS invoke release after using this proxy otherwise your container may run out of proxies since it’ll think you are still using them.
Another aspect not really detailed is the msg parameter on the onMessage method: this one is dependent on your ResourceAdapter – You are the one responsible for defining it (unless you have chosen a regular API like JMS).

Future updates to the JCA tutorial

This week I had a conf-call with a developer from a JavaEE server vendor to figure out what was happening with one of our JCA adaptors when a failure was reported in the underlying system and to my surprise “connectionErrorOccurred” method may not be called as I felt when I read the specs. One of the points that came up in the conf-call were the missing parts of the spec. This became clear when we discussed that on two other containers the adapter that caused the discussion worked fine and was giving problems only on this particular container. The developer argued (and he is right, spec does not cover this) that this vendor chose to disallow “connectionErrorOccurred” calls while connection is not in use (as the developer referred “our connections”, or, connections that are being held by the container in the connection pool). He also argued that this assumption (the one made by the container) was motivated to increase performance as it could reduce synchronization code. So, after these discoveries I’ll be updating the posts related to JCA and in the further weeks, after I have some time to do some updates to my FooJCA connector (this connector was useful to reproduce the bug inside the vendor lab) I’ll release it here for download so anyone willing to see a working example of a JCA connector will have a quick start code.

Complete post series

Outbound connector related posts:

Outbound JCA connectors introduction
Connection request flow on outbound JCA connector
matchManagedConnections short tip
Java Connector Architecture (JCA)

Inbound connector related posts:

Inbound JCA Connectors Introduction
Anatomy of an inbound JCA connector

Inbound JCA Connectors Introduction

After some posts about Outbound JCA connectors, let’s have a look at the concepts related to an Inbound JCA connector.

First question I always hear: “So, the server (legacy application) connects a client socket into a server socket on my J2EE server, right?” and I always answer: “depends”. People tends to think that the term inbound and outbound are related to TCP/IP connection, but, in fact, it is related to the flow of the process. In an outbound connector, our system is the actor and if you trace the flow of the call you’ll notice that it is leaving from our system to the legacy one; or perharps we could say outgoing. On the other hand, in an inbound connector, the actor is the legacy system that is triggering an ingoing message that is started outside our system and goes all the way into it.

Let’s see a sequence diagram of an inbound connector to make things clearer:

JCA Inbound Connector Sequence Diagram

As you can see in the diagram, the component that ties the Application Server, the legacy system and the J2EE Application is the ActivationSpec.

Usually, instance configurations such as port, host and other instance related data is stored on the ActivationSpec, you may also have configuration in the ResourceAdapter itself, but remember that all the configurations placed on the ResourceAdapter will be shared across multiple ActivationSpecs that may be deployed on this ResourceAdapter.

The real class responsible for handling Enterprise Events is ResourceAdapter dependant and will be initialized during endpointActivation method call on a ResourceAdapter. These classes are usually implemented as a Thread or they implement the Work interface and are submitted to the container for execution through the WorkManager instance. If you opt to use a simple Thread, remember to daemonize it otherwise your application server wont be able to properly shutdown.

For the next weeks I’ll be posting some insights about how to implement an Inbound Connector using JCA.

Websphere PMI: enabling and viewing data

For those who ever needed to have a deeper look at application internals that may be impacting performance probably had this impression:

System.out.println with System.nanoTime (or currentTimeMillis) is tedious, errorprone and limited

A profiler is an overkill not to mention cumbersome (and unavailable for certain platforms [eg.:tptp on AIX]*)

This is the scenario where Websphere PMI is a killer feature.

Imagine that your application isn’t performing as expected. Many can be the reasons for the poor performance. I’ve faced myself a scenario where the application was waiting a long time for getting a JMS connection from Websphere internal provider since its default configuration of 10 connections maximum isn’t acceptable for any application with performance requirements of even 100 transactions per second.

Enabling PMI

By default, Websphere 6.1 ND comes with basic PMI metrics enabled. These include for example:

• Enterprise Beans.Create Count
• JDBC Connection Pools.Wait Time
• JDBC Connection Pools.Use Time

If you need anything more than the default, you can change under:

Monitoring and Tuning > Performance Monitoring Infrastructure (PMI)

then click on the desired server.

After you have chosen the desired metrics (remember that more metrics involve more CPU impact on runtime), go to the following menu:

Monitoring and Tuning > Performance Viewer > Current Activity

Now you need to check if your server is in fact already collecting data, if it is already enabled but not collecting, Collection Status will show Available. In order to start collecting, check the desired server and click Start Monitoring button. After clicking the button it will now show Monitored on the status column.

Now you can click on the desired server and tick for example one of your connection pools under the tree on the left, you should see an structure similar to the below:

Performance Modules > JDBC Connection Pools > Oracle XA Provider > yourDataSource

After clicking the metric you’ll have a graph display of the current data and also a tabular with the snapshot of the indicator below.

* note: Eclipse TPTP is said to be supported on AIX on version 4.3.1 but I have not been able to make it work

matchManagedConnections short tip

Be aware that the container may send invalid connections to the matchManagedConnections method and if you happen to return any of the invalid connections it’ll give it back to the application. This check is important since connectionErrorOccurred is only safe to be called while connection is in use. Some containers have optimizations that deny you from calling while connection is not in use by the application (more exactly while in the pool).

So even if your ManagedConnectionFactory implements the interface ValidatingManagedConnectionFactory you should also check for invalid connections during the selection of a matching connection inside matchManagedConnections method.
The getInvalidConnections method usually is called after one connection has been detected as invalid (after a valid call to connectionErrorOccurred) or by a container timer thread (on WebSphere this timeout is specified in the reap time parameter).

Connection request flow on outbound JCA connector

Continuing with the posts about outbound JCA connectors lets have a quick look at one of the main flows of an outbound JCA connector: the connection request flow.

By now you might be wondering: “What on earth is responsible for making my connectors poolable? Is there any dark magic involved?”.

First answer: “the ConnectionManager” and second answer: “Yes and it is container dependant!”. That’s the reason you should keep an instance of the ConnectionManager inside your ConnectionFactory when it is created, that’s how you delegate the connection creation to the pooling implementation of the container.

Enough explanation let’s have a look on a sequence diagram for a connection creation when we have no connections inside the pool.

First flow: connection request with no matching connections

First thing to take note, it is up to you to implement the matchManagedConnections, I’ve put a note on the next diagram that might help on this implementation: making the ManagedConnection be able to compare itself to a ConnectionRequestInfo makes the things easier. Also beaware that some Application Servers (eg.: WebSphere) skip the matchManagedConnections part if it detects by itself that the pool is empty or there isn’t any matching connection.

Sequence Diagram - No Matching Connection

Another point about the specification is that it states that a single managed connection may be responsible for more than a single physical connection (FooConnectionImpl), this is required for scenarios where connection sharing is possible (refer to section 6.5.4.2 of the specification).

Previously I tried implementing using a single handle but I noticed that it does not take much effort to make it fully adherent, in fact, it is only necessary to use a delegate (FooConnectionHandle) that implements the FooConnection interface and delegates almost all the methods to the FooConnectionImpl instance that is inside the FooManagedConnection instance (refer to sections 6.5.4.1 and 6.8.1 of the specifications). The exception to the delegation is the close method that your connection must provide, this method in the delegate will be responsible for raising the Connection Closed Event, this is the way you signal the container that you are giving the connection back to the pool.
Second flow: connection request when there are matching connections inside the pool.

Sequence Diagram - Matching Connections

This flow is executed whenever the container guesses there are potential matching connections. The matchManagedConnection method is invoked with the minimal connection set that the container can identify. The connection set is a key point: specification states that the container is responbile for determining the minimal set that has a potential match to avoid degrading performance while looking for the connection. Also I noticed that some containers don’t check the connections inside this Set before sending them for matching.

Implementation Tips

Here are some decisions that might help on the implementation of the ManagedConnection and ManagedConnectionFactory.

• Store the ConnectionRequestInfo that was passed to the ManagedConnectionFactory on the creation of the ManagedConnection as an attribute inside the ManagedConnection this leads to the next tip
• Use the ConnectionRequestInfo stored in the ManagedConnection as argument to an overload of the equals method of the ManagedConnection class this helps in the implementation of the matchManagedConnection method
• Never ever forget to include your ManagedConnection object on equals(Object) method implementation. The tip above may lead you to forget this details. Dont do this under any circumstance since specification requires this and some containers freak out after some use time if this method is not implemented (connection borrow time goes all way up if not implemented)

Next post I’ll focus on how to signal the container that something went wrong with the connection and how to validate the connection before it is returned to the requestor.

Outbound JCA connectors introduction

Continuing with the posts about JCA I’ll start with the project of an Outbound JCA Connector that I’ll call FooJCA. First of all and let make this clear, I am not stating that this design is an optimal JCA Connector design, what I can say is that I tried to make as compliant as possible from my understanding of the documentation.

Lets start with a overall view of the main classes and interfaces involved in a JCA Connector implementation.

Foo Connector Class Diagram

First class to take note, the FooResourceAdapter, that’s one of the connector classes that is able to directly communicate with the container and use some priviledged services (eg.: scheduling Work instances or creating timers). This ability is provided by the BootstrapContext that is passed as a parameter of the start method by the container.

Another key class that also has the ability of interacting with the container is the FooManagedConnectionFactory. As you may have already noticed, this class isn’t always container dependant, or at least it may not be (that’s up to the developer the choice of providing functionality of the Connector outside the container). The difference (between working inside the container or outside it) is usually detected when the class method createConnectionFactory is called with or without the ConnectionManager. Talking about the ConnectionManager this is a Container implemented class that is responsible for pooling connections and related work. This Container work relies on the matchManagedConnections method that is provided by the ManagedConnectionFactory, so, pay much attention to this method implementation.

Moving on, the FooConnectionRequestInfo is a implementation of the ConnectionRequestInfo interface and is the class responsible for passing Connector dependant details about the request between the Container and the JCA Connector. One of the requisites of this class is that it implements equals and hashCode.

The FooManagedConnectionFactory as you may guess is the class responsible for creating the FooConnectionFactory instances and FooManagedConnection instances as well. It does not matter if the FooConnectionFactoryImpl instance will be Container dependent or not, it will be FooManagedConnectionFactory’s role of creating it.

At least but not last, the FooConnectionFactoryImpl will be the class responsible for creating FooConnectionImpl instances when requested (this will be treated in a separate post since it is a little long topic).

And, for finishing this post, FooManagedConnectionImpl is the class responsible for wrapping a FooConnectionImpl and returning delegates (FooConnectionHandle class) that implement FooConnection interface and notifies the container about the status of the ManagedConnection connection.

That’s enough for this post, keep on watching the blog for the rest of the implementation of the Outbound JCA Connector.

Java Connector Architecture (JCA)

Long time since my last post but job was demanding way too much time from me. Anyways today I’ll start a series of posts about the Java Connector Architecture. Chances are that you have already used a JCA Connector but never needed to write one by yourself. But, why would someone need to write one? Picky question but the JavaEE specification is very clear, you can’t open ServerSockets, Threads, to name a few.

But why would I need a Thread inside an EJB container if it is already built around a Threaded model? Answer is, you need it everytime your component has a sofisticate interaction with an external system for example. To make things clearer let me use the scenario I faced weeks ago.

Communication with Legacy application using Sockets

One of the requisites of the application I am working on is to communicate with a legacy application written in C. If the communication was always started by my application and I didn’t needed a Pooling model I could stick easily with a ClientSocket inside my EJB (since specification only mandates that ServerSockets and Threads are denied inside an EJB), but, since the legacy application sometimes sends a “ping” request and I needed pooling for low latency I had to create my own JCA connector.

First Steps

For those who are starting I’d suggest a few tutorials around the net:

Introduction to the J2EE Connector Architecture

Tutorial: Deploying a JCA Resource Adapter for an EJB Application

Connect the enterprise with the JCA, Part 2

I agree, what is the point of this post if I am only referencing some other tutorials? I only wanted to make a briefing about the posts to come. Personally during the development of my JCA connector I found some dificulties that I think are worthy posting over here.

So, if you are interested in JCA, start reading the tutorials mentioned and take a quick look at the JCA 1.5 specification as well and keep an eye on this blog for the next weeks.

Update for the original post:

Another good reason for learning how to implement a JCA Connector: Aqualogic Service Bus, one of the foundations of Oracle Fusion suite, heavily relies on JCA Connectors for communicating with every kind of application you can imagine. So, if your company is moving to Service Orientation (SOA) and depending on the systems you’ll be integrating you’ll have to roll your own connector.

Complete post series

Outbound connector related posts:

Outbound JCA connectors introduction
Connection request flow on outbound JCA connector
matchManagedConnections short tip
Future updates to the JCA tutorial

Inbound connector related posts:

Inbound JCA Connectors Introduction
Anatomy of an inbound JCA connector