Instrumentation and Monitoring as means of problem detection and resolution

While designing the last big project I worked on we chose to place some monitoring code in specific points of the application. Those metrics were meant for emitting information targeted at business users but also to technical users, this information was then split according to audience through the respective JMX agents used (was it a Tivoli ITCAM agent or a JOPR agent).
At first we were expecting only that this could provide us valuable information for live environment such as when we had anything abnormal on the legacy software we were integrating with but we ended up noticing that this could also provide us valuable informal of the operation behaviour of our software and the best part: on live environment. And in fact it turns out that this is so common that we can find others pointing into this direction as well.
The picture below presents an overview of the application in question as well as the instrumentation points.

Application overview with instrumentation points

The instrumentation points gathered the following information:

• The instrumentation point on the left of the OuterMDB collected the sum of the messages processed in the current and last hour as well as the messages per second.
• The instrumentation point on depicted in the top of the OuterMDB collected the sum of the time spent in pre-processing as well as the number of times pre-processing was invoked.
• The instrumentation point on top of the InnerMDB collected the sum of messages processed in the current and last hour as well as the messages per second.
• And finally, the instrumentation point on the bottom of the InnerMDB collected the sum of time spent in the communication with the legacy system as well as the average of processing time per request in the current and last hour, the minimum and maximum times of processing for current and last hour and the amount of request processed as well as the timeouts.

The comparison between the number of messages processed in the InnerMDB and OuterMDB could provide us means of comparing how we should size the Thread pools for each of these components. This is such an information that would be harder to obtain by any other means. We also used those metrics for detecting misfunction on the pre-processing legacy software that was invoked by our PreProcessing component, this way we could switch off pre-processing and avoid a negative impact on overall system performance.
But this monitoring was key to the detection of a misbehavior of our JCA connector. A misimplementation of the equals/hashcode method pair for the ManagedConnection lead to a huge performance degradation after a few hours of software operation. By using our monitoring infrastructure we could isolate the problematic code area. Sure it did not point towards the equals/hashcode pair but it was clear that it was related to connection acquisition code.
Finally, the monitoring in our application provided us an effective way of monitoring the legacy application we were communicating with since it did not provide any native way of monitoring its operation. We were then able to instantly respond to outages on the legacy application through metrics collected on our software.

Monitoring a secured WebSphere installation through JOPR/RHQ

Monitoring an WebSphere installation with security enabled using JOPR/RHQ may seem at first simple and indeed it is if it weren’t for a few missing parts on the JOPR agent and MC4J library.
For those in a hurry that don’t want to wait till it gets incorporated into JOPR and MC4J I’ll try to summarize the changes and also provide the JMX Agent jar to be replaced. Apart from this, there is a need to import the server certificate for WAS SOAP Admin port into the JVM that JOPR is running (I hope there’ll be a certificate management feature in JOPR in the future to make this even simpler).
Let’s get to the changes.
First of all download RHQ code from SVN.

svn co http://svn.rhq-project.org/repos/rhq/tags/RHQ_1_1_0_GA/

Use tag 1.1.0 since I made the changes on top of that version.
Replace the following file on JMX plugin:
JMXServerComponent

/*
* RHQ Management Platform
* Copyright (C) 2005-2008 Red Hat, Inc.
* All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation version 2 of the License.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
*/
package org.rhq.plugins.jmx;

import java.io.File;
import java.io.FilenameFilter;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Map;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.mc4j.ems.connection.ConnectionFactory;
import org.mc4j.ems.connection.EmsConnection;
import org.mc4j.ems.connection.local.LocalVMFinder;
import org.mc4j.ems.connection.local.LocalVirtualMachine;
import org.mc4j.ems.connection.settings.ConnectionSettings;
import org.mc4j.ems.connection.support.ConnectionProvider;
import org.mc4j.ems.connection.support.metadata.ConnectionTypeDescriptor;
import org.mc4j.ems.connection.support.metadata.J2SE5ConnectionTypeDescriptor;
import org.mc4j.ems.connection.support.metadata.LocalVMTypeDescriptor;

import org.rhq.core.domain.configuration.Configuration;
import org.rhq.core.domain.configuration.PropertySimple;
import org.rhq.core.domain.measurement.AvailabilityType;
import org.rhq.core.domain.resource.Resource;
import org.rhq.core.domain.resource.ResourceType;
import org.rhq.core.pluginapi.inventory.ResourceContext;

/**
* The generic JMX server component used to create and cache a connection to a local or
* remote JMX MBeanServer. This component is responsible for building an isolated connection/classloader
* to the managed resource's JMX MBeanServer. Each connection is isolated from other connections
* created by other instances of this component. This allows for it to do things like manage
* multiple JBossAS servers that are running on the same box, even if they are of different JBossAS
* versions. The same holds true for Hibernate applications - multiple connections can be created
* to different versions of the Hibernate MBean and due to the isolation of each connection, there
* are no version incompatibility errors that will occur.
*
* @author Greg Hinkle
* @author John Mazzitelli
*/
public class JMXServerComponent implements JMXComponent {
private static Log log = LogFactory.getLog(JMXServerComponent.class);

private EmsConnection connection;
private ConnectionProvider connectionProvider;

ResourceContext context;

public void start(ResourceContext context) throws Exception {
this.context = context;
log.info("Starting connection to JMX Server " + context.getResourceKey());

try {
internalStart();
} catch (Exception e) {
log.warn("JMX Plugin connection failure", e);
// The new model is to always succeed in starting, but warn about the errors (we only do this the first request)
/*throw new Exception("Unable to connect to Java VM ["
+ configuration.getSimple(JMXDiscoveryComponent.CONNECTOR_ADDRESS_CONFIG_PROPERTY).getStringValue()
+ "]", e);*/
}

if (connection == null) {
log.warn("Unable to connect to JMX Server " + context.getResourceKey());
}
}

protected void internalStart() throws Exception {
Configuration configuration = context.getPluginConfiguration();

String connectionTypeDescriptorClass = configuration.getSimple(JMXDiscoveryComponent.CONNECTION_TYPE)
.getStringValue();

if (LocalVMTypeDescriptor.class.getName().equals(connectionTypeDescriptorClass)) {
String commandLine = configuration.getSimple(JMXDiscoveryComponent.COMMAND_LINE_CONFIG_PROPERTY)
.getStringValue();

Map vms = LocalVMFinder.getManageableVirtualMachines();
if (vms != null) {
for (LocalVirtualMachine vm : vms.values()) {
if (vm.getCommandLine().equals(commandLine)) {
connectLocal(vm.getVmid());
}
}
}
} else if (JMXDiscoveryComponent.PARENT_TYPE.equals(connectionTypeDescriptorClass)) {
// We're embedded in another jmx server component without jmxremoting set so just use the parent's connection
this.connection = ((JMXComponent) context.getParentResourceComponent()).getEmsConnection();
this.connectionProvider = this.connection.getConnectionProvider();
} else if (J2SE5ConnectionTypeDescriptor.class.getName().equals(connectionTypeDescriptorClass)) {
// We're embedded in a J2SE VM with jmxremote defined (e.g. for jconsole usage)
String principal = null;
String credentials = null;
PropertySimple o = configuration.getSimple(JMXComponent.PRINCIPAL_CONFIG_PROP);
if (o != null) {
principal = o.getStringValue();
}
o = configuration.getSimple(JMXComponent.CREDENTIALS_CONFIG_PROP);
if (o != null) {
credentials = o.getStringValue();
}

ConnectionSettings settings = new ConnectionSettings();
J2SE5ConnectionTypeDescriptor desc = new J2SE5ConnectionTypeDescriptor();
settings.setConnectionType(desc);
settings.setServerUrl(configuration.getSimple(JMXDiscoveryComponent.CONNECTOR_ADDRESS_CONFIG_PROPERTY)
.getStringValue());
if (principal != null) {
settings.setPrincipal(principal);
}
if (credentials != null) {
settings.setCredentials(credentials);
}

prepareConnection(settings);

} else {
// This can handle internal connections (within the same vm as the plugin container) as well as
// any remote connections
ConnectionSettings settings = new ConnectionSettings();

String principal = null;
String credentials = null;
PropertySimple o = configuration.getSimple(JMXComponent.PRINCIPAL_CONFIG_PROP);
if (o != null) {
principal = o.getStringValue();
}
o = configuration.getSimple(JMXComponent.CREDENTIALS_CONFIG_PROP);
if (o != null) {
credentials = o.getStringValue();
}

settings.initializeConnectionType((ConnectionTypeDescriptor) Class.forName(connectionTypeDescriptorClass)
.newInstance());

settings.setConnectionType((ConnectionTypeDescriptor) Class.forName(connectionTypeDescriptorClass)
.newInstance());
settings.setServerUrl(configuration.getSimple(JMXDiscoveryComponent.CONNECTOR_ADDRESS_CONFIG_PROPERTY)
.getStringValue());

String installPath = configuration.getSimpleValue(JMXDiscoveryComponent.INSTALL_URI, null);
if (installPath != null) {
settings.setLibraryURI(configuration.getSimple(JMXDiscoveryComponent.INSTALL_URI).getStringValue());
}

if (principal != null) {
settings.setPrincipal(principal);
}
if (credentials != null) {
settings.setCredentials(credentials);
}
prepareConnection(settings);
}

this.connection.loadSynchronous(false);

}

public void stop() {
if (connection != null) {
connection.close();
connection = null;
}
}

protected void connectLocal(int vmid) {
// TODO GH: Refactor ems to also accept the vm itself
ConnectionSettings settings = new ConnectionSettings();
settings.setConnectionType(new LocalVMTypeDescriptor());
settings.setServerUrl(String.valueOf(vmid));
prepareConnection(settings);
}

protected void prepareConnection(ConnectionSettings settings) {
settings.getControlProperties().setProperty(ConnectionFactory.COPY_JARS_TO_TEMP, String.valueOf(Boolean.TRUE));
settings.getControlProperties().setProperty(ConnectionFactory.JAR_TEMP_DIR,
this.context.getTemporaryDirectory().getAbsolutePath());

addAdditionalJarsToConnectionSettings(settings);

ConnectionFactory cf = new ConnectionFactory();
cf.discoverServerClasses(settings);

this.connectionProvider = cf.getConnectionProvider(settings);
this.connection = this.connectionProvider.connect();
}

public EmsConnection getEmsConnection() {
return this.connection;
}

public AvailabilityType getAvailability() {
if (connectionProvider == null || !connectionProvider.isConnected()) {
try {
internalStart();
} catch (Exception e) {
log.debug("Still unable to reconnect resource: " + context.getResourceKey() + " due to error: "
+ e.getMessage());
}
}

return ((connectionProvider != null) && connectionProvider.isConnected()) ? AvailabilityType.UP
: AvailabilityType.DOWN;
}

public List discoverServices(ResourceType type, Configuration defaultPluginConfiguration) {
defaultPluginConfiguration.getSimple("objectName").getStringValue();

return null;
}

private void addAdditionalJarsToConnectionSettings(ConnectionSettings settings) {
// get the plugin config setting that contains comma-separated list of files/dirs to additional jars
// if no additional classpath entries are specified, we'll return immediately and not do anything to settings
Configuration pc = context.getPluginConfiguration();
PropertySimple prop = pc.getSimple(JMXDiscoveryComponent.ADDITIONAL_CLASSPATH_ENTRIES);
if (prop == null || prop.getStringValue() == null || prop.getStringValue().trim().length() == 0) {
return;
}
String[] paths = prop.getStringValue().trim().split(",");
if (paths == null || paths.length == 0) {
return;
}

// get the current set of class path entries - we are going to add to these
List entries = settings.getClassPathEntries();
if (entries == null) {
entries = new ArrayList();
}

// Get all additional classpath entries which can be listed as jar filenames or directories.
// If a directory has "/*.jar" at the end, all jar files found in that directory will be added
// as class path entries.
final class JarFilenameFilter implements FilenameFilter {
public boolean accept(File dir, String name) {
return name.endsWith(".jar");
}
}

for (String path : paths) {
path = path.trim();
if (path.length() > 0) {
if (path.endsWith("*.jar")) {
path = path.substring(0, path.length() - 5);
File dir = new File(path);
File[] jars = dir.listFiles(new JarFilenameFilter());
if (jars != null && jars.length > 0) {
entries.addAll(Arrays.asList(jars));
}
} else {
File pathFile = new File(path);
entries.add(pathFile);
}
}
}

// now that we've appended our additional jars, tell the connection settings about the new list
settings.setClassPathEntries(entries);

return;
}
}

Also, download MC4J libraries patch it with the changed class below and replace it inside JMX plugin after it is built (sincerely as I don’t use maven I don’t know a cleaner way of replacing it prior to build).
Below the changed code for WebsphereConnectionProvider:

/*
* Copyright 2002-2004 Greg Hinkle
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.mc4j.ems.impl.jmx.connection.support.providers;

import org.mc4j.ems.impl.jmx.connection.support.providers.proxy.GenericMBeanServerProxy;

import javax.management.MBeanServer;
import javax.management.j2ee.Management;
import javax.naming.Context;
import javax.naming.NamingException;
import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.net.URI;
import java.security.Security;
import java.util.Properties;

/**
* This Node acts as a connection to a WebSphere(tm) MBean Server (TMX4J based).
*
* @author Greg Hinkle (ghinkle@users.sourceforge.net), January 2004
* @version $Revision: 575 $($Author: ghinkl $ / $Date: 2006-05-21 23:38:53 -0300 (Dom, 21 Mai 2006) $)
*/
public class WebsphereConnectionProvider extends AbstractConnectionProvider {

protected GenericMBeanServerProxy statsProxy;
protected MBeanServer mbeanServer;
private Management mejb;

private static final String MEJB_JNDI = "ejb/mgmt/MEJB";

protected void doConnect() throws Exception {
Context ctx = null;

ClassLoader contextClassLoader = Thread.currentThread().getContextClassLoader();

Thread.currentThread().setContextClassLoader(this.getClass().getClassLoader());

try {

//System.setProperty("jmx.serial.form", "1.0");

/* From a WS admin article
Properties clientProps = new Properties();
connectProps.setProperty(AdminClient.CONNECTOR_TYPE,
AdminClient.CONNECTOR_TYPE_SOAP);
connectProps.setProperty(AdminClient.CONNECTOR_HOST, "localhost");
connectProps.setProperty(AdminClient.CONNECTOR_PORT, "8879");

AdminClient adminClient = null;
try
{
adminClient = AdminClientFactory.createAdminClient(clientProps);
}
catch (ConnectorException e)
{
System.out.println("Exception creating admin client: " + e);
}
*/

Class adminClientClass =
Class.forName("com.ibm.websphere.management.AdminClient", true, this.getClass().getClassLoader());
Class adminClientFactoryClass =
Class.forName("com.ibm.websphere.management.AdminClientFactory");

// TODO GH: LATEST! This works from a SUN VM...
// Autodetect the vm and suggest the correct factory
// Hashtable env = new Hashtable();
// env.put(Context.INITIAL_CONTEXT_FACTORY,
// "com.sun.jndi.cosnaming.CNCtxFactory");
// env.put(Context.PROVIDER_URL, "corbaname:iiop:localhost:2809/NameServiceServerRoot");
//env.put(Context.PROVIDER_URL, "iiop://localhost:2809/NameServiceServerRoot");
// ctx = new InitialContext(env);
//this.mejb = retrieveMEJB(ctx);

/*
Properties orbprops = new Properties();
orbprops .put("org.omg.CORBA.ORBClass", "com.ibm.CORBA.iiop.ORB");
orbprops .put("com.ibm.CORBA.ORBInitRef.NameService",
"corbaloc:iiop:localhost:2809/NameService");
orbprops .put("com.ibm.CORBA.ORBInitRef.NameServiceServerRoot",
"corbaloc:iiop:localhost:2809/NameServiceServerRoot");
ORB _orb = ORB.init((String[])null, orbprops );

org.omg.CORBA.Object obj = _orb.resolve_initial_references("NameService");
NamingContextExt initCtx = NamingContextExtHelper.narrow(obj);
Object objref = initCtx.resolve_str("java:comp/env/ejb/mgmt/MEJB");
ManagementHome home =
(ManagementHome)PortableRemoteObject.narrow(objref,ManagementHome.class);
this.mejb = home.create();
*/

//props.put(Context.SECURITY_PRINCIPAL, connectionSettings.getPrincipal());
//props.put(Context.SECURITY_CREDENTIALS, connectionSettings.getCredentials());

Properties props = new Properties();
URI serverUrl = new URI(connectionSettings.getServerUrl());

if (serverUrl.getScheme().equalsIgnoreCase("http") || serverUrl.getScheme().equalsIgnoreCase("https")) {
System.setProperty("javax.net.debug", "ssl,handshake,data,trustmanager");
//Security.addProvider(new sun.security.provider.Sun());
System.setProperty("java.protocol.handler.pkgs","com.sun.net.ssl.internal.www.protocol");
//System.setProperty("ssl.SocketFactory.provider", "javax.net.ssl.SSLSocketFactory");
props.put(
getConstant(adminClientClass, "CONNECTOR_TYPE"),
getConstant(adminClientClass, "CONNECTOR_TYPE_SOAP"));
} else {
props.put(
getConstant(adminClientClass, "CONNECTOR_TYPE"),
getConstant(adminClientClass, "CONNECTOR_TYPE_RMI"));
}
String username = connectionSettings.getPrincipal();
String password = connectionSettings.getCredentials();
boolean security = ((username != null) && (!"".equals(username)));
if (security) {
props.setProperty(getConstant(adminClientClass, "CONNECTOR_SECURITY_ENABLED"), Boolean.toString(security));
props.setProperty(getConstant(adminClientClass, "USERNAME"), username);
props.setProperty(getConstant(adminClientClass, "PASSWORD"), password);
}

props.put(
getConstant(adminClientClass, "CONNECTOR_HOST"),
serverUrl.getHost());
props.put(
getConstant(adminClientClass, "CONNECTOR_PORT"),
String.valueOf(serverUrl.getPort()));

Method createMethod =
adminClientFactoryClass.getMethod("createAdminClient", Properties.class);

Object adminClient =
createMethod.invoke(null, props);

this.statsProxy = new GenericMBeanServerProxy(adminClient);
this.mbeanServer = statsProxy.buildServerProxy();

//this.mejb = retrieveMEJB(ctx);

// TODO GH: Customize exception and error messages to help
// with typical problems (jsse jars missing, passwords, etc.)
} finally {
Thread.currentThread().setContextClassLoader(contextClassLoader);
}
}

public String getConstant(Class clazz, String name) throws Exception {
Field field = clazz.getField(name);
return (String) field.get(null);
}

// public Management getMEJB() {
// return mejb;
// }

private Management retrieveMEJB(Context ic) {
try {
java.lang.Object objref = ic.lookup(MEJB_JNDI);
// ManagementHome home =
// (ManagementHome)PortableRemoteObject.narrow(objref,ManagementHome.class);
// Management mejb = home.create();
return mejb;
} catch(NamingException ne) {
// ErrorManager.getDefault().notify(ne);
// } catch(RemoteException re) {
// ErrorManager.getDefault().notify(re);
} catch(Exception ce) {
// ErrorManager.getDefault().notify(ce);
}
return null;
}
public MBeanServer getMBeanServer() {
return this.mbeanServer;
}

public long getRoundTrips() {
return statsProxy.getRoundTrips();
}

public long getFailures() {
return statsProxy.getFailures();
}

}

Now, since the changes I made to the JMX plugin still don’t cover adding WebSphere libs to classpath it still necessary to add it manually, so copy the following jars from WebSphere to the Agent lib directory:

• AppServer/lib/bootstrap.jar
• AppServer/deploytool/itp/plugins/com.ibm.websphere.v61_6.1.200/ws_runtime.jar
• AppServer/runtimes/com.ibm.ws.admin.client_6.1.0.jar
• AppServer/plugins/com.ibm.ws.security.crypto_6.1.0.jar

Now the last part, the JVM that is running your agent needs the server certificate for the SOAP Admin port of your WebSphere installation. If you are using Firefox browse to your server on Admin Port you can check it clicking on the server instance under “Application Servers” and then expanding the Ports element and looking for the “SOAP_CONNECTOR_ADDRESS” entry. It is usually 8880. Double click on the padlock icon located on the bottom right corner, click display certificate then export it using the default format (X.509 Certificate (PEM)). Remember where you saved it for the next step.
Now run the following command (pointing to the JVM that will run your agent):

%JAVA_HOME%\jre\bin\keytool -import -alias somealias -file c:/temp/yourcertificate.crt -keystore %JAVA_HOME%\jre\lib\security\cacerts

Replace the %JAVA_HOME% with your JVM directory (or use the command as is in case the environment variable is set).

Now you are ready to setup the component inside JOPR.
For those that don’t want to build it manually, I’m providing the patched rhq-jmx-plugin-1.1.0.GA.jar attached in this PDF.

Note that the steps cited above for the certificate also applies if you are using IBM Tivoli Monitoring 6.2 (fp1 and onwards) together with a custom agent built using Tivoli Agent Builder. If you miss those steps you’ll see security error messages on the JMX agent log.