Friday, October 19, 2012

Exposing jmx mbeans via spring for tomcat7

I needed to expose some jmx mbeans in tomcat for nagios monitoring and was reading http://tomcat.apache.org/tomcat-7.0-doc/monitoring.html and other things but it seems its a PITA to write an expose a bean. This didnt sounded right because I wanted to make it easy for junior developers to expose jmx monitors and not deal with all this complexity. Then I landed onto http://static.springsource.org/spring/docs/2.0.x/reference/jmx.html and voila the life became easy.  It took me some time to read and understand this but exposing jmx was 5 minutes.

So all I need to do is

1) create a file spring_jmx_mbeans.xml and add

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:aop="http://www.springframework.org/schema/aop"
    xmlns:tx="http://www.springframework.org/schema/tx"
    xsi:schemaLocation="
     http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
     http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-2.5.xsd
     http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-2.5.xsd">
    
    <bean id="exporter" class="org.springframework.jmx.export.MBeanExporter" lazy-init="false">
      <property name="autodetect" value="true"></property>
      <property name="namingStrategy" ref="namingStrategy"></property>
      <property name="assembler" ref="assembler"></property>
    </bean>
   
    <bean id="attributeSource" class="org.springframework.jmx.export.annotation.AnnotationJmxAttributeSource"/>
   
    <bean id="assembler" class="org.springframework.jmx.export.assembler.MetadataMBeanInfoAssembler">
      <property name="attributeSource" ref="attributeSource"/>
    </bean>
   
    <bean id="namingStrategy" class="org.springframework.jmx.export.naming.MetadataNamingStrategy">
      <property name="attributeSource" ref="attributeSource"/>
    </bean>   
</beans>

So I am asking exporter to  auto detect beans. The auto detection strategy is annotation based discovery. The naming strategy is also annotation based.


2) Include this file into applicationContext.xml like




3) Now whatever bean you want to expose just add @ManagedResouce annotation to it. To every getter add @ManagedAttribute to it.

You can override the  name of bean and attribute and give description in these annotation.


4) to see the jmx beans I just fired up jconsole and looked at the bean.


 However this is  not available for prod so thinking of using http://code.google.com/p/jminix/ for exposing jmx over http. Will write another blog post for it.

Spring is awesome!!  I will write another blog post about spring and quartz integration as it made my life easy few days back when I did quartz HA.



SLF4j logging exception with template params and stacktrace

I learn a new thing today.

I knew  that slf4j had two logger methods

org.slf4j.Logger.error(String, Object[])
org.slf4j.Logger.error(String, Throwable)

now If i wanted to call

        logger.error("This is {} message {}", "test" ,"to kp" ,e);


then I was under the impression that it would not print the stack trace as it would use the signature with object array.


But I was wrong :). It seems log4j has a nice trick and it would print the trace if the last argument is an exception object, a fellow programmer told me this that slf4j already handles this http://www.slf4j.org/faq.html#paramException .




No java programmers in Africa or south america?

huh was checking the stats for my blog and no hits from africa or south america. so either this is a bug in google analytics or very few people are programming in java in Africa or south america.

Thursday, October 18, 2012

log4j TCP SyslogAppender

The current SyslogAppender in log4j uses UDP to transmit the logs to syslog server. It potentially has risks of losing data and our ops guy was looking for a TCP based appender and ran into this http://www.rsyslog.com/tcp-syslog-rfc5424-log4j-appender/ and plugged it in. Then something weird happen as an appnode would keep running out of file handles and other app node ran into issue where all threads were stuck. The implementation at this link is buggy so I rewrote this and publishing here in case anyone is interested.

/**
 * TCP appender to syslog. This class uses a blocking queue with 10K message capacity and any requests beyond that would be rejected.
 * The append method from all caller threads inserts the message into blocking queue and there is a single background thread that logs to the syslog.
 * This complex queueing is introduced to relieve the user thread as soon as possible.
 *
 *
 */
public class Syslog4jTCPAppender extends Syslog4jAppender {
    private static final long serialVersionUID = 1L;

    private static ThreadLocal dateFormat = new ThreadLocal() {

        @Override
        protected SimpleDateFormat initialValue() {
            SimpleDateFormat df = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.S'Z'");
            df.setTimeZone(TimeZone.getTimeZone("UTC"));
            return df;
        }

    };

    private Map facilitiesMap = new HashMap();

    private String localHost;

    private BlockingQueue blockingQueue = new ArrayBlockingQueue(10000);

    @Override
    public void activateOptions() {
        super.activateOptions();
        String[] facilities = { "KERN", "USER", "MAIL", "DAEMON", "AUTH", "SYSLOG", "LPR", "NEWS", "UUCP", "CRON", "AUTHPRIV",
                "FTP", "LOCAL0", "LOCAL1", "LOCAL2", "LOCAL3", "LOCAL4", "LOCAL5", "LOCAL6", "LOCAL7" };
        int[] facIntArray = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 16, 17, 18, 19, 20, 21, 22, 23 };
        for (int i = 0; i < facilities.length; i++) {
            facilitiesMap.put(facilities[i], facIntArray[i]);
        }
        try {
            localHost = InetAddress.getLocalHost().getHostName();
        } catch (UnknownHostException e) {
            System.out.println("UnknownHostException" + e.getMessage());
            localHost = "Unknown";
        }
        SocketLoggerThread socketLoggerThread = new SocketLoggerThread();
        socketLoggerThread.setDaemon(true);
        socketLoggerThread.start();
    }

    @Override
    protected void append(LoggingEvent event) {
        int priority = calcPriority(event);
        String trace = super.layout.format(event);
        String newLineChar = "\n";
        String msg = trace.replaceAll(newLineChar, " ");
        Date dt = new Date();
        String dateString = dateFormat.get().format(dt);
        String message = "<" + priority + ">" + dateString + " " + localHost + " " + super.getIdent() + ": " + msg + "\n";
        blockingQueue.offer(message);
    }

    private int calcPriority(LoggingEvent event) {
        String facility = super.getFacility().toUpperCase();
        Integer facPriority = facilitiesMap.get(facility);
        if (facPriority == null) {
            facPriority = 1;
        }
        int level = event.getLevel().getSyslogEquivalent();
        int priority = facPriority * 8 + level;
        return priority;
    }

    private class SocketLoggerThread extends Thread {
        private Socket socket;

        private DataOutputStream os;
        private int counter;

        @Override
        public void run() {
            reinit();
            while (true) {
                try {
                    consume(blockingQueue.take());
                } catch (InterruptedException e) {
                    System.out.println("Syslog socket logger interrupted while waiting" + e.getMessage());
                } catch (Exception e) {
                    System.out.println("Unknown exception " + e.getMessage());
                }
            }
        }

        private void close() {
            try {
                if (os != null) {
                    os.close();
                }
            } catch (IOException e) {
                System.out.println("IOException closing os" + e.getMessage());
            }
            try {
                if (socket != null) {
                    socket.close();
                }
            } catch (IOException e) {
                System.out.println("IOException closing socket " + e.getMessage());
            }
        }

        private void reinit() {
            close();
            try {
                socket = new Socket(Syslog4jTCPAppender.super.getSyslogHost(), Integer.parseInt(Syslog4jTCPAppender.super.getPort()));
                os = new DataOutputStream(socket.getOutputStream());
            } catch (IOException e) {
                System.out.println("IOException opening socket" + e.getMessage());
            }
        }

        private void consume(String message) {
            try {
                counter++;
                os.writeUTF(message);
                if (counter % 5000 == 0) {
                    System.out.println("Reiniting syslog socket");
                    reinit();
                    counter = 0;
                }
            } catch (IOException e) {
                System.out.println("IOException writing message" + e.getMessage());
                reinit();
            }
        }
    }
}

Saturday, October 13, 2012

Dire need for centralized logging

We have HA in some components of the application and any request can go to any of the tomcats in Data centre. The problem is that lately I have been asked to chase a lot of customer issues and its a PITA to grep logs across all machines and make a time line and then figure out what's going on.  What we need is to log to some central server in addition to local box and then all these debugging things can be grepped here.  We are finally picking logstash for this purpose because you can do lucene like search queries on customer name (we append customer name in thread). Also we generate a unique requestId for all requests coming to data centre which gets passed through many components. We would put all those logs in logstash so we can see how the request flows through all components. We would keep 1-2 months of logs else it would be tons of logs.

We are finally picking http://syslog4j.org/docs/javadoc/org/productivity/java/syslog4j/impl/log4j/Syslog4jAppender.html for this but running into some issues with appender. Will write a blog once I solve it.

Saturday, October 6, 2012

Bcrypt and slow tests

We use Bcrypt hash to encrypt user's password. One of the strength of bcrypt is that its inherently slow to compute the hash and that makes the brute force breaking of password almost impossible.  But I recently ran into an interesting issue where almost overnight the build times on jenkins blow up from 12 mins to 22 mins and the build engineer complained about bulk user testcase as the culprit.

I ran it locally and found that the class had close to 15 tests and each test method was taking 20 sec and all I see is in setup we were creating 5 users and deleting 5 users in teardown.  So as usual best way to debug a performance problem is to look at the thread dump. I did a ps -ef on the running build and did a kill -QUIT to print the thread dump and immediately I saw time spent in Bcrypt computation. I took 3-4 more thread dump and all of them were stuck on bcrypt.  So I did two things:

1) I saw many methods in the test class that were testing pure validations and didnt needed the creation/deletion of users. So I moved them to a new Unit test class.
2) I added a JVM property to skip Bcrypt computation during user creation and passed that as a jvm argument to ant test target. This is to prevent not the skip in the prod environment.

And voila the build times were back to 13 mins.

Thursday, October 4, 2012

Ubuntu 12.04 Toshiba satellite L875D keyboard, mouse issues on login

I just bought a new laptop and then tried installing ubuntu 12.04 on it and ran into all sorts of issues from first wireless not working and then keyboard or mouse not working after installing the software updates.

The keyboard and mouse works fine on initial listing of operating systems but when the login prompt comes I cant enter anything on the screen. I was like WTF man.  Googling around gave me an idea of pluggin in an external mouse and usb keyboard and it worked. So I was even more frustrated.  Finally after combining things from various posts I figured out that I have to change grub2 config and add some more settings to it.

Open a terminal and enter

sudo vi /etc/default/grub

and then change the line GRUB_CMDLINE_LINUX="" to GRUB_CMDLINE_LINUX="i8042.nomux=1 i8042.reset"

then run

sudo update-grub

reboot the system and all issues were gone.

Brother MFC-490cw unable to print 70

My wife is studying as last year college student and one of her friend knows that I am a software engineer so yesterday she called my wife that the printer was not working.  Its funny I asked her to do a video chat and bring laptop near the printer and show me the issue.

First issue was that she had recently changed the ink so I asked her to take out the cartridge and show me and it seems she hasnt pulled the plastic wrapper in front of ink nozzle.

Now even after removing it the print was not working so I asked her to put something on scanner tray and try to copy it and again it was same issue "Unable to print 70".

It seems real issue was paper stuck issue near the cartridge and I asked her to remove it and volla the problem was solved.


Funnies part of the story was that I was going ROFL when I found unable to print 70 was related  to paper stuck issue, being an engineer I can feel the pain of a non IT user who would not know how the heck to connect unable to print 70 to paper stuck issue. For us its natural that we tweak different things and try again and repeat the cycle until we have found the issue :).