Friday, June 22, 2012

svn find no of testcases added/modified in last 200 changes

got this requirement from our VP and it seems its easy to do this at command like with a command like

svn log -v -l200 https://svn.cloudXXX.com/repos/trunk |grep  Test|grep -v selenium|grep java|sort|uniq


memcached print server version

it seems its very simple to do it for a node in production. Just do

telnet 192.168.6.34
and then type STATS as command
the output should have a line like
STAT version 1.4.2

Thursday, June 21, 2012

memcached evictions though memory is available

Ran into a performance issue where we would see intermittent slowness on application and it was traced to memcached evictions. Running stats command on memcached telnet port gave

STAT limit_maxbytes 8589934592
STAT bytes 6792297037
STAT total_items 1149728020
STAT evictions 192571322

As you can see even though close to 1G of memory is free we were seeing evictions. This article explained this in great detail and cleared some of my concepts
http://www.mikeperham.com/2009/06/22/slabs-pages-chunks-and-memcached/




Adding hostname or IP address to log4j logging for centralized logging

we use logstash for centralized logging and every tomcat needs to write to it using syslogappender. One requirement was to identify the log line by adding a IP address and port to the logged line.  This can be done using MDC feature of log4j or slf4j but problem is there can be lots of entry points into the logger.  The solution I came up is introduce template parameters (_@server_ip_@ and _@server_port_@) in log4j.xml as shown below and let the installer replace them at install time on each node.

            <param name="ConversionPattern" value="%d{yyyy-MM-dd'T'HH:mm:ss.SSSZ}{GMT+0} %p %t H-_@server_ip_@:_@server_port_@ D-%X{MDC_DOMAIN} %c - %m %throwable"/>

Tuesday, June 19, 2012

Can you spot the bug here? Findbugs can

I love this tool because it can find bugs that are hard to spot for humans.

                for (U user : users) {
                        if (user.getPrefix() != null && user.getPrefix().equals(userPrefix)
                                        && user.getOrgName() != null & user.getOrgName().equals(orgName)) {
                                return user;
                        }
                }


There is a bug in the if statement below that I wasnt able to spot and its an honest mistake on developer's part :).


There is a & instead of && in the if condition.





Monday, June 18, 2012

Memcached bulk api (spymemcached v/s memcached-client)

We use memcached as a cache to front mysql and we do caching at granular level. As we are a cloud filesystem company,  we cache files by path and folders by path. The problems comes when someone renames a top level folder that has 100K or more files in the hierarchy. The db operation is fast because in db all we need to do is one folder update (files are stored in normalized fashion so no rename is required at file level in db) but rename in memcache means add/delete of 100K keys. So that's like 200K operations and production code was spending close to 2 mins in memcache only. We use memcached-client java library in our code.  Memcache protocol doesn't support bulk-api so there was no easy solution than trying to set it in different threads but one colleague came across this interesting optimization that spymemcached guys has done spymemcached write optimizations.

I thought of writing a test program to compare memcached-client v/s spymemcached and spymemcached rocks.

1. On my local laptop(connected to a local memcache)  100K add is 2 sec and 100K delete is 800 msec using spymemcached whereas it took 8 sec for 100K add and 6 sec for 100K deletes using memcached-client.
1. Connecting to a remote memcache and doing 100K adds took 6 sec using spymemcached and 40 sec using memcached-client.

Here are my Test programs
=============SpyMemcached================
import java.net.InetSocketAddress;
import net.spy.memcached.MemcachedClient;
import java.io.*;
public class TestSpyMemcached {
    public static void main(String[] args) throws IOException {
        String msg="{\"name\":\"test\",\"cName\":\"test\",\"cPath\":\"/shared/marketing/engineering/test\",\"path\":\"/Shared/marketing/Engineering/test\",\"folderId\":\"c08a0d93-8216-4262-b8ec-f1f24c9f9844\",\"parentId\":\"7d9e6fbc-9d0d-4bf8-90e3-aaa924df0777\",\"ctime\":1340051093864}";
    MemcachedClient c=new MemcachedClient(
            new InetSocketAddress(args[0], Integer.parseInt(args[1])));
        long start = System.currentTimeMillis();       
        for(int i=0;i<100000;i++) {
            c.set("key"+i, 3600, msg+i);
            if(i%1000==0) {
               System.out.println("added " + i + " objects");
            }   
        }
        System.out.println(System.currentTimeMillis()-start);
        start = System.currentTimeMillis();       
        for(int i=0;i<100000;i++) {
            c.delete("key"+i);
            if(i%1000==0) {
               System.out.println("deleted " + i + " objects");
            }   
        }
        System.out.println(System.currentTimeMillis()-start);
        c.shutdown();
    }
}

=============Memcached-client================
import java.net.InetSocketAddress;
import com.meetup.memcached.MemcachedClient;
import com.meetup.memcached.SockIOPool;
import java.io.*;
public class TestMemcachedClient {
    public static void main(String[] args) throws IOException {
        String msg="{\"name\":\"test\",\"cName\":\"test\",\"cPath\":\"/shared/marketing/engineering/test\",\"path\":\"/Shared/marketing/Engineering/test\",\"folderId\":\"c08a0d93-8216-4262-b8ec-f1f24c9f9844\",\"parentId\":\"7d9e6fbc-9d0d-4bf8-90e3-aaa924df0777\",\"ctime\":1340051093864}";

        SockIOPool pool = SockIOPool.getInstance();
        String[] memcacheServers = new String[]{args[0] + ":" + args[1]};
        pool.setServers(memcacheServers);
        pool.setHashingAlg(SockIOPool.NEW_COMPAT_HASH);
        pool.initialize();
        MemcachedClient mclient = new MemcachedClient();
            mclient.setSanitizeKeys(false);
            mclient.setCompressEnable(false);
        mclient.setPrimitiveAsString(true);
        long start = System.currentTimeMillis();       
        for(int i=0;i<100000;i++) {
            mclient.set("key"+i, msg+i, 3600);
            if(i%1000==0) {
               System.out.println("added " + i + " objects");
            }   
        }
        System.out.println(System.currentTimeMillis()-start);
        start = System.currentTimeMillis();       
        for(int i=0;i<100000;i++) {
            mclient.delete("key"+i);
            if(i%1000==0) {
               System.out.println("deleted " + i + " objects");
            }   
        }
        System.out.println(System.currentTimeMillis()-start);
    }
}

Tuesday, June 5, 2012

From NOSQL to MySql (BDB to mysql) - Final Part

Finally last weekend we migrated all nodes to Mysql. As usual BDB gave last minute hiccups during migration but we were finally able to solve it.  There is BDB used in still some other other parts of the system where its used through background jobs. My next goal is to get rid of BDB from those areas and convert  them to Mysql.

Life after BDB is cool, Today is tuesday and there were no spikes whole Monday and today so far so I can focus on writing more code and solving other issues that were sidetracked before.

Part1
Part2
Part3
Part4
Part5

Friday, June 1, 2012

Engineering discipline in real life

I had signed up for frymire services who comes and tune up your AC 2 times a year for some $300 bucks.  This time the technician came and did some work on the AC in attic and then he went outside and did some work on compressor.  Then he left. On that day a cold front came and temp never go above 78 so my AC was not on, for last 2 days temp went above 78 so AC went on but instead of decreasing the temp it was increasing it. Whole night yesterday the temp kept on increasing from 78 to 84 and then decreased from 84 to 80(as the outside temp cooled off). There was a smell of mist in the air also. I was puzzled as to why the heck it was increasing instead of decreasing.  There was some thunderstorm 1 day back and there was an electrical outage also so my first suspicion was that it was because of that I went and tripped the power supply but that had no effect. Now being an enginner I didnt wanted to give up easily, I had no idea what an AC compressor is so googled a bit and thought of tripping it.

I went out side and when I looked at the compressor outlet I was pissed, the stupid technician after cleaning the compressors hadnt put the plug back in so for last 2 days AC was running without compressor( no wonder it wont cool).  I was both thrilled on solving the puzzle but I was also pissed on the technician.  Engineering discipline it seems helps in real life also and it saved me some $150 bucks that frymire would have charged for coming to check on the AC.