Skip to main content

Posts

Showing posts from June, 2012

memcached evictions though memory is available

Ran into a performance issue where we would see intermittent slowness on application and it was traced to memcached evictions. Running stats command on memcached telnet port gave

STAT limit_maxbytes 8589934592
STAT bytes 6792297037
STAT total_items 1149728020
STAT evictions 192571322

As you can see even though close to 1G of memory is free we were seeing evictions. This article explained this in great detail and cleared some of my concepts
http://www.mikeperham.com/2009/06/22/slabs-pages-chunks-and-memcached/




Adding hostname or IP address to log4j logging for centralized logging

we use logstash for centralized logging and every tomcat needs to write to it using syslogappender. One requirement was to identify the log line by adding a IP address and port to the logged line.  This can be done using MDC feature of log4j or slf4j but problem is there can be lots of entry points into the logger.  The solution I came up is introduce template parameters (_@server_ip_@ and _@server_port_@) in log4j.xml as shown below and let the installer replace them at install time on each node.

            <param name="ConversionPattern" value="%d{yyyy-MM-dd'T'HH:mm:ss.SSSZ}{GMT+0} %p %t H-_@server_ip_@:_@server_port_@ D-%X{MDC_DOMAIN} %c - %m %throwable"/>

Can you spot the bug here? Findbugs can

I love this tool because it can find bugs that are hard to spot for humans.

                for (U user : users) {
                        if (user.getPrefix() != null && user.getPrefix().equals(userPrefix)
                                        && user.getOrgName() != null & user.getOrgName().equals(orgName)) {
                                return user;
                        }
                }

There is a bug in the if statement below that I wasnt able to spot and its an honest mistake on developer's part :).


There is a & instead of && in the if condition.





Memcached bulk api (spymemcached v/s memcached-client)

We use memcached as a cache to front mysql and we do caching at granular level. As we are a cloud filesystem company,  we cache files by path and folders by path. The problems comes when someone renames a top level folder that has 100K or more files in the hierarchy. The db operation is fast because in db all we need to do is one folder update (files are stored in normalized fashion so no rename is required at file level in db) but rename in memcache means add/delete of 100K keys. So that's like 200K operations and production code was spending close to 2 mins in memcache only. We use memcached-client java library in our code.  Memcache protocol doesn't support bulk-api so there was no easy solution than trying to set it in different threads but one colleague came across this interesting optimization that spymemcached guys has done spymemcached write optimizations.

I thought of writing a test program to compare memcached-client v/s spymemcached and spymemcached rocks.

1. On my …

From NOSQL to MySql (BDB to mysql) - Final Part

Finally last weekend we migrated all nodes to Mysql. As usual BDB gave last minute hiccups during migration but we were finally able to solve it.  There is BDB used in still some other other parts of the system where its used through background jobs. My next goal is to get rid of BDB from those areas and convert  them to Mysql.

Life after BDB is cool, Today is tuesday and there were no spikes whole Monday and today so far so I can focus on writing more code and solving other issues that were sidetracked before.

Part1
Part2
Part3
Part4
Part5

Engineering discipline in real life

I had signed up for frymire services who comes and tune up your AC 2 times a year for some $300 bucks.  This time the technician came and did some work on the AC in attic and then he went outside and did some work on compressor.  Then he left. On that day a cold front came and temp never go above 78 so my AC was not on, for last 2 days temp went above 78 so AC went on but instead of decreasing the temp it was increasing it. Whole night yesterday the temp kept on increasing from 78 to 84 and then decreased from 84 to 80(as the outside temp cooled off). There was a smell of mist in the air also. I was puzzled as to why the heck it was increasing instead of decreasing.  There was some thunderstorm 1 day back and there was an electrical outage also so my first suspicion was that it was because of that I went and tripped the power supply but that had no effect. Now being an enginner I didnt wanted to give up easily, I had no idea what an AC compressor is so googled a bit and thought of tri…