Thursday, October 31, 2013

mysql order by and bind params

We use a grid to show files and when user clicks on a header we sort it.

I was using a statement like "order by :orderBy :sortDirection" and binding it in sql using spring but it was not doing any sorting.

After spending 1+hours on it I found out that mysql doesnt support bind parameters in order by and direction, yikes.

This is second weird thing I noticed in 1 day. The other thing I noticed was mysql doesnt support multicolumn updates

you cant do

update files set (size,ctime)=(select ctime,size from versions where versions.file_id=files.file_id) where file_id=:xxx

aparently you need to do a join query in update which makes the query very weird.

But even with issues Mysql rocks in performance.

Wednesday, October 2, 2013

Memcached stale keys on new code deployment and evictions

Scalability problems are interesting. This weekend we deployed a new release and suddenly we started getting ldap alerts from one datacentre. All fingers were pointed to something new in the code making more ldap queries but I didnt thought we made any ldap code changes, infact we were moving away from ldap to mysql.  We use 15G of memcached and looking into memcache stats I found that we were seeing evictions.  We had bumped memory from 12G to 15G in other data centre during same code and that data centre was not having issues. So this problem was interesting.

At last we found out that our new release had made 90% of memcached data stale. We store our objects as Json in memcached and deprecated fields removal from objects made the json in cache unusable. But the code bug was that the old jsons were still sitting in cache consuming memory.  As new jsons was getting pumped into memcached they were fighting for memory and memcached  LRU is slab based so when it evicts a slab it would evict good data also. This in turn was causing Thundering herd issue in LDAP.

LDAP is nice but when it comes to scaling more than 2-3 million objects it sucks as internally openldap uses bdb backend and after a point both read/writes start taking a long time no matter what you do. Good news is that in coming month we are moving from ldap->mysql which would make us scalable again.  I will write a blog on how to migrate millions of customers from ldap->mysql without them knowing about it. The analogy is changing tires of race cars on a race track and letting cars again on race track without disrupting the ongoing race.

Easmock partial mocking

Wow didnt knew I could partially mock a method on easy mock. Earlier i used to inherit the class and override methods to mock and have AtomicIntegers to determine which methods were called or not.  Recently I realized I was dumb and I should use easy mock to do that.

In below example I needed to very that if I get a json parsing exception for data stored in memcached then I need to delete the key from memcached. To test this I had to preserve all the code in MemCacheWrapper class but just mock the deleted method. here is how I did it.

    public void testDeleteOldKeyOnJsonParse() throws Exception {
        String key="user.1";
        MemCacheWrapper sut = EasyMock.createMockBuilder(MemCacheWrapper.class).addMockedMethod("delete").createMock();       
        sut.convertJsonToObject(key, User.class, "crap");

Creating a Platform rather than tool

There are few things that I like the most because they are simple, elegant and flexible by writing plugins.

1) Eclipse
2) Jenkins
3) Firefox
4) Ubuntu

recently I started playing more with jenkins and the more I play with it the better my respect for it grows, same happened with eclipse or firefox or ubuntu.

The reason I think all of these are great is that they are a platform rather than tool. They allow users to plugin their own code seamlessly.  This way you get users to write code for your tool and as soon as one guy writes a plugin you see the network effect because other guys now has more reasons to use the platform.

Offcourse creating a platform is much harder than writing a tool but given a choice I would rather create a platform than just a tool.