Friday, February 28, 2014

taking svn backup

I used to think its rocket science taking svn backup but I was wrong. It seems its a piece of cake. Just login to the server and run

 svnadmin dump /array/xxx/svn/repos/trunk > trunkrepo.dump

Friday, February 21, 2014

Switching to Cloud only email

4 years ago I was a windows guy and used to use outlook for emails. Though I used gmail for personal accounts when it comes to office emails somehow I liked the native look and feel and offline nature of outlook.  Then I joined the startup and the development environment was ubuntu so I first started using thunderbird but then moved to evolution. Now evolution  had its own problems but again it was native and i was using it mainly as a task management. Daily emails would flow in via pop/imap and I would keep a copy of them on server. Then as and when I would  reply to an email or took action on it I would move  it to a folder by year. But evolution used to suck, it had performance issues handling 2000+emails so I started deleting emails to trash but again it had performance issues. But I kept using it as I could integrate gmail calendar and it would notify me about events and contacts were offline so when I was on aeroplane I could use it.

Then I moved to ubuntu 12.04 and again evolution had its own set of issues it would hog the CPU and fan would go crazy, last  week I got tired of it and thought let again try to follow the same model with gmail and using it. So first thing I did is downloaded chrome and downloaded both gmail and calendar offline apps and I changed my workflow to if I took action on email I would mark it as read else mark it as unread.  Calendar and gmail has this desktop notifications so I enabled it also.

The only issue I have faced so far is randomly it would be slow for me and the browser will keep saying  I am disconnected whereas other sites would work fine. Also may times sending emails would take 1+ minutes. But so far I am liking it and life is good and back to normal.

Friday, February 14, 2014

A journey of Scaling generation of file system snapshots

We are a cloud file system startup managing billions of files for thousands of customers and one of the constant scaling issue is "how do you generate a snapshot of cloud filesystem" and send it to the client to start an initial sync. To elaborate more let say a customer has so far uploaded 25M files. Now he starts a new office and wants to setup a Netgear/NAS appliance and install our server sync software. Before the Netgear/NAS appliance could start syncing it needs a consistent snapshot of cloud filesystem so it could start that as a starting point before it could start syncing  changes based on events.  Now 25M is a big number right now but 5 year back the same problem was with if a customer has 500K files and a new employee joins and installs a sync software on his laptop, how do you send the sync client a consistent server snapshot.

It seems every 1-2 year or so we solve this problem and as scale increases we have to come up with something new to make it more faster than before.

Snapshot solution Version 1

When we initially started we were using filers and using NFS and when a customer would add a folder on local or from web ui, we on our server would replicate the operation via NFS.   When  a new user sync install would request a snapshot we would crawl the file system and send him a snapshot.  This worked for sometime but crawling the file system is slow and NFS would freeze the OS.

Snapshot solution Version 2

To solve the crawling issue we started a python process on the filer that would start with an initial image via crawling and it would hold it in memory. It would then listen to INotify events when we would do operation via NFS and it would update the  in memory snapshot.  When a new user sync install would request the snapshot we would serve it from the cached in memory install.  This worked great for sometime but it ran into issues whenever we would bring restart a filer or the python process would restart or the snasphot was so big that it would cause OOM.

Snapshot solution Version 3

We ditched NFS and we started storing filesystem metadata in a NOSQL database called as BerkelyDB. Its a key value database and we would store the file on filer but BerkelyDB would store the metadata about file XYZ belongs to folder ABC and is located on ZZZ filer at this location. When a new user sync install would request the snapshot we would just dump all BerkelyDB data(paths were stored as key and folder metadata as values) and generate  a snapshot.  This worked for a long time until we started reaching customers with 3M+ files and this would buckle under pressure if multiple clients would request snapshots at same time. BerkelyDB would run into memory issues or cache thrashing.

Snapshot solution Version4

Besides cache/memory issues  BerkelyDB's biggest issue was data loss and replication. So we ditched BerkelyDB and started using Mysql. We created Folder,File,Version table and 1000s of shards to store metadata. We are right now running 100+ mysql servers to store this data. One customer's data was stored in one shard so whenever a user sync install would request the snapshot we would join the three tables and spit out the snapshot.  This was one big breakthrough from other areas of application also.  But one big problem with this approach was that we allow customers to upload any number of versions and the snapshot only cares about latest version of the file. So in order to derive latest version of the file we had to discard old versions using a correlated subquery.  So this works fine for 5M files but it really buckled up under 8M+ files as we had to fire a correlated subquery for each file.

Snapshot solution Version5
We found out that if we joined Folder,File,version table and then use unix sort to sort the file and then use python to filter out latest version it was much faster than correlated queries. But this also buckled under pressure when we hit a customer with 25M files. The join in mysql db takes 2 hours to spit the data and the unix sort takes 2-3 hours.  Also the snapshot is large so we started caching this on disk and we used nginx as a reverse proxy to start serving cached snapshot. But again lately some big customers started facing issues with snapshot timeouts.

Snapshot solution Version6
This weekend we are again improvising on this solution as we found out that if we could denormalize the latest version when the file is being added,copied, moved,deleted,restored on the File table record, then to generate a snapshot we just need to join folder,file table and serve it.  From our tests we found out that generating a snapshot on 25M file customer takes only 6 minutes compared to 5 hour process with join query and unix sort approach.  Off course this has twice the space complexity but the time complexity is way better.

Offcourse this requires a lot of grunt work to modify the old code and we would need to migrate billions of files so all this would rollout slowly over the course of this month but I am hoping this solution would again buy us some time.

The core problem is snapshot and we need to find out a better way to bootstrap a client instead of sending him the entire snapshot so may be this saga has more evolution than what we had seen in the past :).

Thursday, February 6, 2014

Redis publish subscribe to node.js

Disclaimer: This is no way a comparison between RabbitMQ and Redis. RabbitMQ is a great product and we use it in our startup as a queuing system. I just wanted to try out redis in my pet project so I did it.

I am intrigued by the hype around node.js so am planning of writing a node.js to push real time events from server and do push notifications in browser from server.

Now mostly I saw that people use a myriad of technologies from websocket, nginx, nodejs, redis/rabbitmq and in backend produce the message from Java or any other app.  Basically node.js is only used for long polling and decoupled from the main app using a message queue as a broker. So my first goal was to push message from Java to message queue and I have already used RabbitMQ in past so I thought let me this time try redis and boy it is much much easy then RabbitMQ (however at this moment I trust rabbitmq more than redis for production as I have much more experience scaling rabbitmq as a queuing system,  I just wanted to try out a different queuing system).

To connect to redis from java there are multiple implementations like JRedis and Jedis, in past in my startup we had used JRedis and then switched to Jedis but I am not happy with that code as we did all code ourselves so thought to check if spring has something because I am a big time fan of spring-jdbc and luckily I found spring has a similar abstraction to hide all this details using its spring-data-redis templates.  I used spring-data-redis-1.1.0.RELEASE and jedis-2.1.0.jar and I ran into issues with spring-3.X so I upgraded to spring-tx-4.0.1.RELEASE.jar

all I needed to do was to add this in spring config

 <bean id="jedisConnectionFactory" class=""
        p:host-name="" p:port="6379"/>
      <bean id="stringRedisTemplate" class=""
      <bean id="messagePublisher" class="">
        <property name="stringRedisTemplate" ref="stringRedisTemplate" />

and then

public class MessagePublisher {
    private static final AppLogger logger = AppLogger.getLogger(MessagePublisher.class);
    private StringRedisTemplate stringRedisTemplate;

    public StringRedisTemplate getStringRedisTemplate() {
        return stringRedisTemplate;

    public void setStringRedisTemplate(StringRedisTemplate stringRedisTemplate) {
        this.stringRedisTemplate = stringRedisTemplate;

    public void publishMessage(String channel, String message) {
        logger.debug("Publishing {} to channel {}", message, channel);
        stringRedisTemplate.convertAndSend(channel, message);

    public void publishJsonMessage(String channel, Object msgObject) {
        publishMessage(channel, JsonUtil.generateJSON(msgObject));

Now can you beat this.  2 lines of code to publish a message to message queue.  Totally impressed and icing on cake is that if you have to switch the library from jedis to Jredis then its just a config change and no code change is required and spring is handing all connection open/close.