Friday, November 30, 2012

solr making AND as default operator

I had a requirement to make all optional parameters in query to match i.e. make all of them mandatory. We didnt wanted to write parsing logic and add + sign in front of each expresion.

It seems edismax was the answer and you have to just set mm to 100%. All you need to do from solrj is

        SolrQuery solrQuery = new SolrQuery();
        solrQuery.set("defType", "edismax");
        solrQuery.set("mm", "100%");

I also had this requirement to match use  user's input against 5-6 fields. There are two solutions:

1) user copyFields in schema.xml and append everything to a field called as "textDump" and then you can make this as the default field.

2) use dismax parser.

I chose to use dismax because copyFields will increase the index size to almost double and solr performance is directly proportional to index size.

Hurray dismax as it neatly solved both my requirements:).

Throwing more Hardware vs developer time in tuning on a single db

Last weekend I read two interesting articles on scaling up vs scaling out and then I read Jeff atwood's .  By the way I am a big fan of Jeff Atwood and if you guys haven't read him you should start reading him :).

But at our company we planned for a scale out model because being a startup sometimes management wont order hardware worth 100-200K or more in one shot and also if you have a table with 1B rows and you are doing agile programming then you are ought to build something and throw it in prod and then refine it. This can sometime lead to alter tables and data migration. While scaling up is good, doing alter table to add a column with default value on even 100M row table will incur significant downtime and we cant afford that. So when I designed our metadata db we chose to scale out.  As of now we are storing billions of rows in mysql database and last weekend we finally started chucking cassandra also out and moving this also to mysql. This would for sure add billions of more rows to the database and we are 30% done migrating the data and this weekend we would probably reach 80% ( this would be an interesting weekend). 

As highlighted in the  that the approach we take for a new feature is to put it in global non sharded db and if it doesn't scale we move it to sharded db.  So for our event store database that we are migrating from cassandra I was 100% sure that it would need sharding because we had billions of rows but for one other functionality the engineer was confident that he would have < 1M rows as he would delete the rows fast.

As it turns out there were other factors out of  control and even with 30% of the nodes the table reached 10M rows on Thanksgiving weekend and man it was chaos.

On Saturday morning   the load on db that always used to be 1-2 shot up to 10-15 and there was everyone from VP engg to CTO to VP of Ops and the developer.  There were some indexes missing so we added them and that brought the load down to 7-10 but still we can't put more nodes on this code.

Then we found that for debugging purpose we were keeping data for 1 week and almost half the table was completed task so we cronned a job that would move the completed data to some archive table and that brought the load down to 5.

But still we cant put more nodes so on Sunday we changed the real time polling of table to every 30 sec, which defeats the purpose of the final functionality, I mean indexing was still better than before but its not the performance we wanted.

Ultimately we decided that in 1-2 weeks we would change the code to move to the sharded db. But was it worth it, I mean initially during when proposed to the idea of writing the code to sharded db, one argument that was made that do we need this complexity and other was that he can scale it on a single db table.  I mean I also know that people store Billions of rows in a single mysql instance like what the above links of Plentyoffish and stackexchange team did, but these guys are may be exceptions and think about spending almost 1-2 days coordinating so many people on a single issue and a 1 week delay to adding more nodes to the system and that also on Thanks giving(every one in family was raising eyebrows). If you incur all the costs I am definitely sure we can buy 2-3 mysql servers in that cost :).

So in short if I have a slight hint that this feature wont scale on a single db then I would rather throw more hardware on it by adding it to sharded db from start.  Anyways in 1-2 weeks we would move to sharded db and all problems would be solved.

Soon I think this sharded db would require more mysql servers as we are going to outrun the current 12 Mysql master/slave pairs but with some automation and graphite we can predict when to add more servers. Right now as these servers have spare capacity so that buys me some time to work on some other critical infrastructure changes but soon I will come back to this to scale it to next level.

hierarchical locking using mysql database

We are a cloud file server company and one of the requirements we have is to be able to hierarchically lock the paths i.e. for a particula customer if one user is trying to move a path /shared/marketing/Dallas then no one should be allowed to add/delete files/folders in Dallas and its children, also no one should be able to make any edits to shared or marketing folder.  But users should be able to add files to any sibling hierarchies like /Shared/Engineering/test or /private/kpatel or /Shared/QA.

so the requirement is to have an api where I can do

try {
lockManger.lockPaths("/Shared/source", "/Shared/sbc", "/Shared/
Do some long running task (may be 1msec to 10 minutes).
} finally {

I evaluated some alternatives so here is the summary :
1) Apache commons has this locking api with class org.apache.commons.transaction.locking.DefaultHierarchicalLockManager
and it has a method called as lockInHierarchy that matches our reuirement 100% but only downside is that it uses inmemory locks and we are moving our servers to HA so this is not a solution.

2) Another alternative is to start a seprate  JVM and run DefaultHierarchicalLockManager inside it and expose apis to lock/unlock paths. I have been badly bitten by SPOFs and this is a SPOF because what if this tomcat goes down then you all your running jvms are in a limbo.

3) Another solution is to use Terracotta distributed locking api but I would have to roll out my own hierarchical locking on top of terracotta distributed locks.

4) Same problem was with hazlecast distributed locks.  Also the ops need to be famaliar with it and who will train ops and what if there are bugs then its a whole new learning curve.

5) Last alternative was to design our own locking api based on top of a mysql database table. From our graphite dashboard I can find that for a particular customer the concurrency is not more than 8-10 request per sec so I think designing a locking based on database will work.  The only issue is when you release the lock how do you notify the other waiting threads in a distributed manner. Now I could have used rabbitmq or something JMS for this but it sounds like an overkill considering most of the time a customer is working on diff paths. So I thought to start with a polling based approach +reverse exponential backoff for now and later introduce JMS if required which I dont think we will require.

So I created these tables

CREATE TABLE path_locks (
    customer_id BIGINT NOT NULL,
    path                   VARCHAR(4000) NOT NULL,
    creation_time         DATETIME NOT NULL,
    thread_id             VARCHAR(64) NOT NULL,
    node_id             VARCHAR(64) NOT NULL,

CREATE TABLE customer_locks (
    customer_id BIGINT NOT NULL,

and the logic was to

1)We delete all locks for that node_id on startup of a node and destroy of a node.
2) A quartz job will delete all expired locks (older than timeoutsec+1min).
3)Now to get a lock for /Shared/kpatel/Software/Releases
  while not lock acquired   
    lock sentinel row in customer_locks table for that customer_id to guard against race conditions between query and acquisition steps.
    Fire this query to detect lock conflicts

    select * from path_lock where threadId !='Thread1' and (creation_time-systime) < lockTimeout
          and (path like '/Shared/kpatel/Software/Releases/%' or path in ('/Shared','/Shared/kpatel','/Shared/kpatel/Software', '/Shared/kpatel/Software/Releases'))
        If you dont find any conflict then
            Insert the lock row
            Release sentinel lock.
        If you get any conflicts then
            check if you timedout waiting for lock then throw LockTimeoutException?.
            release sentinel lock
            wait for max(10, 1024/(retryCount%15))msec (reverse exponential backoff to reduce thread starvation). retryCount%15 is done to backoff and not pound the db continously as possibly this is some dead node issue or some long running lock hogger is doing its job and we have to wait.

The real thik in this solution is this reverse exponential backoff where if 10 threads come to acquire locks then we would not bombard the table or we would not starve the threads also as the thread that came first would be more aggresively querying than the thread that came late

This is going to be live soon and I am confident that this should be able to handle the load.

User perception using elevator joke

I am attending a startup lab workshop by Steve souders and he makes a very interesting note about user perception. I mean I had read about this in his book but he explains it much better using this joke. So it starts like:

An Apartment complex has two elevators and tenants are complaining about longer waits times and slow elevators so the owner calls a civil engineer and asks him what it takes to fix the issue. The civil engineer replies we need to add few more elevators and it would take 5 Million dollars and we have to close the complex for 6 months.

The owner was like hmm so he called an computer engineer and asks him what it takes to fix the issue. The computer engineer was like hmm i would need to first monitor the pattern over a time as to when and how people are using the elevator write and come up with some AI algorithms to optimize the solution.

The owner was like hmm so he called a systems engineer and he was like hmm all we need are TVs. The owner was like what? So the system engineer says yes just put some TVs and it doesnt matter even if its running crappy programs, all you need is to distract the user.

Moral of the story is that when you page loads you need to start rendering things ASAP, so that the user is engaged and knows something is going on and wont leave even if its slow.  That is the main reason windows and all installers shows a progress bar even if its not accurate , the user remains engaged in the progress bar and is constantly told that its coming , its coming :).

Thursday, November 1, 2012

Should you join a startup?

You should if you are passionate about learning new things and going through the pains of scaling the team and inventing something new.  But there are some cons also:

1) your healthcare benefits may suck. So if your wife has good benefits then this may be a moot point.

2) Your HR may suck, you might have no or non existent HR and sometimes small things can take an ample amount of things to be done. Like getting a document for your greencard process can take 2-3 months so dont join a startup if you are waiting for greencard as you can run into complications if the startup closes shop.

3) Startup may be demanding on you but at other times it can be a lot flexible also.

So if point 1 and 2 are not of concern then yes offcourse you should join it.

Brother MFC 490CW unable to clean 46

Day before yesterday after printing one page it started giving "unable to clean 46" and then I thought its an ink issue, as the yellow ink had ran out. I ordered new ink and plugged it in but it again gave same crap. Finally found this blog post with detailed steps and solution 2 worked for me

Thanks to whoever wrote this post.