Sunday, August 23, 2009

RabbitMQ java clients for beginners

Here is a sample of a consumer and producer example for RabbitMQ. The steps are
  1. Download Erlang
  2. Download Rabbit MQ Server
  3. Download Rabbit MQ Java client jars
  4. Compile and run the below two class and you are done.
This sample create a Durable Exchange, Queue and a Message. You will have to start the consumer first before you start the for the first time.

For more information on AMQP, Exchanges, Queues, read this excellent tutorial
http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/

+++++++++++++++++RabbitMQProducer.java+++++++++++++++++++++++++++

import com.rabbitmq.client.Connection;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.*;

public class RabbitMQProducer {
   public static void main(String []args) throws Exception {
 ConnectionFactory factory = new ConnectionFactory();
 factory.setUsername("guest");
 factory.setPassword("guest");
 factory.setVirtualHost("/");
 factory.setHost("127.0.0.1");
 factory.setPort(5672);
 Connection conn = factory.newConnection();
      Channel channel = conn.createChannel();
      String exchangeName = "myExchange";
      String routingKey = "testRoute";
      byte[] messageBodyBytes = "Hello, world!".getBytes();
      channel.basicPublish(exchangeName, routingKey
,MessageProperties.PERSISTENT_TEXT_PLAIN, messageBodyBytes) ;
      channel.close();
      conn.close();
      }
}

+++++++++++++++++RabbitMQConsumer.java+++++++++++++++++++++++++++

import com.rabbitmq.client.Connection;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.*;
public class RabbitMQConsumer {
    public static void main(String []args) throws Exception {
 ConnectionFactory factory = new ConnectionFactory();
 factory.setUsername("guest");
 factory.setPassword("guest");
 factory.setVirtualHost("/");
 factory.setHost("127.0.0.1");
 factory.setPort(5672);
 Connection conn = factory.newConnection();
      Channel channel = conn.createChannel();
      String exchangeName = "myExchange";
      String queueName = "myQueue";
      String routingKey = "testRoute";
      boolean durable = true;
      channel.exchangeDeclare(exchangeName, "direct", durable);
      channel.queueDeclare(queueName, durable,false,false,null);
      channel.queueBind(queueName, exchangeName, routingKey);
      boolean noAck = false;
      QueueingConsumer consumer = new QueueingConsumer(channel);
      channel.basicConsume(queueName, noAck, consumer);
      boolean runInfinite = true;
      while (runInfinite) {
            QueueingConsumer.Delivery delivery;
            try {
               delivery = consumer.nextDelivery();
            } catch (InterruptedException ie) {
               continue;
            }
         System.out.println("Message received" 
+ new String(delivery.getBody()));
         channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
      }
      channel.close();
      conn.close();
      }
}


See also 
  1. Purging a queuein rabbitmq
  2. Thumbnail generation using RabbitMQ
  3. Dumping Queue message contents
  4. Retrying failed messages
  5. Synchronously consume messages from a queue 

Saturday, August 22, 2009

Why do you become so valuable when you are leaving the company?

I work for a good software company and for the last 2 years they didn't do any appraisals. This year we worked like hell on a big project within the company and were expecting to get a decent raise but due to bad economy the appraisal were not done. The company still pays the market rate and due to bad economy none of the employees were thinking of switching jobs. Suddenly I got this job offer from another cool startup and I told my CTO on phone (because he was on vacation) that I am leaving. We had a long conversation on phone and I asked for a day to decide on it. He calls up again after 10 minutes and told me that if its a monetary decision to leave the company than I can tell him and the company do something about it and the company willing to offer me employee stock options to retain me. Not that I am leaving the company because of monetary reasons but how come suddenly I became so valuable that I am being offered stock options and some more money whereas no one talked about it for the last 2 years?

Is it a market phenomenon to get a raise by threatening to leave?

I didn't accepted the counter offer as my decision to leave was purely to work on some new technology but I feel bad about the other employees who are not given the option and still working hard. I guess they are just waiting for the right opportunity to make a switch as soon as the market improves.

Sql Optimization Tricks in oracle

When optimizing queries, the first thing I do with a slow query is figure out what it's trying to do. You can't fully optimize a query unless you know how to consider alternative ways to write it, and you can't do that unless you know what the query "means." I frequently run into a situation where I'm forced to stop and ask what they were trying to do with the query. This is database-agnostic, not related to Oracle.

  • Dont optimize if you dont need to :)

  • Use Oracle Grid Control to find the top queries and only work on them.
  • Check Oracle Grid Alerts to see if you are doing more I/O or more CPU or insufficient SGA is allocated, if queries are doing more sorts or using sort joins than a bigger sort area can help, similarly a bigger hash area can help for hash joins.
  • Avoid Union and use Union All if possible: - Union will require elimination of duplicates and if you know that the sets to union does not have duplicates than use Union all.
  • Avoid correlated subqueries: - If possible use denormalized fields. Like Weight and Volume were denormalized on Movement to avoid subqueries for each execution of report.
  • If operating on a large dataset than avoid filtering on multiple tables and than joining the result. In Transaction report we are filtering on shipment first to reduce the datset and than do all the big joins.
  • Avoid Sorting if possible: - this will consume CPU/memory resources on server and sometimes if sorting is on disk it will be I/O bound.
  • Avoid OR clause if possible: - This will cause the dataset to be iterated more than once depending on if the OR clause is the primary index.
  • Reduce the size of union set on temporary views: - The number of fields in the select clause should be reduced if we are doing a union. This will reduce the dataset size for union.
  • Use clustered indexes: - Clustered indexes will improve performance over non-clustered indexes as the data to filter is already in the index leading to less I/O.
  • Use function based indexes for leading wild card search: - We are using Reverse function based indexes for this.
  • Reduce the size of Dataset by adding Date filters if you can (e.g some page shows last 90 days and some pages shows last 5 days worth of data as thats what customer is mostly interested in) a
  • Use less logic in sql and if possible put it on app layer: - Previously cost component link logic was in sql causing unneccessay joins. We improved it by adding a temporary redirect jsp to have this logic.
  • Look at explain plan and besides cost look at the cardinality of dataset and try to reduce the no of bytes in operation
  • Look at CPU cost and see if a better index lookup can reduce it: - Sometimes queries can be optimized by adding a clustered index on _id and start_date, earlier we had single column indexes leading to bitmap conversion of individual indexes and than a bitmap join which was cpu intensive for 3 million rows.
  • Denormalize fields if you can at write time if read is costly remove joins
  • Sometimes you can't do much with the query than a better way is to step back and think if we need to design the tables in a different way or we need to do some logic in the app layer.
  • Use a covering index to reduce table lookup by rowId. Sometimes you join to a table only to fetch two to three columns and adding extra columns at the end of index might help the query as all data can be fetched from index instead of going to the table. Use these kind of indexes only for large dataset operations. For smaller dataset queries which access only 1000 or so rows dont use this trick.
  • Join condition with mismatched data types will never use an index. Using a cast to convert both to same datatype will do the trick
  • If Deletes or update on tables are slow and Oracle Grid control shows TM Contention as the wait event than this happening due to missing foreign key indexes. Use FindMissingFKIndex link to find missing foreign key indexes and create them.

  • Avoid Index full scans. For example a query like

    (SELECT SYS_ORG_ID FROM ORGANIZATION WHERE NVL(ORG_NAME, '$null') = ? AND NVL(ENT_NAME, '$null') = ? AND NVL(VC_ID, '-128') = ? ) 

    will use a full index scan whereas a query like

    (SELECT SYS_ORG_ID FROM ORGANIZATION WHERE ORG_NAME = ? AND ENT_NAME= ? AND VC_ID = ? )
    will use a unique scan.

Oracle TM contention

Assume there exists a table DEPARTMENT. Table EMPLOYEE references DEPARTMENT using field SYS_DEPT_ID. When SYS_DEPT_ID is not indexed, and EMPLOYEE has large no of rows, this can cause long-running TM lock on delete of a DEPARTMENT row because oracle has to figure out if there are any employees working on this department or not. You can use Oracle Grid control to see the wait events and if you see a TM contention wait event than use the following query to identify these types of missing indexes so they can be added.

SELECT * FROM (
SELECT c.table_name, cc.column_name, cc.position column_position
FROM user_constraints c, user_cons_columns cc
WHERE c.constraint_name = cc.constraint_name
AND c.constraint_type = 'R'
MINUS
SELECT i.table_name, ic.column_name, ic.column_position
FROM user_indexes i, user_ind_columns ic
WHERE i.index_name = ic.index_name
)
ORDER BY table_name, column_position;

How to find constraints with same columns but different name between two databases.

How to find constraints with same columns but different name between two databases.

select
prodschema.cons prod_constraint, prodschema.tbl prod_table, prodschema.cols prod_cols, localschema.cons local_constraint, localschema.tbl local_table, localschema.cols local_cols,
'ALTER table '|| prodschema.tbl ||' rename constraint ' || prodschema.cons || ' TO ' || localschema.cons || ';' sqlchg
from
(select
constraint_name cons,
constraint_type cons_type,
table_name tbl,
ltrim(max(sys_connect_by_path(column_name, ','))
keep (dense_rank last order by curr), ',') as cols
from
(select
ucc.constraint_name,
uc.constraint_type,
ucc.table_name,
ucc.column_name,
row_number() over (partition by ucc.constraint_name, ucc.table_name order by ucc.position) as curr,
row_number() over (partition by ucc.constraint_name, ucc.table_name order by ucc.position) -1 as prev
from
user_cons_columns@build3 ucc, user_constraints@build3 uc
where ucc.constraint_name = uc.constraint_name
and uc.constraint_name not like 'SYS_%')
group by constraint_name,constraint_type, table_name
connect by prev = PRIOR curr and constraint_name = PRIOR constraint_name and table_name = PRIOR table_name
start with curr = 1
order by cons,tbl) localschema,
(select
constraint_name cons,
constraint_type cons_type,
table_name tbl,
ltrim(max(sys_connect_by_path(column_name, ','))
keep (dense_rank last order by curr), ',') as cols
from
(select
ucc.constraint_name,
uc.constraint_type,
ucc.table_name,
ucc.column_name,
row_number() over (partition by ucc.constraint_name, ucc.table_name order by ucc.position) as curr,
row_number() over (partition by ucc.constraint_name, ucc.table_name order by ucc.position) -1 as prev
from
user_cons_columns ucc, user_constraints uc
where ucc.constraint_name = uc.constraint_name
and uc.constraint_name not like 'SYS_%'
)
group by constraint_name,constraint_type, table_name
connect by prev = PRIOR curr and constraint_name = PRIOR constraint_name and table_name = PRIOR table_name
start with curr = 1
order by cons,tbl) prodschema
where
prodschema.tbl = localschema.tbl
and prodschema.cols = localschema.cols
and prodschema.cons_type = localschema.cons_type
and prodschema.cons != localschema.cons
and prodschema.tbl not like 'BIN%'
order by prodschema.cons, prodschema.tbl;

How to find index with same columns but different name between two oracle databases.

How to find index with same columns but different name between two databases.

select
targetDB.idx targetDB_index, targetDB.tbl targetDB_table, targetDB.cols targetDB_cols, sourceDB.idx sourceDB_index, sourceDB.tbl sourceDB_table, sourceDB.cols sourceDB_cols,
'ALTER INDEX ' || targetDB.idx || ' RENAME TO ' || sourceDB.idx || ';' sqlchg
from
(select
index_name idx,
table_name tbl,
ltrim(max(sys_connect_by_path(column_name, ','))
keep (dense_rank last order by curr), ',') as cols
from
(select
index_name,
table_name,
column_name,
row_number() over (partition by index_name, table_name order by column_position) as curr,
row_number() over (partition by index_name, table_name order by column_position) -1 as prev
from
user_ind_columns@sourceDB
where index_name not like 'SYS_%')
group by index_name, table_name
connect by prev = PRIOR curr and index_name = PRIOR index_name and table_name = PRIOR table_name
start with curr = 1
order by idx, tbl) sourceDB,
(select
index_name idx,
table_name tbl,
ltrim(max(sys_connect_by_path(column_name, ','))
keep (dense_rank last order by curr), ',') as cols
from
(select
index_name,
table_name,
column_name,
row_number() over (partition by index_name, table_name order by column_position) as curr,
row_number() over (partition by index_name, table_name order by column_position) -1 as prev
from
user_ind_columns
where index_name not like 'SYS_%')
group by index_name, table_name
connect by prev = PRIOR curr and index_name = PRIOR index_name and table_name = PRIOR table_name
start with curr = 1
order by idx, tbl) targetDB
where
targetDB.tbl = sourceDB.tbl
and targetDB.cols = sourceDB.cols
and targetDB.idx != sourceDB.idx
order by targetDB.idx, targetDB.tbl;

Generating lots of dummy dual rows in oracle

SELECT ROWNUM AS rn
FROM dual
CONNECT BY ROWNUM < 1000000

Be remarkable

Seth Godin talks on why the Software you build should market itself instead of Marketing department doing it for you. Be Remarkable and create a tribe of your own if you want to succeed.

Seth Godin on why marketing is too important to be left to the marketing department

Thursday, August 13, 2009

Changing jobs in recession

Its a tough decision to change jobs in recession as finding a new job in case of eventuality can be tough. I have been through those times once when I had landed in US and just 3 months after the VCs decided to close the company. It was a tough time at that moment to find a job without a car.

But this time its a different story. I am now in a stable company and it has a recurring revenue model and it will run for 5-10 years easily. Looks like I have become comfortable in the job. Even though I put in 10-11 hours daily at the job for 5 days a week I just feel from inside that its time to change. The job is great and I am working in the platform team that develops frameworks that are used by other teams but something is missing. Not sure what but sometimes you keep on asking yourself this question is it time for a change?

I need a new spark in my life and job. That's why when the co-founders of my previous company contacted me for a job I was dreading on it for 3-4 months but were persistent to call me. Than one day one of the co-founder called and did a man to man talk and said "Looks like you have became comfortable at your job and do not want to take few risks". His argument was that when he at the age of 38 with two kids can take a risk of starting a company why can't I take a risk of just switching jobs?

I read it once that you grow only when you bring some change in your life and being comfortable in a job definitely means a chance of becoming obsolete soon.

Friday, August 7, 2009

Hadoop - Thinking at scale

Nice article on how to think for problems that are distributed and how not to think like distributed computing of 90s. Yahoo is running a 10000 cluster of machine to drive it web search. Now that microsoft will power yahoo search, what will yahoo do with that cluster ;).

http://www.cloudera.com/hadoop-training-thinking-at-scale

Samsung Epix is the worst phone I have used

I was using Samsung BlackJack II before and was pretty happy with its features and performance. Though I am not much of a windows mobile fan it did solved the purpose pretty well. I was very excited when my contract was getting over and saw that the big brother Samsung Epix was available from AT&T. I placed the order and waited for a loooong week to get my hands on it and Boy the touch screen and optical mouse was amazing. Problems started when I tried to connect this phone to wifi. Stupid phone wont connect to wifi. It will keep asking me for the network key. I turned off security on my wireless router and still this phone will continuously ask for Network key. I did a hard reset on phone and tried again and finally it connected to wifi. But after few mins the wifi connection was lost and it keeps asking for network key again. After 4 hours of struggling to get wifi working I gave up on it. Than I saw that an ID1 patch from Samsung is available. It took me 1 hour to get it up and running on my phone. But after I installed some other issues came up. The phone will go into hibernate/power save mode and in that mode it will drop calls. It will take 30 sec to 1 min for the phone to come up. 6-8 hours of frustration and finally I gave up on this phone. I am now ordering Blackberry curve. Lets see how it goes. I have high hopes on Curve.