Monday, April 4, 2016

WSO2 ESB Kafka Inbound and Kafka Connector Performance tuning

In this post, I am going to explain about Kafka connector and Kafka inbound endpoint performance with WSO2 ESB 4.9 and how to tune the performance for producer and consumer use cases. The performance test of Kafka inbound endpoint and Kafka connector was carried out on below machine configuration:

Performance Test Environment :
OS: Ubuntu 64-bit
Memory: 16GiB
Processor: Intel Core i7-4800MQ CPU 2.70GHzx8

ESB Kafka Connector

WSO2 ESB Kafka connector allows you to send the data to Kafka message broker.  


ThreadsMessages per a ThreadNo of MessagesA Message Size(byte)Throughput (/s)
101001,000106831
105005,0001061590
2050010,0001062251
10025025,0001062940
1,00015001,500,0001062780

Average Throughput(/s) - 2767

You can get highest throughput with default ESB configuration.

ESB Kafka Inbound Endpoint


WSO2 ESB Kafka Inbound endpoint allows you to consume the data from Kafka message broker. 


https://docs.wso2.com/display/ESB490/Working+with+Inbound+Endpoints



Above document includes inbound endpoint common parameters and how to change them for best performance. Even though I provide some recommended values with the performance tuning stats.

You can tune the Kafka inbound endpoint performance by changing the inbound thread pool values. The values can be changed in the location.


<ESB_HOME>/repository/conf/synapse.properties file


On  Sequential Mode

The following performance stat is measured with default inbound thread pool values and sequential mode.

ThreadsMessages per a ThreadNo of MessagesA Message Size(byte)Throughput (/s)
101001,00010616
105005,00010616
2050010,00010615

Average Throughput(/s) - 15

On Non Sequential Mode


The following performance stat is measured with default inbound thread pool values and non sequential mode.

inbound.threads.core = 20

inbound.threads.max = 100  

ThreadsMessages per a ThreadNo of MessagesA Message Size(byte)Throughput (/s)
101001,000106150
105005,000106151
2050010,000106126
10025025,000106136

We have gone through the performance test with default inbound thread pool.
Average Throughput(/s) - 135

Now we have increased the thread pool values and gone through the performance test.

inbound.threads.core = 200

inbound.threads.max = 1000

ThreadsMessages per a ThreadNo of MessagesA Message Size(byte)Throughput (/s)
10010010,000106769
100010001,000,000106800

We can get the throughput around 800 with above thread pool values. 
Average Throughput(/s) - 799

Now we have increased the inbound thread pool values further. The performance is measured with below inbound thread pool values and non sequential mode. The throughput is 711. There is no significant improvement compared with previous throughput.


inbound.threads.core = 500

inbound.threads.max = 2000

ThreadsMessages per a ThreadNo of MessagesA Message Size(byte)Throughput (/s)
10010010,000106757
1,00010001,000,000106710

Average Throughput(/s) - 711


You can configure the inbound common parameters according to this performance stats and see massive performance improvements with the inbound thread pool. To get highest performance, I recommend you to use the following inbound thread pool values with non sequential mode.

inbound.threads.core = 200
inbound.threads.max = 1000

Algorithm

BinaryGap A  binary gap  within a positive integer N is any maximal sequence of consecutive zeros that is surrounded by ones at both end...