WSO2 ESB Kafka Inbound and Kafka Connector Performance tuning

In this post, I am going to explain about Kafka connector and Kafka inbound endpoint performance with WSO2 ESB 4.9 and how to tune the performance for producer and consumer use cases. The performance test of Kafka inbound endpoint and Kafka connector was carried out on below machine configuration:

Performance Test Environment :
OS: Ubuntu 64-bit
Memory: 16GiB
Processor: Intel Core i7-4800MQ CPU 2.70GHzx8

ESB Kafka Connector
WSO2 ESB Kafka connector allows you to send the data to Kafka message broker.  

Threads	Messages per a Thread	No of Messages	A Message Size(byte)	Throughput (/s)
10	100	1,000	106	831
10	500	5,000	106	1590
20	500	10,000	106	2251
100	250	25,000	106	2940
1,000	1500	1,500,000	106	2780

Average Throughput(/s) - 2767

You can get highest throughput with default ESB configuration.

ESB Kafka Inbound Endpoint

WSO2 ESB Kafka Inbound endpoint allows you to consume the data from Kafka message broker. 

https://docs.wso2.com/display/ESB490/Working+with+Inbound+Endpoints


Above document includes inbound endpoint common parameters and how to change them for best performance. Even though I provide some recommended values with the performance tuning stats.

You can tune the Kafka inbound endpoint performance by changing the inbound thread pool values. The values can be changed in the location.

<ESB_HOME>/repository/conf/synapse.properties file

On  Sequential Mode

The following performance stat is measured with default inbound thread pool values and sequential mode.

Threads	Messages per a Thread	No of Messages	A Message Size(byte)	Throughput (/s)
10	100	1,000	106	16
10	500	5,000	106	16
20	500	10,000	106	15

Average Throughput(/s) - 15

On Non Sequential Mode

The following performance stat is measured with default inbound thread pool values and non sequential mode.

inbound.threads.core = 20
inbound.threads.max = 100  

Threads	Messages per a Thread	No of Messages	A Message Size(byte)	Throughput (/s)
10	100	1,000	106	150
10	500	5,000	106	151
20	500	10,000	106	126
100	250	25,000	106	136

We have gone through the performance test with default inbound thread pool.
Average Throughput(/s) - 135

Now we have increased the thread pool values and gone through the performance test.

inbound.threads.core = 200
inbound.threads.max = 1000

Threads	Messages per a Thread	No of Messages	A Message Size(byte)	Throughput (/s)
100	100	10,000	106	769
1000	1000	1,000,000	106	800

We can get the throughput around 800 with above thread pool values. 
Average Throughput(/s) - 799

Now we have increased the inbound thread pool values further. The performance is measured with below inbound thread pool values and non sequential mode. The throughput is 711. There is no significant improvement compared with previous throughput.

inbound.threads.core = 500
inbound.threads.max = 2000

Threads	Messages per a Thread	No of Messages	A Message Size(byte)	Throughput (/s)
100	100	10,000	106	757
1,000	1000	1,000,000	106	710

Average Throughput(/s) - 711

You can configure the inbound common parameters according to this performance stats and see massive performance improvements with the inbound thread pool. To get highest performance, I recommend you to use the following inbound thread pool values with non sequential mode.

inbound.threads.core = 200
inbound.threads.max = 1000