In this post, I am going to explain about Kafka connector and Kafka inbound endpoint performance with WSO2 ESB 4.9 and how to tune the performance for producer and consumer use cases. The performance test of Kafka inbound endpoint and Kafka connector was carried out on below machine configuration:
Performance Test Environment :
OS: Ubuntu 64-bit
Memory: 16GiB
Processor: Intel Core i7-4800MQ CPU 2.70GHzx8
ESB Kafka Connector
WSO2 ESB Kafka connector allows you to send the data to Kafka message broker.
Average Throughput(/s) - 2767
You can get highest throughput with default ESB configuration.
ESB Kafka Inbound Endpoint
WSO2 ESB Kafka Inbound endpoint allows you to consume the data from Kafka message broker.
https://docs.wso2.com/display/ESB490/Working+with+Inbound+Endpoints
Above document includes inbound endpoint common parameters and how to change them for best performance. Even though I provide some recommended values with the performance tuning stats.
You can tune the Kafka inbound endpoint performance by changing the inbound thread pool values. The values can be changed in the location.
On Sequential Mode
The following performance stat is measured with default inbound thread pool values and sequential mode.
Average Throughput(/s) - 15
On Non Sequential Mode
The following performance stat is measured with default inbound thread pool values and non sequential mode.
inbound.threads.core = 20
inbound.threads.max = 100
We have gone through the performance test with default inbound thread pool.
Average Throughput(/s) - 135
Now we have increased the thread pool values and gone through the performance test.
inbound.threads.core = 200
inbound.threads.max = 1000
We can get the throughput around 800 with above thread pool values.
Average Throughput(/s) - 799
Now we have increased the inbound thread pool values further. The performance is measured with below inbound thread pool values and non sequential mode. The throughput is 711. There is no significant improvement compared with previous throughput.
inbound.threads.core = 500
inbound.threads.max = 2000
Average Throughput(/s) - 711
You can configure the inbound common parameters according to this performance stats and see massive performance improvements with the inbound thread pool. To get highest performance, I recommend you to use the following inbound thread pool values with non sequential mode.
inbound.threads.core = 200
inbound.threads.max = 1000
Performance Test Environment :
OS: Ubuntu 64-bit
Memory: 16GiB
Processor: Intel Core i7-4800MQ CPU 2.70GHzx8
ESB Kafka Connector
WSO2 ESB Kafka connector allows you to send the data to Kafka message broker.
| Threads | Messages per a Thread | No of Messages | A Message Size(byte) | Throughput (/s) |
| 10 | 100 | 1,000 | 106 | 831 |
| 10 | 500 | 5,000 | 106 | 1590 |
| 20 | 500 | 10,000 | 106 | 2251 |
| 100 | 250 | 25,000 | 106 | 2940 |
| 1,000 | 1500 | 1,500,000 | 106 | 2780 |
Average Throughput(/s) - 2767
You can get highest throughput with default ESB configuration.
ESB Kafka Inbound Endpoint
WSO2 ESB Kafka Inbound endpoint allows you to consume the data from Kafka message broker.
https://docs.wso2.com/display/ESB490/Working+with+Inbound+Endpoints
Above document includes inbound endpoint common parameters and how to change them for best performance. Even though I provide some recommended values with the performance tuning stats.
You can tune the Kafka inbound endpoint performance by changing the inbound thread pool values. The values can be changed in the location.
<ESB_HOME>/repository/conf/synapse.properties fileOn Sequential Mode
The following performance stat is measured with default inbound thread pool values and sequential mode.
| Threads | Messages per a Thread | No of Messages | A Message Size(byte) | Throughput (/s) |
| 10 | 100 | 1,000 | 106 | 16 |
| 10 | 500 | 5,000 | 106 | 16 |
| 20 | 500 | 10,000 | 106 | 15 |
Average Throughput(/s) - 15
On Non Sequential Mode
The following performance stat is measured with default inbound thread pool values and non sequential mode.
inbound.threads.core = 20
inbound.threads.max = 100
| Threads | Messages per a Thread | No of Messages | A Message Size(byte) | Throughput (/s) |
| 10 | 100 | 1,000 | 106 | 150 |
| 10 | 500 | 5,000 | 106 | 151 |
| 20 | 500 | 10,000 | 106 | 126 |
| 100 | 250 | 25,000 | 106 | 136 |
We have gone through the performance test with default inbound thread pool.
Average Throughput(/s) - 135
Now we have increased the thread pool values and gone through the performance test.
inbound.threads.core = 200
inbound.threads.max = 1000
| Threads | Messages per a Thread | No of Messages | A Message Size(byte) | Throughput (/s) |
| 100 | 100 | 10,000 | 106 | 769 |
| 1000 | 1000 | 1,000,000 | 106 | 800 |
We can get the throughput around 800 with above thread pool values.
Average Throughput(/s) - 799
Now we have increased the inbound thread pool values further. The performance is measured with below inbound thread pool values and non sequential mode. The throughput is 711. There is no significant improvement compared with previous throughput.
inbound.threads.core = 500
inbound.threads.max = 2000
| Threads | Messages per a Thread | No of Messages | A Message Size(byte) | Throughput (/s) |
| 100 | 100 | 10,000 | 106 | 757 |
| 1,000 | 1000 | 1,000,000 | 106 | 710 |
Average Throughput(/s) - 711
You can configure the inbound common parameters according to this performance stats and see massive performance improvements with the inbound thread pool. To get highest performance, I recommend you to use the following inbound thread pool values with non sequential mode.
inbound.threads.core = 200
inbound.threads.max = 1000
