neoptex
Kafka Streams .NET – Is this the right approach for merging product data from Debezium?
Yes, I've already tested the NumStreamThreads parameter, and it does improve performance. But honestly, I expected better baseline performance even with just a single thread. Anyway, I definitely don't want to bother you. I was just wondering if anyone had already experienced similar performance. If not, it's clearly up to me to investigate further, not you! 😄
Thanks a lot for your help and your advice! ❤️
11 replies
Kafka Streams .NET – Is this the right approach for merging product data from Debezium?
When I look at typical large-scale data processing scenarios, like in ETL pipelines where joins are also commonly used, I feel like I can’t be the only one doing this kind of operation. So even though I’m only using a single Kafka partition for now, the huge gap between the speed of raw message ingestion and the slowness of the join really makes me think I’m doing something wrong somewhere
11 replies
Kafka Streams .NET – Is this the right approach for merging product data from Debezium?
From Debezium to the table topics, there's no issue, everything is very fast. What really surprises me is the performance difference between simply sending messages and performing the JOIN operation. Kafka itself can handle millions of messages without problems, and I expected Kafka Streams to offer similar performance
11 replies