Problem
The workload processed large settlement files through an allocation-heavy execution model. Memory demand exceeded the practical limit and processing time varied between roughly thirty and fifty seconds.
Investigation
Profiling showed excessive intermediate objects, avoidable copying and insufficient use of CPU parallelism. The limiting factor was the execution model, not Kafka throughput or storage bandwidth.
Design
- introduced a dedicated parallel compute engine,
- partitioned work into independent CPU-bound units,
- removed high-volume temporary allocations,
- reworked data structures for better locality and bounded memory use.
Outcome
Processing fell to approximately four seconds and memory demand to about 2.5GB. The important result was not merely throughput. The workload became easier to size, reason about and operate.