Context
A payment settlement workflow processed large batch files on a schedule tied to a Kafka ingestion pipeline. Processing time and memory demand had grown with volume to the point where both were operational concerns, not just performance ones.
Problem
The workload processed large settlement files through an allocation-heavy execution model. Memory demand exceeded the practical limit and processing time varied between roughly thirty and fifty seconds.
Constraints
The fix had to fit inside the existing Kafka-based ingestion pipeline and file format — this was a redesign of the compute step, not a re-architecture of the surrounding system, and it had to run on the memory budget already provisioned rather than assume more RAM.
Investigation
Profiling showed excessive intermediate objects, avoidable copying and insufficient use of CPU parallelism. The limiting factor was the execution model, not Kafka throughput or storage bandwidth.
Approach
- introduced a dedicated parallel compute engine,
- partitioned work into independent CPU-bound units,
- removed high-volume temporary allocations,
- reworked data structures for better locality and bounded memory use.
Trade-offs
Adding another queue, another broker, or simply a larger heap were all rejected: profiling had already shown Kafka throughput and storage bandwidth were not the limiting factor, so those changes would have added operational surface (more infrastructure to run and monitor) without addressing the actual bottleneck, which was the allocation-heavy execution model itself.
Measured outcome
Processing fell to approximately four seconds and memory demand to about 2.5GB. The important result was not merely throughput. The workload became easier to size, reason about and operate.