Thread-per-Core Architecture

Performance question and hypothesis

Question: can ownership partitioning remove synchronization from the hot path, one core at a time?

Hypothesis: when each thread exclusively owns its partition of state, the hot path needs no lock and no atomic read-modify-write, so per-core throughput scales with core count where a shared worker pool's collapses — the costs don't vanish, they move to cross-core handoff and load imbalance.

What would disprove it: if the partitioned design's aggregate throughput failed to scale versus the shared-pool baseline, or if its hot-path per-operation cost still carried contention signatures (e.g. false sharing between partitions), or if a perfectly balanced load showed the same win as a skewed one, the model's claims would be wrong.

Learning objective

Explain thread-per-core as an ownership and execution model, contrast it against a shared worker pool, show how partitioned ownership eliminates lock contention but introduces cross-core handoff cost, demonstrate that partitioning alone does not fix load imbalance, explain the affinity/scheduler and NUMA caveats that determine whether the model's assumptions hold on real hardware, and show bounded-queue backpressure as the honest response to overload.

Prerequisites

• The SPSC Ring Buffer lab — cross-core handoff below is exactly one SPSC ring buffer per core-to-core link.
• The CAS Contention and Backoff lab — the shared worker pool's lock contention below is the same cost that lab describes, via a lock rather than a bare CAS loop.

Theory

Thread-per-core is an ownership model, not a thread count

"One thread per logical CPU" describes a number. Thread-per-core describes a discipline: each thread exclusively owns a partition of the application's state for its entire lifetime, and no other thread ever touches that partition directly. A program that spawns one thread per core but still lets any thread touch any shared data through a lock is a worker pool sized to the core count, not a thread-per-core architecture.

Shared worker pool vs. owned-state execution

Shared worker pool: N worker threads, all able to process any request, coordinating through shared state protected by a lock. Simple and flexible — but every request touching shared state pays the lock's cost, and only one can hold that lock at a time, regardless of how many workers exist.

Thread-per-core ownership: each core's thread owns a disjoint partition and only that thread ever touches it. No lock, because there is no sharing — the "single-writer alternative" from the CAS Contention lab, applied as the primary architecture. This buys real parallelism at the cost of needing an explicit mechanism for requests that arrive on the wrong core.

Cross-core handoff

A request for partition P arriving on a core that isn't P's owner must be handed off — never processed directly by the wrong core. The standard mechanism is one bounded SPSC channel per ordered pair of cores, using the same reservation → publication → read → acknowledgement discipline the SPSC Ring Buffer lab establishes. Handoff is not free: it costs at least one extra scheduling round through the channel.

Hot partitions: ownership does not equal balance

Partitioning by owner does not guarantee even traffic distribution. If one key is disproportionately popular, its owning core's inbox grows while every other core sits idle — the same total capacity exists but cannot spill onto underused cores the way it could in a shared pool. Repartitioning, splitting a hot key, or accepting bounded queueing/backpressure on the hot core are the standard responses.

Backpressure

Every core's inbox is bounded, exactly like the SPSC Ring Buffer lab's capacity. When full, reject (or have the sender block) rather than let the queue grow without bound — an unbounded queue under sustained overload only converts memory pressure into unbounded latency before failing anyway.

Affinity and scheduler caveats

Thread-per-core's zero-contention argument assumes each owning thread keeps running on (approximately) the same physical core. This is not automatic: CPU affinity support differs by OS — Linux exposes sched_setaffinity/taskset fairly directly; macOS does not expose hard affinity to user processes (THREAD_AFFINITY_POLICY is only a hint); Windows exposes SetThreadAffinityMask with its own quirks. Containers and cgroups add another layer — CPU quotas and cgroup CPU-sets interact with affinity, and orchestrators may migrate containers across hosts entirely outside the application's control. Even with affinity requested, the OS scheduler can still migrate a thread for priority, power, or thermal reasons. A migration does not break correctness — the thread still owns the same partition — it degrades performance by losing cache locality on the old physical core.

NUMA caveats

On multi-socket (NUMA) hardware, memory attached to one socket is more expensive for a core on a different socket to access. Thread-per-core's benefit compounds with NUMA-aware placement — pin a partition's thread to a core and allocate that partition's memory on the same NUMA node — but this lab's model does not simulate NUMA distance or memory placement. A production system needs an explicit memory allocation strategy (numactl, libnuma) in addition to CPU affinity.

Java and Rust

public final class PartitionedCounter {
    private long value;
    public long increment() { return ++value; }
    public long get() { return value; }
}

public final class SharedCounterPool {
    private final long[] counters;
    private final ReentrantLock lock = new ReentrantLock();

    public SharedCounterPool(int partitions) { counters = new long[partitions]; }

    public long increment(int partition) {
        lock.lock();
        try {
            return ++counters[partition];
        } finally {
            lock.unlock();
        }
    }
}

@State(Scope.Benchmark)
@BenchmarkMode(Mode.Throughput)
@OutputTimeUnit(TimeUnit.MILLISECONDS)
@Fork(1)
@Warmup(iterations = 5, time = 1)
@Measurement(iterations = 10, time = 1)
public class ThreadPerCoreBenchmark {
    private static final int PARTITIONS = 4;
    private final SharedCounterPool sharedPool = new SharedCounterPool(PARTITIONS);

    @Benchmark @Threads(4)
    public long sharedPoolIncrement() {
        int partition = ThreadLocalRandom.current().nextInt(PARTITIONS);
        return sharedPool.increment(partition);
    }

    @State(Scope.Thread)
    public static class OwnedState {
        final PartitionedCounter counter = new PartitionedCounter();
    }

    @Benchmark @Threads(4)
    public long ownedPartitionIncrement(OwnedState state) {
        return state.counter.increment();
    }
}

pub struct PartitionedCounter {
    value: u64,
}

impl PartitionedCounter {
    pub fn new() -> Self { Self { value: 0 } }
    pub fn increment(&mut self) -> u64 {
        self.value += 1;
        self.value
    }
    pub fn get(&self) -> u64 { self.value }
}

use std::sync::Mutex;

pub struct SharedCounterPool {
    counters: Mutex<Vec<u64>>,
}

impl SharedCounterPool {
    pub fn new(partitions: usize) -> Self {
        Self { counters: Mutex::new(vec![0; partitions]) }
    }

    pub fn increment(&self, partition: usize) -> u64 {
        let mut guard = self.counters.lock().unwrap();
        guard[partition] += 1;
        guard[partition]
    }
}

fn run_shared_pool() {
    let pool = Arc::new(SharedCounterPool::new(PARTITIONS));
    let handles: Vec<_> = (0..PARTITIONS).map(|partition| {
        let pool = Arc::clone(&pool);
        thread::spawn(move || { for _ in 0..INCREMENTS_PER_THREAD { pool.increment(partition); } })
    }).collect();
    for h in handles { h.join().unwrap(); }
}

fn run_owned_partitions() {
    let handles: Vec<_> = (0..PARTITIONS).map(|_| {
        thread::spawn(move || {
            let mut counter = PartitionedCounter::new();
            for _ in 0..INCREMENTS_PER_THREAD { counter.increment(); }
        })
    }).collect();
    for h in handles { h.join().unwrap(); }
}

Benchmark methodology

Partitioned per-thread ownership is measured against a shared-pool baseline with JMH and a matching Rust harness; a correctness gate runs before any timing is trusted.

Common mistakes

• Calling a design "thread-per-core" because it spawns one thread per CPU, while still routing every request through shared, lock-guarded state — that's a worker pool sized to the core count, not ownership.
• Assuming partitioning alone balances load — a skewed key distribution still overloads one partition's owning core while others idle (see "Hot partitions").
• Processing a request directly on the "wrong" core to avoid handoff latency — this silently reintroduces the shared-mutable-state problem ownership was meant to eliminate.
• Assuming CPU affinity is a hard guarantee on every platform — macOS in particular treats it only as a hint the scheduler may ignore.
• Letting a core's inbox grow unboundedly under overload instead of rejecting — this converts memory pressure into unbounded latency rather than failing fast.

Investigation task

Using the lab's Java or Rust implementation :

1. Run the benchmark and record the shared-pool vs. owned-partition throughput ratio on your own hardware.
2. Vary the thread/partition count (2, 4, 8) and re-run — observe how the shared pool's throughput changes (or doesn't) as thread count grows, versus the owned-partition case.
3. If your OS supports it, pin the benchmark process to specific cores (e.g. taskset on Linux) and re-run — note whether this changes the owned-partition result, and by how much.
4. In the interactive model, work through "Hot partition" and record the step at which core 1 first rejects a request — then work through "Backpressure" and compare how quickly rejection occurs with a smaller burst against a single core.
5. Write down your CPU model and core count, your measured numbers, and explain any difference from the disclosed numbers above.

Accessibility notes

• The interactive model is fully keyboard operable: the scenario selector and the Java/Rust code tabs are roving-tabindex tablists (arrow keys, Home/End); every step/reset control is a native <button>.
• Every state change is announced to screen readers via an aria-live="polite" region, in addition to the visible per-core status badges and the state inspector table — state is never colour-only.
• This page has no auto-playing or looping animation; step transitions resolve instantly, so there is nothing beyond the site-wide CSS rule to gate on prefers-reduced-motion.
• With JavaScript disabled, this page's theory, Java and Rust code, benchmark methodology, trade-offs and sources remain fully readable — only the interactive model and the graded quiz require script.

Review questions

Self-check before or after the graded quiz below:

1. Why does a shared worker pool's throughput not scale with worker count, even though many workers are available?
2. What specific cost does cross-core handoff add that direct, same-core dispatch does not?
3. Why doesn't partitioning state by owner automatically balance load across cores?
4. Give one reason CPU affinity requested by an application might not actually be honored by the OS.

Exercises

One diagnosis exercise and one implementation exercise, each with success criteria. Hints and solutions are collapsed — attempt the exercise first. The executable benchmark and correctness suite are used internally to produce the evidence shown by this laboratory.

1. Diagnosis — partitioned, and still not scaling

Per-thread partitions store 3 × 8-byte stat slots contiguously in one big array; 8 threads barely reach 2× single-thread throughput with no locks anywhere and all cores at ~100%. Explain why per-thread slots are not per-thread cache lines, quantify the collision for this layout, and give two independent fixes preserving exact counts. Success criteria: you name the mechanism, count partitions per 64-byte line, and your fixes keep the final counts identical.

2. Implementation — make imbalance visible, then bound it

Add a skewed-load scenario (80% of operations to one partition) and the simplest mitigation that doesn't reintroduce sharing: bounded queues with explicit backpressure between ingress and partition owners. Success criteria (measure, don't assert): nothing is silently lost (processed = offered − explicitly rejected); the skewed run shows the hot owner saturating while others idle (record the per-owner spread); queue depth is provably capped under overload; and you can state what partitioning cannot fix — one partition's ceiling is one core.

Sources

• Scylla: Thread-Per-Core Architecture — ScyllaDB engineering documentation — scylladb.com
• Seastar: An Advanced, Open-Source Framework For High-Performance Server Applications — seastar.io
• The Art of Multiprocessor Programming — Herlihy & Shavit, Morgan Kaufmann, 2020.
• sched_setaffinity(2) — Linux manual page — man7.org
• numactl / libnuma documentation — github.com/numactl/numactl
• java.util.concurrent.locks.ReentrantLock — Java SE documentation — docs.oracle.com
• The Rust Reference — `std::sync::Mutex` — doc.rust-lang.org/std/sync
• JMH: Java Microbenchmark Harness — openjdk.org/projects/code-tools/jmh
• Criterion.rs User Guide — bheisler.github.io/criterion.rs