Thread-per-Core Architecture

Learning objective

Explain thread-per-core as an ownership and execution model, contrast it against a shared worker pool, show how partitioned ownership eliminates lock contention but introduces cross-core handoff cost, demonstrate that partitioning alone does not fix load imbalance, explain the affinity/scheduler and NUMA caveats that determine whether the model's assumptions hold on real hardware, and show bounded-queue backpressure as the honest response to overload.

Prerequisites

• The SPSC Ring Buffer lab — cross-core handoff below is exactly one SPSC ring buffer per core-to-core link.
• The CAS Contention and Backoff lab — the shared worker pool's lock contention below is the same cost that lab describes, via a lock rather than a bare CAS loop.

Theory

Thread-per-core is an ownership model, not a thread count

"One thread per logical CPU" describes a number. Thread-per-core describes a discipline: each thread exclusively owns a partition of the application's state for its entire lifetime, and no other thread ever touches that partition directly. A program that spawns one thread per core but still lets any thread touch any shared data through a lock is a worker pool sized to the core count, not a thread-per-core architecture.

Shared worker pool vs. owned-state execution

Shared worker pool: N worker threads, all able to process any request, coordinating through shared state protected by a lock. Simple and flexible — but every request touching shared state pays the lock's cost, and only one can hold that lock at a time, regardless of how many workers exist.

Thread-per-core ownership: each core's thread owns a disjoint partition and only that thread ever touches it. No lock, because there is no sharing — the "single-writer alternative" from the CAS Contention lab, applied as the primary architecture. This buys real parallelism at the cost of needing an explicit mechanism for requests that arrive on the wrong core.

Cross-core handoff

A request for partition P arriving on a core that isn't P's owner must be handed off — never processed directly by the wrong core. The standard mechanism is one bounded SPSC channel per ordered pair of cores, using the same reservation → publication → read → acknowledgement discipline the SPSC Ring Buffer lab establishes. Handoff is not free: it costs at least one extra scheduling round through the channel.

Hot partitions: ownership does not equal balance

Partitioning by owner does not guarantee even traffic distribution. If one key is disproportionately popular, its owning core's inbox grows while every other core sits idle — the same total capacity exists but cannot spill onto underused cores the way it could in a shared pool. Repartitioning, splitting a hot key, or accepting bounded queueing/backpressure on the hot core are the standard responses.

Backpressure

Every core's inbox is bounded, exactly like the SPSC Ring Buffer lab's capacity. When full, reject (or have the sender block) rather than let the queue grow without bound — an unbounded queue under sustained overload only converts memory pressure into unbounded latency before failing anyway.

Affinity and scheduler caveats

Thread-per-core's zero-contention argument assumes each owning thread keeps running on (approximately) the same physical core. This is not automatic: CPU affinity support differs by OS — Linux exposes sched_setaffinity/taskset fairly directly; macOS does not expose hard affinity to user processes (THREAD_AFFINITY_POLICY is only a hint); Windows exposes SetThreadAffinityMask with its own quirks. Containers and cgroups add another layer — CPU quotas and cgroup CPU-sets interact with affinity, and orchestrators may migrate containers across hosts entirely outside the application's control. Even with affinity requested, the OS scheduler can still migrate a thread for priority, power, or thermal reasons. A migration does not break correctness — the thread still owns the same partition — it degrades performance by losing cache locality on the old physical core.

NUMA caveats

On multi-socket (NUMA) hardware, memory attached to one socket is more expensive for a core on a different socket to access. Thread-per-core's benefit compounds with NUMA-aware placement — pin a partition's thread to a core and allocate that partition's memory on the same NUMA node — but this lab's model does not simulate NUMA distance or memory placement. A production system needs an explicit memory allocation strategy (numactl, libnuma) in addition to CPU affinity.

Java and Rust

public final class PartitionedCounter {
    private long value;
    public long increment() { return ++value; }
    public long get() { return value; }
}

public final class SharedCounterPool {
    private final long[] counters;
    private final ReentrantLock lock = new ReentrantLock();

    public SharedCounterPool(int partitions) { counters = new long[partitions]; }

    public long increment(int partition) {
        lock.lock();
        try {
            return ++counters[partition];
        } finally {
            lock.unlock();
        }
    }
}

@State(Scope.Benchmark)
@BenchmarkMode(Mode.Throughput)
@OutputTimeUnit(TimeUnit.MILLISECONDS)
@Fork(1)
@Warmup(iterations = 5, time = 1)
@Measurement(iterations = 10, time = 1)
public class ThreadPerCoreBenchmark {
    private static final int PARTITIONS = 4;
    private final SharedCounterPool sharedPool = new SharedCounterPool(PARTITIONS);

    @Benchmark @Threads(4)
    public long sharedPoolIncrement() {
        int partition = ThreadLocalRandom.current().nextInt(PARTITIONS);
        return sharedPool.increment(partition);
    }

    @State(Scope.Thread)
    public static class OwnedState {
        final PartitionedCounter counter = new PartitionedCounter();
    }

    @Benchmark @Threads(4)
    public long ownedPartitionIncrement(OwnedState state) {
        return state.counter.increment();
    }
}

pub struct PartitionedCounter {
    value: u64,
}

impl PartitionedCounter {
    pub fn new() -> Self { Self { value: 0 } }
    pub fn increment(&mut self) -> u64 {
        self.value += 1;
        self.value
    }
    pub fn get(&self) -> u64 { self.value }
}

use std::sync::Mutex;

pub struct SharedCounterPool {
    counters: Mutex<Vec<u64>>,
}

impl SharedCounterPool {
    pub fn new(partitions: usize) -> Self {
        Self { counters: Mutex::new(vec![0; partitions]) }
    }

    pub fn increment(&self, partition: usize) -> u64 {
        let mut guard = self.counters.lock().unwrap();
        guard[partition] += 1;
        guard[partition]
    }
}

fn run_shared_pool() {
    let pool = Arc::new(SharedCounterPool::new(PARTITIONS));
    let handles: Vec<_> = (0..PARTITIONS).map(|partition| {
        let pool = Arc::clone(&pool);
        thread::spawn(move || { for _ in 0..INCREMENTS_PER_THREAD { pool.increment(partition); } })
    }).collect();
    for h in handles { h.join().unwrap(); }
}

fn run_owned_partitions() {
    let handles: Vec<_> = (0..PARTITIONS).map(|_| {
        thread::spawn(move || {
            let mut counter = PartitionedCounter::new();
            for _ in 0..INCREMENTS_PER_THREAD { counter.increment(); }
        })
    }).collect();
    for h in handles { h.join().unwrap(); }
}

Benchmark methodology

Method: 4 threads, two architectures, otherwise identical work. Shared pool: all 4 threads increment a randomly chosen partition (0–3) in one array guarded by a single lock. Owned partitions: each thread increments only its own, unshared counter.

Java (JMH, ops/ms, 4 threads, higher is better):

Rust (Criterion, 80,000 increments total per sample):

Owned partitions outperform the shared, lock-guarded pool by roughly 38× in Java and roughly 36× in Rust — consistent in magnitude across both languages, even though absolute baselines differ (see the SPSC Ring Buffer lab's benchmark notes on why absolute numbers don't transfer across language/JIT boundaries). This is the concrete cost of one shared lock serializing 4 threads' worth of work down to roughly one thread's worth of throughput. This benchmark isolates the lock/no-lock difference in the purest possible case — a trivial increment with no real request-processing, handoff, or queueing on either side — so treat it as evidence of the mechanism's raw cost, not a deployable system's expected end-to-end speedup.

Re-run the projects in code/ on your own hardware before treating any of this as authoritative for a design decision.

Common mistakes

• Calling a design "thread-per-core" because it spawns one thread per CPU, while still routing every request through shared, lock-guarded state — that's a worker pool sized to the core count, not ownership.
• Assuming partitioning alone balances load — a skewed key distribution still overloads one partition's owning core while others idle (see "Hot partitions").
• Processing a request directly on the "wrong" core to avoid handoff latency — this silently reintroduces the shared-mutable-state problem ownership was meant to eliminate.
• Assuming CPU affinity is a hard guarantee on every platform — macOS in particular treats it only as a hint the scheduler may ignore.
• Letting a core's inbox grow unboundedly under overload instead of rejecting — this converts memory pressure into unbounded latency rather than failing fast.

Investigation task

Using the Java or Rust project in code/:

1. Run the benchmark and record the shared-pool vs. owned-partition throughput ratio on your own hardware.
2. Vary the thread/partition count (2, 4, 8) and re-run — observe how the shared pool's throughput changes (or doesn't) as thread count grows, versus the owned-partition case.
3. If your OS supports it, pin the benchmark process to specific cores (e.g. taskset on Linux) and re-run — note whether this changes the owned-partition result, and by how much.
4. In the interactive model, work through "Hot partition" and record the step at which core 1 first rejects a request — then work through "Backpressure" and compare how quickly rejection occurs with a smaller burst against a single core.
5. Write down your CPU model and core count, your measured numbers, and explain any difference from the disclosed numbers above.

Accessibility notes

• The interactive model is fully keyboard operable: the scenario selector and the Java/Rust code tabs are roving-tabindex tablists (arrow keys, Home/End); every step/reset control is a native <button>.
• Every state change is announced to screen readers via an aria-live="polite" region, in addition to the visible per-core status badges and the state inspector table — state is never colour-only.
• This page has no auto-playing or looping animation; step transitions resolve instantly, so there is nothing beyond the site-wide CSS rule to gate on prefers-reduced-motion.
• With JavaScript disabled, this page's theory, Java and Rust code, benchmark methodology, trade-offs and sources remain fully readable — only the interactive model and the graded quiz require script.

Review questions

Self-check before or after the graded quiz below:

1. Why does a shared worker pool's throughput not scale with worker count, even though many workers are available?
2. What specific cost does cross-core handoff add that direct, same-core dispatch does not?
3. Why doesn't partitioning state by owner automatically balance load across cores?
4. Give one reason CPU affinity requested by an application might not actually be honored by the OS.

Sources

• Scylla: Thread-Per-Core Architecture — ScyllaDB engineering documentation — scylladb.com
• Seastar: An Advanced, Open-Source Framework For High-Performance Server Applications — seastar.io
• The Art of Multiprocessor Programming — Herlihy & Shavit, Morgan Kaufmann, 2020.
• sched_setaffinity(2) — Linux manual page — man7.org
• numactl / libnuma documentation — github.com/numactl/numactl
• java.util.concurrent.locks.ReentrantLock — Java SE documentation — docs.oracle.com
• The Rust Reference — `std::sync::Mutex` — doc.rust-lang.org/std/sync
• JMH: Java Microbenchmark Harness — openjdk.org/projects/code-tools/jmh
• Criterion.rs User Guide — bheisler.github.io/criterion.rs