8.5 MB vs. 571 MB: LTSeq's Performance Engineering Playbook

Summary: LTSeq gets 1.4x to 1.5x throughput gains over DuckDB on ordered-sequence workloads while using 67x to 112x less memory. This article walks every layer of the optimization stack, from Cargo release profiles and Tokio runtime design through fused evaluation loops to Rayon-based parallel Parquet scanning, and validates each with ClickBench numbers on 100 million rows.

---

1. Methodology: ClickBench on 100M rows

Any analysis of optimizations needs a credible benchmark first. LTSeq uses the ClickBench dataset, 100 million web analytics events from Yandex, across three rounds that test very different workload shapes.

1.1 The three rounds

Round	Operation	What It Tests
R1	Top-10 URLs by hit count	Aggregation + sort (DuckDB's strength)
R2	30-minute gap sessionization	Sequential boundary detection (LTSeq's strength)
R3	3-step URL funnel	Pattern matching across ordered rows

R1 is a deliberate control: pure aggregation has no sequential semantics, so DuckDB's hash-aggregate should dominate. R2 and R3 are the real contest, since both require row-order awareness.

1.2 The benchmark harness

# benchmarks/bench_vs.py (simplified)
import tracemalloc, time

def measure(fn, label):
    tracemalloc.start()
    t0 = time.perf_counter()
    result = fn()
    elapsed = time.perf_counter() - t0
    _, peak_mem = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    return {"label": label, "time": elapsed, "peak_memory_mb": peak_mem / 1e6}

Key measurement decisions:

• tracemalloc tracks Python-visible memory. Rust allocations through jemalloc are invisible, so LTSeq's real memory usage is lower than reported.
• 5 warm-up runs, 3 measured runs, median reported. Parquet metadata is cached by the OS after the first run.
• The machine is 32 cores and 125 GB RAM. Both engines have access to all of it.

1.3 Results

Round	DuckDB	LTSeq	Throughput	LTSeq Peak Memory	DuckDB Peak Memory	Memory Ratio
R1: Top URLs	1.039s	1.754s	0.6x (DuckDB wins)	532 MB	~800 MB	~0.7x
R2: Sessionization	1.366s	0.971s	1.4x LTSeq	8.5 MB	571 MB	67x less
R3: Funnel	3.534s	2.363s	1.5x LTSeq	10.4 MB	1161 MB	112x less

R1: DuckDB's hash-aggregate engine is highly optimized for GROUP BY workloads. LTSeq delegates aggregation to DataFusion, which is competent but not specialized. The 0.6x result is expected, since LTSeq makes no claim to be better at set-oriented operations.

R2: 1.4x faster with 67x less memory. The memory number is the headline. 8.5 MB for 100 million rows means LTSeq is streaming through the data without materializing it.

R3: 1.5x faster with 112x less memory. The funnel's sparse evaluation pattern, where step 1 gates step 2 and beyond, means most rows are never fully evaluated.

---

2. Layer 0: compiler-level optimization

Performance engineering starts before any Rust code runs.

2.1 The release profile

# Cargo.toml
[profile.release]
opt-level = 3       # Maximum optimization (loop unrolling, vectorization)
lto = "fat"          # Cross-crate Link-Time Optimization
codegen-units = 1    # Single codegen unit for maximum inlining
strip = true         # Strip debug symbols

Each setting has a specific purpose:

• lto = "fat" enables whole-program optimization across all crate boundaries. That matters because LTSeq calls deep into DataFusion and Arrow, and without LTO those cross-crate function calls cannot be inlined. Fat LTO raises compile time from ~30s to ~120s but removes call overhead on hot paths.

• codegen-units = 1 forces LLVM to see the entire crate as a single compilation unit. Combined with fat LTO, this gives the optimizer maximum visibility for inlining decisions. The tradeoff is single-threaded compilation.

• opt-level = 3 enables aggressive loop optimizations including auto-vectorization. For tight loops over Arrow arrays (the core of fused evaluation), this can generate SIMD instructions automatically.

2.2 What this means in practice

Consider the read_value() method on TypedColumn:

#[inline(always)]
fn read_value(&self, row: usize) -> Value {
    match self {
        TypedColumn::Int64(arr) => {
            if arr.is_null(row) { Value::Null }
            else { Value::Int64(arr.value(row)) }
        }
        // ...
    }
}

The #[inline(always)] hint, combined with lto = "fat" and codegen-units = 1, means this function is inlined at every call site. The match arms for unused types are eliminated by dead code analysis. The result: a single array index operation with a null check, no function call overhead.

---

3. Layer 1: runtime architecture

3.1 Dual SessionContext design

LTSeq maintains two DataFusion SessionContext instances, each configured for a different workload pattern (src/engine.rs):

// Parallel context: for aggregation, joins, set operations
pub fn create_session_ctx() -> SessionContext {
    let config = SessionConfig::new()
        .with_target_partitions(num_cpus::get())    // Use all cores
        .with_repartition_joins(true)                // Parallel hash joins
        .with_repartition_aggregations(true)         // Parallel aggregation
        .set_bool("datafusion.execution.parquet.pushdown_filters", true)
        .set_bool("datafusion.execution.parquet.reorder_filters", true);
    SessionContext::new_with_config(config)
}

// Sequential context: for ordered operations
pub fn create_sequential_session_ctx() -> SessionContext {
    let config = SessionConfig::new()
        .with_target_partitions(1)                   // Single partition
        .set_bool("datafusion.execution.parquet.pushdown_filters", true)
        .set_bool("datafusion.execution.parquet.reorder_filters", true);
    SessionContext::new_with_config(config)
}

Why two contexts? DataFusion's target_partitions > 1 triggers SortPreservingMergeExec on sorted data: it splits the input into N partitions, processes them in parallel, then merges the sorted results. For ordered-sequence operations like group_ordered, the split-merge overhead cancels out any parallelism gains. The sequential context avoids it entirely.

Parquet pushdown filters are enabled on both contexts. When a filter references only Parquet-native columns, DataFusion pushes the predicate down to the Parquet reader, which uses row group statistics (min/max) to skip entire row groups without reading them.

3.2 Global Tokio runtime

static TOKIO_RT: LazyLock<Runtime> = LazyLock::new(|| {
    Runtime::new().expect("Failed to create Tokio runtime")
});

pub fn block_on<F: Future>(future: F) -> F::Output {
    TOKIO_RT.block_on(future)
}

DataFusion operations are async. LTSeq wraps them with block_on() to present a synchronous API to Python. A single global runtime is shared across all operations, avoiding the cost of creating and destroying a runtime per query.

3.3 Batch size tuning

// Default DataFusion batch size
const DEFAULT_BATCH_SIZE: usize = 16_384;   // 16K rows per batch

// Parallel scan uses larger batches
const PARALLEL_BATCH_SIZE: usize = 65_536;  // 64K rows per batch

The parallel scan uses 4x larger batches to amortize per-batch overhead (column type resolution, state management) across more rows. At 64K rows per batch, the per-batch setup cost becomes negligible.

---

4. Layer 2: algorithmic fusion

This is where the largest performance gains come from.

4.1 The three-pass problem

Standard sessionization through DataFusion requires three full-table passes:

Pass 1: LAG(userid) OVER (ORDER BY eventtime)     → 100M rows read + written
Pass 2: CASE WHEN userid IS DISTINCT FROM lag_uid  → 100M rows read + written
         OR eventtime - lag_time > 1800
Pass 3: SUM(boundary) OVER (ORDER BY eventtime)   → 100M rows read + written

Each pass allocates new Arrow arrays, copies data, and deallocates the old arrays. For 100M rows with two columns (userid: Utf8, eventtime: Timestamp), each pass moves roughly 1.6 GB of data through memory.

4.2 Fused single-pass evaluation

LTSeq's linear scan engine merges all three passes into a single loop (src/ops/linear_scan.rs). The fuse_eval function pattern-matches common predicate shapes:

Pattern 1, Column != Column.shift(1):

// Detects: r.userid != r.userid.shift(1)
fn fuse_ne_shift(col_name: &str, batch: &RecordBatch, state: &mut StreamState) -> Vec<bool> {
    let col = batch.column_by_name(col_name).unwrap();
    let typed = TypedColumn::from_array(col);
    let mut boundaries = Vec::with_capacity(batch.num_rows());

    for i in 0..batch.num_rows() {
        let curr = typed.read_value(i);
        let prev = if i == 0 {
            state.prev_values.get(col_name).cloned().unwrap_or(Value::Null)
        } else {
            typed.read_value(i - 1)
        };
        boundaries.push(curr != prev);
    }

    // Save last row for next batch
    state.prev_values.insert(col_name.to_string(), typed.read_value(batch.num_rows() - 1));
    boundaries
}

Pattern 2, (Column - Column.shift(1)) > Literal, for time gap detection:

// Detects: r.eventtime - r.eventtime.shift(1) > 1800
// Uses direct i64 subtraction, no Arrow kernel overhead
fn fuse_sub_gt_shift(col: &str, threshold: i64, batch: &RecordBatch, state: &mut StreamState) -> Vec<bool> {
    let arr = batch.column_by_name(col).unwrap();
    let i64_arr = arr.as_any().downcast_ref::<Int64Array>().unwrap();
    let mut boundaries = Vec::with_capacity(batch.num_rows());

    for i in 0..batch.num_rows() {
        let curr = i64_arr.value(i);
        let prev = if i == 0 {
            state.get_i64(col).unwrap_or(i64::MIN)
        } else {
            i64_arr.value(i - 1)
        };
        boundaries.push(curr.wrapping_sub(prev) > threshold);
    }

    state.set_i64(col, i64_arr.value(batch.num_rows() - 1));
    boundaries
}

Pattern 3, compound OR, combining both:

// Detects: (r.userid != r.userid.shift(1)) | (r.eventtime - r.eventtime.shift(1) > 1800)
for i in 0..num_rows {
    let uid_changed = uid_vals[i] != uid_prev;
    let gap_exceeded = time_vals[i].wrapping_sub(time_prev) > 1800;
    boundaries[i] = uid_changed || gap_exceeded;

    uid_prev = uid_vals[i];
    time_prev = time_vals[i];
}

What that eliminates:

• Zero intermediate arrays (no shift result array, no difference array, no comparison array)
• Zero Arrow kernel dispatch overhead
• Zero cross-batch data copying

4.3 Fallback: compiled bytecode evaluation

When the expression doesn't match a known fused pattern, the linear scan engine compiles it to stack-based bytecode:

enum Instruction {
    PushColumn(usize),           // Push column[row] onto stack
    PushShiftedColumn(usize),    // Push column[row-1] onto stack
    PushLiteral(Value),          // Push constant
    BinOp(BinOpKind),            // Pop 2 values, push result
    Not,                         // Pop 1 value, push !value
    IsNull,                      // Pop 1 value, push is_null
}

struct CompiledEvaluator {
    instructions: Vec<Instruction>,
    column_indices: Vec<usize>,  // Pre-resolved column name → index
}

Column name resolution happens once at compile time rather than per row. The instruction set is deliberately minimal: no branches, no function calls, just stack operations. That keeps the evaluation loop tight enough for the CPU's branch predictor to work well.

4.4 Quantifying the gain

For the sessionization query on 100M rows:

Approach	Passes	Intermediate Arrays	Time
DataFusion standard	3	6 (shift, diff, cmp × 2 columns)	~2.5s
LTSeq fused	1	0	0.97s

The gain comes from three sources:

1. Reduced memory bandwidth. 1 pass instead of 3 means 3x less data movement.
2. No allocation or deallocation. Zero intermediate arrays saves ~200ms of allocator overhead.
3. Better cache utilization. The single-pass access pattern is L1-cache-friendly, while multi-pass re-reads data that has already been evicted.

---

5. Layer 3: direct Parquet engine

5.1 Bypassing DataFusion entirely

For group_ordered on Parquet files, LTSeq has a fast path that bypasses DataFusion completely (src/ops/parallel_scan.rs):

Standard path:
  Parquet → DataFusion PhysicalPlan → RecordBatchStream → group_ordered

Direct path:
  Parquet → arrow-rs ParquetRecordBatchReader → streaming_fuse_eval → group_id array

The direct path eliminates:

• Logical plan construction
• Physical plan optimization
• SortPreservingMergeExec (even in sequential mode, there's overhead)
• Schema validation and type coercion

5.2 Column projection

The direct engine only reads columns referenced in the predicate:

// Extract column names from the predicate expression
let needed_cols = extract_referenced_columns(predicate);

// Build Parquet projection mask
let proj_indices: Vec<usize> = needed_cols.iter()
    .filter_map(|name| parquet_schema.columns().iter()
        .position(|c| c.name() == name))
    .collect();
let projection = ProjectionMask::roots(schema_descr, proj_indices);

For the sessionization query (r.userid != r.userid.shift(1)) | (r.eventtime - r.eventtime.shift(1) > 1800), only 2 of the dataset's 105 columns are read. This reduces I/O from ~14 GB to ~0.8 GB.

5.3 Immediate memory release

A critical detail in the parallel pattern matching path:

// Each row group is processed and immediately dropped
row_group_indices.into_par_iter().map(|rg_idx| {
    let file = File::open(path)?;
    let reader = ParquetRecordBatchReaderBuilder::try_new(file)?
        .with_row_groups(vec![rg_idx])
        .with_projection(mask.clone())
        .build()?;

    let batch = reader.into_iter().next().unwrap()?;
    let count = eval_pattern_in_batch(&batch, predicates);

    // `batch` is dropped here, memory is freed immediately
    count
}).collect()

Without this pattern, holding all 814 row groups in memory at once would need ~1.2 GB. With immediate release, peak memory stays at roughly batch_size × num_threads × column_size, about 10 MB for the funnel query.

The benchmark data confirms this: R3 funnel uses 10.4 MB peak memory vs. DuckDB's 1161 MB.

5.4 Rayon work-stealing

The parallel engine uses Rayon's par_iter for row-group-level parallelism:

let counts: Vec<usize> = row_group_indices
    .into_par_iter()
    .map(|rg_idx| process_row_group(rg_idx, path, predicates))
    .collect();

Rayon's work-stealing scheduler handles load imbalance automatically. If some row groups are larger or have more matches, idle threads steal work from busy threads. No manual thread pool management is needed.

Each task opens its own file handle to avoid lock contention:

// Inside the parallel closure
let file = File::open(parquet_path)?;  // Independent file descriptor per task

On Linux this rides on the kernel's page cache: the same physical pages are shared across file descriptors, but there's no mutex on the file handle.

---

6. Layer 4: type system optimizations

6.1 TypedColumn: amortized type dispatch

Arrow arrays are type-erased (ArrayRef = Arc<dyn Array>). Every value access requires a downcast_ref call:

// Standard Arrow access, downcast per row (bad)
for row in 0..batch.num_rows() {
    let val = batch.column(0)
        .as_any()
        .downcast_ref::<Int64Array>()  // Runtime type check
        .unwrap()
        .value(row);
}

TypedColumn resolves the type once per batch:

// TypedColumn, downcast once per batch (good)
let typed = TypedColumn::from_array(batch.column(0));  // One downcast
for row in 0..batch.num_rows() {
    let val = typed.read_value(row);  // Direct index via enum match
}

For 100M rows at 16K rows per batch ≈ 6,100 batches. With 2 columns, that's 12,200 downcasts instead of 200,000,000. A 16,000x reduction in type dispatch overhead.

6.2 Timestamp zero-copy

Timestamps in Arrow are stored as i64 values (seconds, milliseconds, or microseconds since epoch). The linear scan engine reads them directly as Int64:

DataType::Timestamp(_, _) => {
    if let Some(arr) = array.as_any().downcast_ref::<TimestampSecondArray>() {
        Value::Int64(arr.value(row))  // Zero-copy: read i64 directly
    } else if let Some(arr) = array.as_any().downcast_ref::<TimestampMicrosecondArray>() {
        Value::Int64(arr.value(row))  // Same, the underlying bits are i64
    }
}

This means eventtime - eventtime.shift(1) > 1800 is computed as plain integer subtraction, not as a timestamp duration comparison. No chrono library, no timezone handling, no format parsing. Just i64.wrapping_sub(i64) > 1800.

6.3 Cross-type numeric promotion

When comparing Int64 with Float64 (e.g., column value vs. literal), the engine promotes to f64 automatically:

impl PartialOrd for Value {
    fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
        match (self, other) {
            (Value::Int64(a), Value::Int64(b)) => a.partial_cmp(b),
            (Value::Int64(a), Value::Float64(b)) => (*a as f64).partial_cmp(b),
            (Value::Float64(a), Value::Int64(b)) => a.partial_cmp(&(*b as f64)),
            (Value::Float64(a), Value::Float64(b)) => a.partial_cmp(b),
            _ => None,
        }
    }
}

This avoids type coercion at the column level, so there's no need to cast an entire Int64 column to Float64 before comparison.

---

7. Layer 5: Arrow shift bypass

7.1 The problem with window functions

DataFusion implements shift(1) as a LAG window function. This requires:

1. Sorting (verifying sort order)
2. Window frame computation
3. Full output array allocation

For a simple "previous row" operation, this is massive overkill.

7.2 Direct array slicing

LTSeq's arrow_shift.rs implements shift as direct array manipulation:

// shift(1): prepend one NULL, take all rows except the last
pub fn arrow_shift_down(array: &ArrayRef, n: usize) -> ArrayRef {
    let len = array.len();
    if n >= len {
        return new_null_array(array.data_type(), len);
    }
    // Slice [0..len-n] from the original array
    let shifted = array.slice(0, len - n);
    // Prepend n NULLs
    let nulls = new_null_array(array.data_type(), n);
    concat(&[&nulls, &shifted]).unwrap()
}

This uses Arrow's zero-copy slice(), which adjusts the offset and length pointers without copying data. The only allocation is for the NULL prefix.

7.3 Partition-aware shift with boundary caching

When shift is used with partition_by, boundaries between partitions must be respected (values don't leak across partitions). LTSeq detects partition boundaries using Arrow's vectorized neq kernel:

// Detect partition boundaries using SIMD-accelerated comparison
let boundaries = arrow::compute::neq(
    &partition_col.slice(0, len - 1),
    &partition_col.slice(1, len - 1)
)?;

This produces a BooleanArray marking where partition transitions occur. The boundary array is cached, so when multiple shift operations use the same partition_by key, boundary detection runs only once.

---

8. Layer 6: streaming architecture

8.1 The cursor

For datasets too large to fit in memory, LTSeq provides a streaming cursor (src/cursor.rs):

cursor = LTSeq.scan("events.csv")
for batch in cursor:
    process(batch)  # Each batch is a small LTSeq table

Under the hood, the cursor wraps DataFusion's SendableRecordBatchStream and serializes each RecordBatch to Arrow IPC format for transfer across the Rust→Python boundary:

pub fn next_batch(&mut self) -> PyResult<Option<PyObject>> {
    let batch = self.rt.block_on(async { self.stream.next().await });
    match batch {
        Some(Ok(batch)) => {
            // Serialize to IPC bytes
            let mut buf = Vec::new();
            let mut writer = StreamWriter::try_new(&mut buf, &self.schema)?;
            writer.write(&batch)?;
            writer.finish()?;
            // Return bytes to Python
            Ok(Some(PyBytes::new(py, &buf).into()))
        }
        _ => Ok(None),
    }
}

This means memory usage is bounded by batch_size × row_size, regardless of total dataset size.

8.2 Direct stream counting

For operations like group_ordered().first().count(), LTSeq has a direct_streaming_group_count fast path that never materializes the grouped table:

pub fn direct_streaming_group_count(
    table: &LTSeqTable, predicate: &PyExpr, parquet_path: &str,
) -> PyResult<usize> {
    // Stream through batches, counting boundaries
    let mut state = StreamState::new();
    let mut group_count: usize = 1;  // First row is always group 1

    for batch in reader {
        let boundaries = streaming_fuse_eval(predicate, &batch, &mut state);
        group_count += boundaries.iter().filter(|&&b| b).count();
    }

    Ok(group_count)
}

No group_id array, no metadata table, no first-row extraction. Just count the boundaries. This turns a multi-GB operation into a single integer.

---

9. Layer 7: expression optimization

Before expressions reach the evaluation engine, they pass through an optimization pass (src/transpiler/optimization.rs):

9.1 Constant folding

// Before: 3 + 4 > 5
// After:  7 > 5
// After:  true
fn fold_constants(expr: &PyExpr) -> PyExpr {
    match expr {
        PyExpr::BinOp { op, left, right } => {
            let left = fold_constants(left);
            let right = fold_constants(right);
            if let (PyExpr::Literal { value: l }, PyExpr::Literal { value: r }) = (&left, &right) {
                // Evaluate at compile time
                evaluate_constant(op, l, r)
            } else {
                PyExpr::BinOp { op, left, right }
            }
        }
        // ...
    }
}

9.2 Boolean simplification

// x AND true  → x
// x AND false → false
// x OR true   → true
// x OR false  → x
// NOT NOT x   → x

These optimizations are simple but eliminate unnecessary runtime work, especially in generated expressions from the lambda capture system.

---

10. Putting it all together: the sessionization query

The R2 benchmark query, step by step:

t = LTSeq.read_parquet("hits.parquet")
sessions = t.sort("UserID", "EventTime").group_ordered(
    lambda r: (r.UserID != r.UserID.shift(1)) |
              (r.EventTime - r.EventTime.shift(1) > 1800)
)
result = sessions.first()

Execution path:

1. sort("UserID", "EventTime")
   → Assumes data is pre-sorted (Parquet was generated sorted)
   → Sets _sort_keys, no actual sort

2. group_ordered(lambda)
   → Python: SchemaProxy captures expression tree
   → Python: Serializes to dict (BinOp[Or, BinOp[Ne, Col, Shift], BinOp[Gt, BinOp[Sub, Col, Shift], Lit]])
   → Rust:   dict_to_py_expr() deserializes
   → Rust:   can_linear_scan() → true (contains shift(1), all ops supported)
   → Rust:   source_parquet_path is set → use direct_streaming_group_ordered
   → Rust:   extract_referenced_columns → ["UserID", "EventTime"]
   → Rust:   Open Parquet with projection mask (2 of 105 columns)
   → Rust:   streaming_fuse_eval detects compound Or pattern
   → Rust:   Fused evaluation: single loop over rows
              - TypedColumn resolves types once per batch
              - Timestamps read as i64
              - wrapping_sub for time difference
              - No intermediate arrays
   → Rust:   StreamState carries last row across 6,100 batches
   → Rust:   Build group_id, group_count, row_number arrays
   → Rust:   Return metadata table

3. first()
   → Filter: row_number == 1 for each group
   → Returns first row of each session

Memory profile:

Component	Size
Parquet reader buffer (2 columns × 64K rows)	~1 MB
Boundary flags per batch	~64 KB
StreamState (2 prev values)	~64 bytes
group_id / group_count / rn arrays	~2.4 MB each
Total	~8.5 MB

Compare that to DuckDB's 571 MB, which materializes intermediate arrays for LAG, CASE WHEN, and SUM OVER.

---

11. Where LTSeq loses, and why that's fine

11.1 R1: pure aggregation

DuckDB's hash-aggregate engine is a serious piece of engineering: vectorized, SIMD-optimized, with perfect hash tables and automatic parallelism. LTSeq delegates aggregation to DataFusion, which is good but not at DuckDB's level for this workload.

Aggregation is a set operation, not a sequence operation, so it doesn't benefit from ordered-sequence optimizations. LTSeq doesn't try to beat DuckDB here.

11.2 Cold start

LTSeq's first query incurs PyO3 module loading and Tokio runtime initialization overhead (~50ms). For interactive one-off queries, this is noticeable. For batch analytics or repeated queries, it's amortized to nothing.

11.3 Single-node ceiling

LTSeq is a single-node library. Datasets that exceed one machine's storage need a distributed engine such as Spark or Trino. LTSeq's sweet spot is the "too big for pandas, too small for Spark" range, roughly 10M to 1B rows on a single machine.

---

12. Summary: the optimization stack

┌─────────────────────────────────────────────────────────────┐
│  Layer 7: Expression Optimization                           │
│  Constant folding, boolean simplification                   │
├─────────────────────────────────────────────────────────────┤
│  Layer 6: Streaming Architecture                            │
│  Cursor, direct stream counting, bounded memory             │
├─────────────────────────────────────────────────────────────┤
│  Layer 5: Arrow Shift Bypass                                │
│  Zero-copy slice, SIMD boundary detection, boundary cache   │
├─────────────────────────────────────────────────────────────┤
│  Layer 4: Type System Optimizations                         │
│  TypedColumn amortized dispatch, timestamp zero-copy        │
├─────────────────────────────────────────────────────────────┤
│  Layer 3: Direct Parquet Engine                             │
│  Bypass DataFusion, column projection, Rayon parallel       │
├─────────────────────────────────────────────────────────────┤
│  Layer 2: Algorithmic Fusion                                │
│  Fused evaluation, zero intermediate arrays, bytecode       │
├─────────────────────────────────────────────────────────────┤
│  Layer 1: Runtime Architecture                              │
│  Dual SessionContext, global Tokio, batch size tuning       │
├─────────────────────────────────────────────────────────────┤
│  Layer 0: Compiler-Level Optimization                       │
│  Fat LTO, codegen-units=1, opt-level=3, strip               │
└─────────────────────────────────────────────────────────────┘

No single optimization explains the 67x memory reduction or the 1.5x throughput gain. It's the composition of all seven layers, from compiler flags up to algorithmic fusion, that produces the result. Each layer builds on the one below it.

The biggest lesson: for domain-specific workloads, a well-optimized specialized engine beats a general-purpose engine not by doing things faster, but by doing less work. LTSeq doesn't read 100M rows faster than DuckDB. It reads 2 columns instead of 105 and makes 1 pass instead of 3. The fastest code is code that never runs.

Next article: all of this in practice, across stock analysis, user session modeling, and multi-table analytics, plus the pitfalls that come with them.