Phase 6.6: Adaptive Speculative Decoding, Draft-Free Prediction, and Cost-Aware Inference

Date: October 21, 2026
Baseline Commit: afcdd1d (Phase 6.5 complete)
Target Models:

• Ornith 1.5 35B (qwen35moe): /mnt/models-ssd/Strata-data/gpacks/ornith-1.5-35b-ext.gpack (256 routed experts/layer, top-8, 40 layers, no MTP weights)
• Qwen3.8-Flash-Next (qwen4_exp): /mnt/models-ssd/Strata-data/gpacks/qwen3.8-flash-next-ext.gpack (512 routed experts/layer, top-10, 48 layers, GPU MTP draft layer in /mnt/models-ssd/Strata-data/mtp/rt)

---

1. Executive Summary

Phase 6.6 designs, implements, validates, and benchmarks an adaptive, cost-aware speculative decoding system across Guild’s hybrid GPU+CPU Mixture-of-Experts (MoE) inference pipeline. Rather than maximizing draft acceptance rate or draft depth in isolation, the runtime scheduler explicitly minimizes expected wall-clock milliseconds per committed target-model output token:

$C(s, k) = \frac{\mathbb{E}[\text{iteration wall-clock time (ms)}]}{\mathbb{E}[\text{committed output tokens}]}$

where iteration wall-clock time includes drafting/lookup/retrieval latency, target-model MoE verification across window size $T = 1 + k$, GPU-CPU synchronization, and KV/GDN/QSA commit or rollback, and committed output tokens count actual target-model tokens emitted ($\text{accepted draft prefix} + 1$ target/bonus token).

Headline Measured Outcomes (2× Xeon Gold 6130 + GTX 1070 8 GiB, GEXEC Active)

1. Ornith 1.5 35B (ornith:35b, no MTP weights — previously limited to T=1 single-token decode):

   • Unlocked multi-token speculative verification (T=1..5) for draftless Prompt-Lookup (lookup), Session-Local Retrieval (retrieval), and Cost-Aware Adaptive (adaptive) modes without requiring any draft model or extra VRAM weights.
   • Across input-grounded, structured, RAG, multi-turn, and long-context workloads (W2–W6, W10), adaptive delivers 1.395x to 1.608x end-to-end decode speedups over normal decoding (50.0 $\rightarrow$ 80.65 tok/s on W2 summarization, 77.75 tok/s on W4 RAG, 69.79 tok/s on W5 multi-turn, and 80.03 tok/s at p=512 / 55.31 tok/s at p=32,704 populated context).
   • On partially repetitive code editing (W1) and multi-turn conversation (W5), adaptive beats both normal AND fixed lookup/retrieval (52.98 tok/s vs 50.29 normal and 50.91 lookup on W1; 69.79 tok/s vs 50.02 normal, 68.27 lookup, and 68.38 retrieval on W5) by dynamically modulating draft depth and backing off on low-confidence spans.
   • On speculation-hostile creative prose (W7) and mathematical reasoning (W8), fixed lookup regresses to 48.84 tok/s (-1.4%) and 47.91 tok/s (-4.5%), whereas adaptive automatically falls back to normal decoding (49.53 tok/s on W7, 50.01 tok/s on W8 — within 0.04%–0.28% of normal).
   • Under 2-session concurrent execution (--sessions 2), adaptive scales median decode throughput from 49.75 tok/s (normal) to 83.68 tok/s (1.682x), cutting median round wall time from 7,058.17 ms to 5,993.70 ms (-15.1%) with 100% deterministic session isolation.

2. Qwen3.8-Flash-Next (qwen3.8-flash-next, hybrid arbitration across normal, mtp, lookup, retrieval):

   • Arbitrates dynamically between GPU MTP drafting (k=1, T=2), CPU Prompt-Lookup (k=1..4, T=2..5), Session-Local Retrieval (k=1..4), and Normal decoding (k=0, T=1).
   • On input-grounded/repetitive workloads (W2 summarization, W4 RAG, W6 boilerplate, W10 32K–65K long context), adaptive shifts from mtp (23.3–25.7 tok/s) to deep lookup (k=4), reaching 32.02–33.70 tok/s (1.746x–1.843x over normal’s 18.3 tok/s, and +31.0%–+38.3% faster than mtp alone).
   • On workloads where prompt-lookup fails or regresses (W1 code editing where lookup drops to 17.85 tok/s; W7 creative prose where lookup drops to 17.88 tok/s; W8 math reasoning where lookup drops by -13.8% to 15.91 tok/s), adaptive selects mtp and achieves 25.39 tok/s (W1), 22.15 tok/s (W7), and 24.31 tok/s (W8, +52.8% faster than fixed lookup and +31.7% faster than normal).
   • On W5 multi-turn conversation and W10 16,384-token populated context, adaptive beats every fixed strategy (normal, mtp, lookup, retrieval) by combining MTP on novel transitions with Lookup/Retrieval on repeated spans (23.38 tok/s on W5 vs 18.37 normal, 20.58 lookup, 19.97 retrieval, 22.87 mtp; 24.50 tok/s at p=16,384 vs 17.92 normal, 20.44 lookup, 23.94 mtp).
   • At 65,472 populated context tokens (65K), adaptive achieves 31.78 tok/s (31.47 ms/tok) vs 17.49 tok/s (normal) and 24.20 tok/s (mtp) — a 1.817x decode speedup at 65K context.

3. 100% Exact Greedy & Target-Distribution Sampling Parity:

   • Fixed a subtle #152 reduction-order gate in src/kernels/cpu/gexec_expert.cpp (gexec_iq4nl_down_rows, iq256_row_dot_1row, gexec_iq4xs_gu_rows_nt) so GEXEC’s AVX-512/AVX2 CPU expert providers honor GUILD_IQ_MT_MIN=1 for nt == 1, achieving 100% bit-for-bit token ID parity across normal, mtp, lookup, retrieval, and adaptive on every workload (W1–W10) on both Ornith 35B and Qwen3.8-Flash-Next.
   • Verified 100% bitwise seed-invariant sampling parity under non-greedy sampling (temperature=0.7, top_p=0.9, min_p=0.05, top_k=40, seed=12345) on real model execution and 25,000-token Monte Carlo distributional tests (TVD < 0.01, zero mass outside top_k/top_p/min_p support).

---

2. External Research & Systems Survey (Through October 2026)

---

3. Phase 6.6 Architecture & Implementation

3.1 Upgraded Prompt-Lookup Drafter (include/guild/spec/suffix_drafter.hpp, src/spec/suffix_drafter.cpp)

• Multi-Length Anchors & Bounded Probing: Indexes 3-gram anchors (plus 2-gram anchors with distinct hash tag bits when min_match == 2) in a power-of-two open-addressing table with WAYS = 4 positions per slot, bounded linear probing (kMaxProbe = 16), and oldest-position LRU slot eviction on probe saturation.
• Frequency- & Recency-Aware Continuation Ranking: Extends candidate matches backwards up to max_match (64 tokens) and ranks equal-length matches by continuation token agreement frequency (last_freq()) first, breaking ties by recency.
• Sliding-Window Compaction: Enforces set_max_history(spec_lookup_window) via automatic half-window compaction and re-indexing so memory stays strictly bounded (~16.8 MiB for a 65,536-token window) across arbitrarily long sessions.

3.2 Session-Local Retrieval Drafter (include/guild/spec/retrieval_drafter.hpp, src/spec/retrieval_drafter.cpp)

• Two-Tier Session-Local Datastore:
  1. corpus_: session-local persistent token store across turns or explicitly ingested reference documents (GuildSession::ingest_retrieval_tokens, clear_retrieval_corpus), bounded by max_corpus_tokens.
  2. stream_: active request prompt + committed output tokens, reset at the start of each generation (reset_stream()).
• REST-Style Length-Weighted Continuation Voting: Gathers up to kMaxCandidates = 16 deduplicated matching spans across 4-gram and 3-gram anchors from both stream_ and corpus_, weights each span $c$ by: $w_c = (\text{match\_len}_c)^2 \cdot \text{source\_boost}_c \cdot \frac{1}{1 + 0.08\,\text{rank}_c}$ and votes at each continuation step $j \in [0, k-1]$, pruning diverging spans and stopping early if continuation confidence $w_{\text{best}} / \sum w_c < \text{min\_confidence}$.
• Strict Session Isolation: Owned exclusively by GuildSession::Impl under spec_mutex; never shared across sessions or global state.

3.3 Cost-Aware Adaptive Decoding Scheduler (include/guild/spec/adaptive_scheduler.hpp, src/spec/adaptive_scheduler.cpp)

• Objective Function: At every safe verification boundary, evaluates candidate pairs $(s, k)$ across Normal(0), Mtp(1..mtp_cap_k), Lookup(1..max_draft), and Retrieval(1..max_draft) using: $C_{\text{model}}(s, k) = \frac{\hat{D}(s, k) + \hat{V}(1 + k) + \mathbb{I}[\text{MTP catch-up}]\,\hat{M}_{\text{catchup}}}{\mathbb{E}[A_{s,k} + 1]}$ blended with the strategy’s decayed empirical cumulative wall-clock time-to-output ratio $\hat{C}_{\text{emp}}(s) = \frac{\sum \gamma^i \Delta t_i}{\sum \gamma^i \Delta n_i}$ whenever empirical cost exceeds the analytical acceptance model.
• Live $T=1$ Seeding & Graph-Capture Subtraction: In GuildSession::generate (src/runtime/session.cpp), one-time CUDA graph capture time (m_impl->ver->ms_capture delta) is subtracted from verification observations so graph compilation never pollutes steady-state cost estimates, and the initial $T=1$ TTFT window seeds ver_decayed_ms_[1] directly for the current prompt’s exact KV context length.
• Conservative Short-Match Prior & Hysteresis:
  • Uses a conservative prior (q = 0.32) for isolated 3-gram matches (match_len == 3, freq == 1) so accidental 3-token function-word overlaps in creative prose or math (W7, W8) do not trigger unprofitable verification windows, while immediately boosting repeated 3-grams (freq >= 2) and longer matches (match_len >= 4, q = 0.78..0.95).
  • Requires a 5% (hysteresis_margin = 0.05) expected ms/token improvement and min_dwell_steps = 3 before switching strategies, with immediate backoff after 2 consecutive zero-acceptance windows.
• MTP Confidence EMA & Cap Separation: Tracks mtp_prob_ema in Adaptive mode so a single uncertain token (dprob[0] < spec_min_p) falls back for that window without latching MTP off for the rest of the sequence, and bounds MTP draft steps by mtp_cap_k while allowing Lookup/Retrieval to propose up to spec_max_draft.

3.4 Target-Distribution-Preserving Speculative Sampling Verification

• Guild’s Verifier::run (src/core/verify.cpp) evaluates the exact target sampling chain (repetition/frequency/presence penalties $\rightarrow$ top_k $\rightarrow$ top_p $\rightarrow$ min_p $\rightarrow$ temperature) at every window row $t \in [0, T-1]$, drawing uniform variates from position-indexed Philox(seed, pos0 + t).
• For deterministic proposals (Lookup and Retrieval), accepting draft prefix $d_0 \dots d_{a-1}$ iff $d_t = Y_{\text{pos0}+t}$ and emitting $Y_{\text{pos0}+a}$ produces the exact same token sequence (bitwise identical for any fixed seed) as single-token Normal sampling because every committed position $q$ emits $Y_q = F^{-1}_{P_{\text{target}}(\cdot \mid x_{\le q})}(\text{Philox}(\text{seed}, q))$.
• In GuildSession::generate, PenaltyHistoryGuard stages per-row penalty histories via kernels::penalty_rows across the window so repetition/frequency/presence penalties are also exact across all $T$ rows.

3.5 GEXEC Speculative Cost Calibration (include/guild/gexec/gexec_contract.hpp, src/gexec/gexec_optimizer.cpp)

• Extended GExecHeterogeneousPlan with hardware-calibrated speculative fields (spec_calibrated, verify_window_ms[1..8], verify_cpu_expert_ms[1..8], distinct_expert_ratio[1..8], mtp_draft_step_ms, mtp_kv_catchup_ms, lookup_draft_ms, retrieval_draft_ms, kv_commit_ms) protected by the section’s SHA-256 integrity digest.
• During guild optimize (Pass 4), calibrate_speculative_costs microbenchmarks SuffixDrafter::propose and RetrievalDrafter::propose on the host CPU and computes the MoE multi-token verification cost curve $V(1..8)$ and distinct expert expansion ratio $U(T)/U(1)$ from the Pass 3 tuned CPU expert decode latency and model MoE geometry.
• guild inspect <artifact.gexec> --plan displays the calibrated Verify T=1..4, Distinct Experts, and Lookup / Retr timings.

---

4. Hardware, Software & Artifact Provenance

• Primary Host Hardware: Dual-socket Intel Xeon Gold 6130 (32 physical cores / 64 logical threads, Skylake-SP, AVX-512BW/DQ/VL, 2 NUMA nodes, 376 GiB DDR4 RAM) + NVIDIA GeForce GTX 1070 (8 GiB GDDR5, Pascal sm_61, CUDA 12.8, driver 570.124.06).
• Secondary Workstation Target (Ryzen 5 5600X + RTX 5070): Pending hardware access; no numbers are fabricated or extrapolated for unavailable hardware.
• Artifact SHA-256 Digests:
  • /mnt/models-ssd/Strata-data/gpacks/ornith-1.5-35b-ext.gpack: ac41e2e04d14949e998f6aade627de034b013b1f1d4670329f2e1e5efd67acdf (manifest SHA-256: b7631598f1cc21bbaf0e0e73844ed86d4cc90a9027823b03d0172438cb86e0ea)
  • /mnt/models-ssd/Strata-data/gpacks/qwen3.8-flash-next-ext.gpack: 93f3fcd86786870c487ee72b8dbd362973bdd71c1b0d94bdad6ee689b98856e5 (manifest SHA-256: 62eca78bdc5544191166276cfea50cfa58f150ff9d00a3498cb633c20b91c146)
  • build-cuda12/phase66/gexec/ornith-65k-s0.gexec: cache key bcddd79ec7030fe4... (cpu.expert_pool.gexec_fused_avx512, spec_calibrated = true)
  • build-cuda12/phase66/gexec/qwen-65k-s0.gexec: cache key 4fd75c92366672d5... (cpu.expert_pool.gexec_fused_avx512, spec_calibrated = true)
  • build-cuda12/phase66/gexec/qwen-65k-s2.gexec: cache key 1950047812444efd... (cpu.expert_pool.gexec_fused_avx512, spec_calibrated = true)

---

5. Measured Benchmark Results

5.1 Ornith 1.5 35B (ornith:35b) — 10 Workload Categories (W1–W10)

All runs executed with --gexec build-cuda12/phase66/gexec/ornith-65k-s0.gexec (gexec_active: true), --pcie-frac 0, --spec-max-draft 4, and verified for 100% bit-for-bit greedy token ID parity against normal.

5.2 Ornith 1.5 35B (ornith:35b) — W10 Populated-Context Scaling (512 to 32,704 Tokens)

---

5.3 Qwen3.8-Flash-Next (qwen3.8-flash-next) — Workloads W1–W9 across normal, mtp, lookup, retrieval, adaptive

All runs executed with GUILD_IQ_MT_MIN=1, --gexec qwen-65k-s0.gexec / qwen-65k-s2.gexec (gexec_active: true), and verified for 100% bit-for-bit greedy token ID parity across all strategies.

5.4 Qwen3.8-Flash-Next (qwen3.8-flash-next) — W10 Populated-Context Scaling (512 to 65,472 Tokens)

---

6. Analysis of Negative / Neutral Cases & Non-Negotiable Acceptance Gates

1. Why Fixed Prompt-Lookup Regresses on W7 (Creative Prose) and W8 (Mathematical Reasoning):
   • On W8 (qwen3.8-flash-next), mathematical text contains frequent 3-token structural phrases (" = 1 + ", " for real x ") that match earlier equations in the prompt, causing fixed lookup (k=4) to propose 36 draft tokens across 9 four-token verification windows (T=5), of which only 3 tokens (8.3%) are accepted.
   • Because each T=5 window on Qwen3.8-Flash-Next activates ~3.32x as many distinct CPU experts per layer as T=1, those 9 failed T=5 windows cost +555 ms of wasted CPU MoE verification, dropping throughput from 18.46 tok/s (normal) to 15.91 tok/s (-13.8% regression).
   • Similarly, on Ornith 35B W8, fixed lookup (k=4) accepts only 41.1% (30 / 73), dropping throughput from 50.15 tok/s (normal) to 47.91 tok/s (-4.5% regression).
2. How AdaptiveDecodeScheduler Eliminates Those Regressions:
   • On Ornith 35B (W7 and W8, where no MTP layer exists), adaptive ignores isolated single-occurrence 3-grams (kPriorLookupQ[0] = 0.32), limits initial probes on longer matches, and backs off to normal after 2 consecutive rejections (fallback_to_normal_count), finishing W7 at 49.53 tok/s (vs 49.55 tok/s normal) and W8 at 50.01 tok/s (vs 50.15 tok/s normal).
   • On Qwen3.8-Flash-Next (W7 and W8, where MTP is available), adaptive evaluates $C(\text{Mtp}, 1) \approx 41.1\text{ ms/tok}$ against $C(\text{Lookup})$ and $C(\text{Normal}) \approx 54.2\text{ ms/tok}$, selecting Mtp for 30 windows (0 Lookup windows) and achieving 22.15 tok/s (+20.0% over normal) on W7 and 24.31 tok/s (+31.7% over normal, +52.8% over fixed lookup) on W8.

---

7. Verification Gates & Remaining Limitations

• CPU Build & CTest (build-cpu): 33 / 33 tests passed (100%).
• CUDA Build & CTest (build-cuda12): 97 / 97 runnable tests with present fixtures passed (0 regressions; adaptive_scheduler_test, suffix_drafter_test, controller_test, draft_policy_test, coupled_draft_test, gexec_contract_test, gexec_optimizer_test, ornith_step_test, ornith_prefill_test all passed).
• Sanitizers:
  • build-asan (-fsanitize=address,undefined): 0 first-party errors across all Phase 6.6 test binaries.
  • build-tsan (-fsanitize=thread): 0 data races across all Phase 6.6 test binaries (including 4-thread concurrent RetrievalDrafter session isolation stress tests).
• Remaining Limitations:
  • Secondary workstation validation (AMD Ryzen 5 5600X + NVIDIA RTX 5070) remains pending physical hardware access.
  • On Qwen3.8-Flash-Next, exact bit-for-bit greedy parity across different verification window sizes ($T=1$ vs $T \ge 2$) requires GUILD_IQ_MT_MIN=1 so IQ3_S/IQ2_S/IQ4_NL CPU expert kernels use the same SIMD reduction tree at nt == 1 and nt >= 2.

---

8. Phase 6.6.1: Hardware-Calibrated Adaptive Decode Optimization

Following Phase 6.6 (ed7a5db), we executed a focused, measurement-driven optimization pass on the adaptive decoding scheduler, GEXEC hardware calibration, and MTP catch-up path.

8.1 Audit of Performance-Sensitive Constants & Heuristics

Every parameter in AdaptiveDecodeScheduler, SuffixDrafter, RetrievalDrafter, GExecSpecCalibration, and GuildSession::generate was audited and classified into:

• A: Correctness, ABI, or bounded-$O(1)$ memory/latency safety invariant.
• B: Explicit user-configurable policy.
• C: Hardware-dependent performance estimate (measured in guild optimize and refined online).
• D: Workload-dependent statistical estimate (learned online per session via Beta-Binomial posteriors).
• E: Unnecessary or poorly justified heuristic (replaced or eliminated).

---

8.2 Root-Cause Resolution of Targeted Phase 6.6 Gaps

1. Qwen W3 (JSON Schema): 21.66 tok/s (Phase 6.6) $\rightarrow$ 23.52 tok/s (Phase 6.6.1), matching/beating Fixed MTP (23.43 tok/s):
   • Root Cause: In Phase 6.6, MTP achieved 80.6% (25/31) acceptance on W3, but hit two consecutive Bernoulli rejections mid-stream (which occurs with probability $1 - (1 - 0.194^2)^{31} = 69.5\%$ over 32 windows). Multiplying by 0.68^2 = 0.46 and enforcing reexplore_interval = 24 locked Adaptive in Normal for 25 windows (+220 ms).
   • Fix: Sample-size-aware Beta-Binomial posterior (mtp_succ_[0] / (mtp_trials_[0] + 0.55 * r * (r + 1))) + SuffixDrafter counter-slot consensus filtering (last_consensus_k() == 0 after {"record_id": ). Adaptive now executes all 32 MTP windows (0 wasted Lookup windows), improving throughput from 21.66 tok/s (46.17 ms/tok) to 23.52 tok/s (42.52 ms/tok, +8.6%).
2. Qwen W2 (Summarization) & W4 (RAG Q&A):
   • Root Cause: During Lookup(k=4) windows (a + 1 = 5), MtpDrafter::prefill entered the multi-group prompt-prefill path (n > 0), allocating a heap std::vector, reallocating pf_dev_ via cudaFree/cudaMalloc, and capturing prefill_dev_exec_[5] inside the timed decode loop.
   • Fix: Routing n <= max_t_ to the mapped-memory prefill_exec_[T] graph in src/core/mtp.cpp improved Qwen W2 Adaptive from 33.70 tok/s to 34.16–34.49 tok/s and Qwen W4 Adaptive from 32.02 tok/s to 32.15–32.37 tok/s.
3. Qwen W9 (Short Interactive Chat p=64, g=64): 18.61 tok/s (Phase 6.6) $\rightarrow$ 22.07 tok/s (Phase 6.6.1, +18.6%):
   • Root Cause: Early misses in short interactive responses triggered the 24-step Normal lock before g=64 completed.
   • Fix: Scaling neural MTP re-exploration as clamp(2 * r, 2, 6) allows rapid recovery after opening-token uncertainty, lifting W9 Adaptive from 18.61 tok/s to 22.07 tok/s (45.30 ms/tok).

---

8.3 Controlled Phase 6.6 vs Phase 6.6.1 Benchmark Comparison

A. Ornith 1.5 35B (ornith:35b, GEXEC Calibrated — 100% Bitwise Greedy Parity)

B. Qwen3.8-Flash-Next (qwen3.8-flash-next, GEXEC Calibrated — 100% Bitwise Greedy Parity)

---

8.4 Held-Out Validation Workload Suite (H1–H4, Unseen During Tuning)

To verify that Phase 6.6.1 generalizes without overfitting to W1–W10, we evaluated 4 held-out prompts (H1: SQL Migration View, H2: Linux Kernel/Daemon Syslog Extraction, H3: Deep-Space Sci-Fi Dialogue, H4: 2-Sphere Riemann Curvature Tensor Derivation):

---

8.5 Reproducible Calibration & Validation Instructions for the Secondary Desktop (Ryzen 5 5600X + RTX 5070)

When running on the secondary workstation (AMD Ryzen 5 5600X 6C/12T AVX2, 32 GiB DDR4 + NVIDIA RTX 5070 Founders Edition 12 GiB GDDR7, sm_120), server .gexec files are automatically rejected by GExecArtifactIdentity::validate (CpuIsaMismatch / NumaTopologyMismatch / GpuArchitectureMismatch) and regenerated locally:

# 1. Configure & build for RTX 5070 (Blackwell sm_120) + Ryzen 5 5600X (AVX2)
cmake -S . -B build -DGUILD_ENABLE_CUDA=ON -DGUILD_BUILD_TESTS=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j12

# 2. Generate hardware-calibrated GEXEC artifacts on the desktop (measures Ryzen 5600X AVX2 ExpertPool T=1..8,
#    6-core concurrency contention slope, lookup/retrieval latency, and timing noise CV)
./build/guild optimize <path/to/model.gpack> --mode auto --context 32768 --prefill-batch 512 --spec 0 -o desktop-s0.gexec
./build/guild inspect desktop-s0.gexec --plan

# 3. Run adaptive decoding with live GPU+CPU T=1 split seeding
./build/guild run <model> --gexec desktop-s0.gexec --decode-strategy adaptive --spec-max-draft 4