▌EVOKE

Long-running LLM agent sessions outgrow the physical KV cache budget within a few turns. EVOKE evicts low-relevance blocks under budget pressure and recovers them recompute-free via a custom save/restore primitive in a forked llama.cpp. Restoring a block is 20–32× faster than re-prefilling the same tokens, and the full save-then-restore cycle is 5.9–7.5× cheaper.

Demos

Qwen 2.5 7B, pure attention. A 14-turn session with a 1024-token budget. A fact is planted at turn 1 (favorite number = 4242), 12 unrelated knowledge questions fill the session, and at turn 14 the fact is probed. The session survives 68 evictions and 10 recoveries, and the model recalls “4242”.

Qwen 3.5 9B, hybrid Mamba/Attention + mrope, thinking-mode. Same demo at a 2048-token budget (thinking traces run longer). The model emits a <think>...</think> trace each turn; with EVOKE_SUPPRESS_THINKING_STRIP=1 the server keeps the trace in the returned content so cached state stays aligned with what the client echoes back, and no session resets fire. 20 evictions, 4 recoveries, fact recalled.

What it actually is

• Two new C++ primitives in a forked llama.cpp: llama_kv_block_save and llama_kv_block_load. They serialise a position range’s K/V tensors to a host buffer and splice them back with per-cell RoPE re-anchoring, with no llama_decode call.
• A third C primitive llama_attn_capture_* that taps per-head softmax attention weights from one or more chosen layers (up to 16) into a host buffer once per decode. Used by the relevance scorer to learn what the model is actually attending to.
• A per-layer residual capture (llama_set/get_embeddings_layer_inp) feeding a workspace probe: a ridge probe distilled from the model’s Jacobian-lens workspace readout scores blocks by content at prefill time, one dot product per position. Probes and the offline validation pipeline live in the companion j-space repo.
• A Python policy layer (evoke/manager.py, evoke/scorer.py, evoke/attention_scorer.py) that drives eviction under a watermark policy via a multi-signal scorer (model attention + harness priority + task-focus coherence + recency) and routes recovery through three pluggable backends: discard, breadcrumb, or kv_restore (the recompute-free splice).
• An OpenAI-compatible chat-completions server that exposes EVOKE as a stateful endpoint. The persistent KV cache survives across requests; only the new tail of each prompt is decoded.
• Cross-architecture coverage: pure attention with standard RoPE (Qwen 2.5, Llama 3), hybrid Mamba/Attention (Qwen 3.5), MoE attention with mrope and thinking mode (Qwen 3.6 35B-A3B).

How does the system know what’s relevant?

Five signals, combined into a per-block score in [0, 1]. Lowest scores get evicted first when the cache exceeds budget.

• The workspace probe (new, and the headline result). Attention-history selectors can only rank blocks the model has already attended to. The workspace probe is forward-looking: it predicts, from content alone at prefill time, whether a block holds the kind of workspace representation later reasoning reads from. On pre-registered offline gates it ranks the task-critical block at fact-AUC 0.891 vs SnapKV’s 0.622 (Qwen 2.5 7B), replicated 0.865 vs 0.563 on Qwen3-8B and 0.939 vs 0.541 on Qwen3-4B with zero hand-tuning; under masked eviction at a 25% budget it keeps the fact answerable in 35/36 episodes vs SnapKV’s 25/36 on Qwen 2.5 7B (recency: 8/36), and 35/36 vs 14/36 on Qwen3-8B; on the live agent bench it passes every budget with recovery disabled while H2O fails all three, with no measurable decode overhead (-0.6%, within noise).

• The model’s own attention. A second softmax for one or more chosen transformer layers runs alongside the main attention path, writing per-head softmax weights to a host buffer once per decode. The scorer maintains a sliding window of recent attention mass per block (last 64 decode steps, EWMA decay 0.95). Blocks the model is actually attending to score high. This is the strongest single signal — the truest answer to “what’s relevant right now.”

• Harness-supplied priority tags. A coding harness like opencode or Claude Code can set evoke_priority (a float multiplier) and evoke_pinned (boolean, excluded from eviction entirely) on each chat request. Useful when the harness knows things the model can’t see: a file read is the central artifact of the current task; a tool scratch output is one-shot. Defaults to 1.0 / false.

• Task-focus coherence. The scorer tracks a single task-focus embedding that updates via EMA on new user messages but snaps to the new message when a topic shift is detected (cosine drop below 0.3) or signaled by the harness via evoke_task_boundary=true. Blocks from a prior task lose their coherence score in one pass instead of decaying over five turns.

• Recency, sink protection, source-type floors. Stability priors: prevent thrashing on a single attention spike; protect StreamingLLM-style sink tokens; give USER and ASSISTANT turns a floor so the conversation backbone isn’t evicted before document content.

Final score: min(priority * (w_jlens·workspace + w_attn·attn + w_rec·recency + w_coh·coherence) / Σw, 1.0) lifted by a source-type floor (USER blocks 0.6, ASSISTANT blocks 0.5 by default) and with pinned-block protection.

Latency

Measured on Qwen 2.5 7B, RTX 4070 Ti SUPER, Flash Attention enabled. kv_block_load is the EVOKE recovery path; re-prefill is the cost of re-encoding the same tokens via llama_decode.

Block (tokens)	save (ms)	load (ms)	re-prefill (ms)	speedup
20	1.10	0.48	11.90	25×
40	1.61	0.70	13.78	20×
160	4.69	1.50	32.60	22×
640	16.37	4.34	118.36	27×
1280	31.90	7.25	232.18	32×

Speedup is re-prefill divided by load, the latency paid at recovery. Save is paid once at eviction, so the full evict-then-recover cycle (save + load) is 5.9–7.5× cheaper than re-prefill on Qwen 2.5 7B and 2.6–2.8× on Llama 3.1 8B, whose larger per-token K/V raises transfer cost.

The gap widens linearly with block size: re-prefill is O(tokens × model_FLOPs), load is O(tokens × bytes).

Verified end-to-end

The mechanism has been run, measured, and stress-tested against a real coding agent.

• Live opencode session on Qwen3-8B (thinking model, budget = 2048) building a notes webapp, with two control arms on the same task, model and server. With recovery, 59 of 59 evicted blocks came back by identity with 0 mismatches, and turn 2 restored its whole 7,678-token turn-1 prefix without a forward pass. Evicting without recovery kept the budget but re-decoded the full prompt every turn; not evicting decoded cheaply, but its resident set grew without bound (10,952 tokens by the end of a short build).
• Four model families evaluated on one 16 GB GPU: full NIAH, multi-fact and agent-bench budget sweeps on Qwen 2.5 7B and Llama 3.1 8B, plus NIAH and multi-fact grids at a 1024-token budget on Qwen 3.5 9B (hybrid Mamba/Attention + thinking) and Qwen 3.6 35B-A3B (MoE + thinking). On all four, recovery-bearing policies pass and recovery-less baselines fail once the probed content has been evicted.

Status

Research prototype with a written paper. The mechanism is evaluated across four model families. Latest milestone: the workspace eviction signal (v2 line), validated on pre-registered gates across three Qwen models and folded into the paper with figures. Previously closed: tools-aware Jinja chat template (so tool-using turns no longer trigger session resets), multi-session pool with state swap on a custom X-EVOKE-Session header, iSWA dual-cache support in the fork primitives, multi-layer attention capture (up to 16 layers per decode).

License: Apache-2.0 on the policy layer; MIT-licensed on the forked llama.cpp work.