▌redraft

Interactive LLM applications keep re-invoking the model over contexts that barely change: a user fixes a typo in a document and the summary regenerates, an agent updates one line of a plan and the review regenerates. Prefix caching already reuses the unedited part of the prompt. On the output side, open serving stacks throw away whatever part of the previous answer would survive the edit and re-decode it serially, token by token, at full price. redraft targets that output side, so decoding cost tracks how much of the answer the edit changed.

The mechanism: the stale answer is replayed as a self-speculative draft under the edited context, verified in batched windows, and re-anchored after each divergence. Spans the model still agrees with are held at batched-verification price; where it diverges, a short serial burst decodes the correction and the draft re-anchors further down the old answer, so agreement past the divergence is salvaged too, not just the common prefix.

Why exact matching falls short

Teacher-forcing the stale answer under the new context reports 85–98% agreement even when free decoding would flip the answer, because conditioning on the stale prefix keeps the model agreeing with itself. Measured on free decodes instead, reuse ceilings are low: even an edit that changes nothing the answer depends on leaves only about half the output tokens reusable, since greedy decoding is sensitive to small context changes. Exact greedy equality, the lossless contract of standard speculative decoding, inherits that sensitivity, and exact matching holds a median of only 36% of tokens.

redraft instead accepts a draft token when its logprob sits within a tolerance band of the argmax, with the band scaled by the local top-k entropy and a floor. In a sweep of tolerance rules on the 36-case suite, this rule held the most tokens (median 68%) without correctness regressions across 3B, 14B, and 35B models. The selection is in-sample and the suite is small, so a held-out evaluation is the next step.

Numbers

Qwen2.5-14B on one RTX 4070 Ti Super, 36-case edit suite, speedup in wall time per emitted token vs. full regeneration:

• Document summaries: 2.28× median
• Factual answers: 2.65× median
• Code review: 1.27× median
• Up to 10.2× on cosmetic edits whose answer is fully held

Inside the engine a held token costs about 0.08× a serial decode. The same algorithm driven by a client over a scoring API stays near parity because each verified token costs far more on that path and every call pays HTTP overhead. On a hybrid Gated DeltaNet 35B, where the recurrent state can’t be rewound, a reprocess-on-divergence verifier with a cached post-template state brings document summaries to 1.33× per token, with long, mostly stable answers at 1.8–3.5×; short answers stay near or below parity.

Layout

• paper/: preprint with method, acceptance-rule math, cost model, all results.
• engine/: llama.cpp server patches and the C++ stabilizer core.
• bench/: the measurement harness, raw results, and the annotated 36-case edit suite with six fact-flip canaries (bench/cases/).
• demo/: interactive demo, a reactive document editor racing redraft against regeneration over a FastAPI app in front of the patched server.

History

The idea started as the incr question (track dependencies, recompute only the delta) applied end to end to inference. Milestone zero found the obvious metric invalid, since teacher-forcing the stale answer masks real divergence; the free-decode ceiling turned out low even for edits that change nothing relevant, which pointed at output stabilization as the primitive. The milestone trail includes the negatives that redirected it: client-side placement stays near parity, a fixed tolerance band regressed where the entropy-scaled one did not, and recurrent memory silently corrupts under dense rollback.

License: Apache-2.0.