Hosted query-aware compression. Scores lines against this query and keeps original wording.
v2 launch · Sep 2026 · 390 public cases · evidence containment
SuperCompress benchmarks.
Prove what stayed.
SuperCompress v2 is the hosted query-aware Neural Keep model. It reads the question, keeps the original context lines that answer it, and drops the rest. We measure whether required evidence survives compression, not whether a downstream chat model finishes the task. Downstream LLM eval has not been run yet.
Public cases
390
Launch suite across needles, LongBench slices, multihop, coding agents, scaling.
Metric
Evidence
Required lines must survive in original wording, not a chat-model grade.
Engine
Neural Keep
Hosted query-aware model. Reads the question; drops irrelevant log and tool-output lines.
Path
Hosted API
Fly CPU Neural Keep, not the millisecond compiler path. MCP + hooks for coding agents.
Two engines, do not mix numbers
Lightweight local path. CPU. Millisecond-class preprocessing. Legacy results live in the labeled section below, separate product.
We score containment of required lines. Prefer “preserves required evidence” over “same answers” until downstream eval ships.
Coding-agent suite (B5)
Held-out coding-agent dumps. Same scorer for every method. Tokenizer: tiktoken cl100k_base. We measure whether the required evidence survives in original wording, stack traces, IDs, paths, error lines, not whether a downstream chat model finishes a task.
Headline cut figures from this suite are on hold pending re-verification. Do not treat summary JSON as marketing numbers.
Raw methodology JSON remains at launch-benchmark.json for researchers. Product writeup: v2 launch.
Comparators on the same dumps (qualitative, not a scoreboard): truncation is position-blind; summarization rewrites original lines; LLMLingua-2 is research-style perplexity prune; Headroom leans on content-type / local proxy workflows. SuperCompress is query-aware extractive keep on the hosted API, with MCP + hooks for coding agents.
Aggregate honesty (all 390 cases)
Mean cut across the full 390-case mix is conservative by design. On needle / LongBench / multihop slices the engine often keeps nearly everything when every line could matter, it knows when not to compress. Query-aware keep shines on noisy coding-agent dumps (B5), where most lines are irrelevant log or tool output.
Downstream LLM completion eval: not run.
How we measured
- Quality metric: answer / needle / evidence containment in the compressed text (original wording).
- Not measured yet: downstream LLM task success after compression.
- v2 inference: hosted query-aware Neural Keep, Fly CPU path for the API.
- Comparators: Headroom 0.36.4, LLMLingua-2, truncation, and SuperCompress v1 compiler on B5.
Limitations
- Latency: Neural v2 B5 p50 is ~5.8s on the hosted Fly CPU path, not millisecond-class.
- Downstream eval: evidence containment ≠ proven answer quality from a chat model.
- Workload fit: strongest on coding-agent / tool-dump noise; conservative on dense needle/QA mixes.
Compiler / v1 results (not Neural v2)
These numbers are from the lightweight compiler path (CPU, millisecond-class). They are not SuperCompress v2 Neural Keep results. Kept here for history and for teams still on the local compiler.
| Legacy metric | Result | Notes |
|---|---|---|
| Held-out gold containment | 99.4% (180/181) | Compiler suites real / fresh4 / fresh5 · Aug 2026 |
| Mean cut (compiler) | ~58–66% | Maximize drop with an answer-keep gate (compiler-era; not Neural Keep) |
| Fixed 35% budget oracle recall | 100% | Public 8-seed suite vs ~25% truncation |
| Typical latency | ~10–47 ms CPU | Local compiler, not the hosted Neural Keep path |
FAQ
Why isn’t aggregate mean cut a blanket token-cut claim?
Across all 390 cases many slices are already dense; v2 refuses to over-cut when evidence is everywhere. Query-aware keep is strongest on noisy coding-agent dumps, not as a product-wide percentage.
Does evidence containment mean the same answers from GPT/Claude?
No. It means required evidence lines were present after compression. Downstream completion quality is a separate eval we have not published yet.
GPU or CPU?
Hosted Neural v2 → Fly CPU today (seconds-scale on long dumps). Compiler → CPU, milliseconds. Pick the path that matches the product you ship.