Hi r/huggingface ,
EDIT: For those really interested: Basically it's a new high-speed deep-storage mechanism.
It uses "scar" specializations over generalization, increasing the capabilities of pre-existent mechanisms, quoted in the paper – there's even one from 1960 lol.
What you seen is the v1, which focused on the premise of compressing pre-trained models like qwen2.5 3b, 7b and 72b. The result was impressive, but didn't represent the full capacity of this new mechanism.
The v2 is now under testing. I used the Weight-Space Programming premise to "export" the 70b knowledge into 64 ~ 256 KB cartridges of specialized knowledge, allowing to use the 0.5B as a "beater" (8x K+1 checks ~ free), the 3B as the Trunk and the 70B knowledge completely imported on-the-fly downloading from the cloud those 64 ~ 256KB cartridges.
We're about to testify the new era of AI. Something very similar to plug-and-play components customizing head/trunk and streaming knowledges like a Netflix – except better lol.
I understand this is all very new and unbelievable, but I don't expect trust until trial. So I'm about to release the V2 with the full open-source online store (free) for download those on-demand knowledge cartridges.
Ps: English isn't my main language, sorry.
---
Over the past months I’ve been working on attacking both halves of the Transformer memory wall at the same time — static weights `O(N)` and linear KV cache growth `O(T)` — without retraining models from scratch. Today I’m releasing `v1.0.0` (MIT) alongside a permanent technical preprint and a **7-vector Adversarial Red-Team Falsification Suite** designed specifically to try to break my own claims.
* **GitHub (`v1.0.0`):** https://github.com/byte-od/mixture-of-scars
* **Preprint & Replication Archive (Zenodo DOI):** https://doi.org/10.5281/zenodo.24143839
---
### What it actually is (mechanically, zero buzzwords)
**`SCacheDynamicCache` (`O(1)` HuggingFace `Cache` drop-in):*** Keeps `N_sink` attention sinks + a small rolling local FIFO window `W`, and instead of throwing away evicted tokens (like `StreamingLLM`) or summing them naively (Hebbian fast weights, which ghost/hallucinate when facts change), it routes evicted `(k, v)` pairs per KV head into persistent associative slots updated via the orthogonal **Widrow-Hoff Delta Rule**:
`W_scar <- gamma * W_scar + eta * ReLU(k_hat · c_m) * (v - W_scar k_hat) k_hat^T`
* Tracks monotonic cumulative RoPE positions separately from physical buffer length so `transformers` v5 4D masks and RoPE phases don't freeze once the window saturates.
2. **`Mixture of Scars (MoS)` (Static Compression + Group Streaming):**
* Quantizes dense linear layers to symmetric `INT4` and captures heavy-tailed activation outliers via **Activation-Aware SVD (`ASVD`)** low-rank (`r=32`) FP16/BF16 residual "scar" experts (`< 64 KB`), plus sequential layer-group streaming for 34B–123B models (`0.04 GiB` post-layer residual VRAM on Dual-T4).
---
### Why I built an Adversarial Red-Team Suite instead of just running WikiText PPL
Efficiency benchmarks are notorious for accidental facades (hidden `O(T)` Python lists in cache objects, or cache modules that never actually move logits). So the test suite enforces two hard physical invariants on real pretrained weights (`Qwen/Qwen2.5-0.5B-Instruct` across all 24 layers + `Qwen2.5-72B-Instruct` 80L MoS):
**Recursive `untyped_storage().data_ptr()` walker:** traverses every Python/PyTorch attribute of the cache object to assert `reported_bytes == actual_recursive_bytes` (`0.00 KB` discrepancy) and `0` bytes growth from `L=128` to `L=1,280`.
**In-flight Active Causality Ablation:** zeroes the scar buffers mid-generation and asserts immediate logit displacement (`max|Δlogits| = 0.6431`), proving attention is actively reading the evicted scars.
### Key Audited Results (`7/7` Red-Team Vectors)
* **Strict `O(1)` KV Memory:** `2,631.9 KB` (`2.57 MB`) constant across `128 -> 256 -> 512 -> 1,024 -> 1,280` tokens (`-91.4%` vs `30.7 MB` standard KV cache at `1,280` tokens; `+0.00 bytes` marginal growth).
* **Evicted Passcode Recall (Zero Fine-Tuning):** Passcode `7942-OMEGA` placed at token ~20, buried under 514 tokens of distractor text (`window=64`, so evicted ~470 tokens ago). `StreamingLLM` NLL = `4.301` (forgot it); `SCache O(1)` NLL = `1.184` (**`22.58x` higher exact target probability** with zero fine-tuning).
* **Contradictory State Overwrite (`34.8 KB`):** Updating `Alice=Paris -> London -> Tokyo`: additive Hebbian memory hallucinates (`cos=0.467`, old-state ghosting `+0.501`), while Widrow-Hoff Delta hits `cos=1.00000` (`ghost=-0.1377`).
* **Out-of-Domain Outlier Resilience (`MoS` calibrated on a *single* 24-token generic English sentence):** Evaluated on unseen Dijkstra Python code, Quantum/Maxwell equations, Technical Portuguese, and Strict JSON — cuts logit KL divergence vs FP16 by **`2.45x–4.22x`** compared to pure `INT4`, while cutting MLP RAM by **`-68.6%`** (`-92.6%` / `13.58x` across 8 multi-tenant specialist scar banks).
---
### Honest Post-Mortem: 3 Bugs the Red-Team Caught + Where It Breaks
Writing hostile tests against my own repo actually caught 3 real bugs before release (fixed in `v1.0.0`):
An old benchmark path had `project_slots=False` by default, silently bypassing scar readouts during attention.
Inheriting `DynamicLayer.get_seq_length()` in `transformers` v5 froze `position_ids` at `64` after the window filled up.
Duplicate fast-weight buffers per head were wasting ~`4.8 MB` before unifying `MultiSlotSCache` (`slot_dim=head_dim, num_slots=n_kv_heads`).
**Known limitations / boundary conditions:**
* **Cross-layer shared bases on off-the-shelf SwiGLU:** Sharing a single dense base across *different* pretrained layers without neuron-permutation alignment degrades quality (`KL ~ 0.78`) because `SiLU(W_gate x) * (W_up x)` breaks under arbitrary channel ordering. For off-the-shelf models without distillation, use per-layer `INT4` bases + ASVD scars + group streaming; reserve shared bases for multi-tenant specialists anchored to the same backbone.
* **Lossy associative compression vs. verbatim copy:** `SCache` compresses evicted history into fixed-size fast weights (`NLL 1.184` vs `4.301` StreamingLLM, vs `0.047` full exact KV). If you need bit-exact verbatim copying of a 2,000-char random hash evicted 10k tokens ago, fixed slots are lossy unless you increase `num_slots` or do lightweight scar read-alignment.
---
### Reproduce in ~3 minutes on CPU/GPU
```bash
git clone https://github.com/byte-od/mixture-of-scars.git
cd mixture-of-scars
pip install torch transformers safetensors pytest numpy
python scripts/adversarial_mos_scache_suite.py
pytest tests/test_adversarial_mos_scache.py -v
```
Happy to answer any questions on the Widrow-Hoff update math, `transformers` v5 cache internals, or failure modes!