r/huggingface • • Aug 29 '21

r/huggingface Lounge

9 Upvotes

A place for members of r/huggingface to chat with each other


r/huggingface • • 5h ago

built a tiny vision model from scratch just for learning and somehow it got 50+ downloads on huggingface πŸ˜…

3 Upvotes

so ik there are way better vision models out there yaar. i built this one just out of curiosity only, cause i wanted to learn how these things actually work from inside.

and to be honest its not entirely mine, i took a lot of help from claude to build it. like the architecture, debugging, training setup, figuring out why my mac was training so slow, all that. but i learned a lot in the process which was the main goal anyway.

its called tiny-vision. its around 24M params and trained completely from scratch, no pretrained weights or anything. i trained it on my macbook pro m5 and it took like 42 min on flickr8k (only 6k images).

basic idea is pretty simple:

image/video β†’ small ViT encoder β†’ adapter β†’ tiny decoder β†’ caption text

for videos it takes frames every few seconds, captions them and makes a timeline like [00:04–00:10] a dog runs on the beach. so any llm can kinda β€œwatch” a video even if it dont support video natively.

i also made an MCP server for it, and tbh i was surprised that it actually gives a little bit better context in claude when i throw images or videos at it. i even tested it with a 10-15 min youtube video and it still worked fine. not perfect at all, but chal raha hai so i’m happy.

i uploaded it to huggingface just to have it somewhere, and also cause i kinda like the open source thing. later when i checked it was at like 44 downloads, and now its 50+. ik thats not a lot bhai, but its my first time putting something open source out in this field so it felt pretty cool ngl. a lot of people have built way cooler stuff than this, i just made it to learn.

some honest stuff:

  • it gets the scene right most of time (snow, grass, water etc) but the subject is often wrong lol

  • it loves saying β€œa man in a blue shirt” for everything, pata nahi kyun πŸ˜‚

  • its v0.1 and trained on very small data, so dont expect too much

random thing i learned on mac (MPS):

  • swapping the conv2d patch embedding for reshape + linear was like 30x faster

  • manual attention was ~1.5x faster than pytorch’s sdpa

next i want to train on flickr30k and maybe try making it plug directly into qwen/llama through llama.cpp (mmproj). lets see kitna ho pata hai.

code: https://github.com/amar683/tiny-vision
model: https://huggingface.co/amartomar683/tiny-vision-flickr8k

would love any feedback or tips


r/huggingface • • 7h ago

Gemma 4 31B fine-tune for drug-level sentiment in medical text, appreciate feedback

3 Upvotes

I fine-tuned Gemma 4 31B for drug-level sentiment.

Input: a pharma or medical text plus a list of drugs.

Output: one Positive/Neutral/Negative label per drug, as JSON. I'm looking for people to test it on their own data.

Apache 2.0. Gated with automatic approval (you tick a box saying you won't use it for diagnosis or treatment decisions).

Sentiment, 473 held-out labels

ModelAccuracy
This fine-tune91.3%
Gemma 4 31B base90.5%
Qwen3.8-27B83.5%
Qwen3-32B83.3%
MedGemma 27B82.9%
Llama 3.1 8B76.3%

The other models were asked about one drug at a time with a long labelling prompt. Asked for all drugs at once in JSON, base Gemma scored 32% because 59% of its answers didn't parse. The fine-tune had no malformed JSON in 1,500 test answers. Negative precision: 84% vs 71% for base (45 Negatives in the test set).

Medical benchmarks

BenchmarksThis fine-tuneBest frontier score found
MedQA (thinking)94.296.5 (o1)
MedMCQA, 1k sample (thinking)81.483.9 (o1-preview)
MMLU medical (step by step)94.296.0 (GPT-5)
PubMedQA (thinking)80.282.0 (GPT-4 + Medprompt)

Frontier scores are from the Vals.ai MedQA leaderboard and published papers. Fine-tuning kept Gemma's medical knowledge: the scores match the base model.

Testing

Use the prompt format from the model card, thinking off and temperature 0. The FP8 build runs on vLLM (command on the card) and needs about 31 GiB of VRAM for the weights. I've only run it on a 96 GB RTX PRO 6000.

Useful feedback:

  • wrong labels, with the text, the drugs and the label you expected

  • document types, languages or lengths (over 4k tokens) it wasn't trained on

  • hardware and VRAM reports, especially from 48 GB cards

Trained on ~50k curated pharma and medical documents, LoRA rank 16.


r/huggingface • • 6h ago

Marimba: A System One Model for Speech

2 Upvotes

Marimba is Jev for audio: a System One model for speech. You ask typed questions about live audio and get typed answers back every 250 ms: yes/no, a choice, or a score.

Is someone speaking? Did the turn end? Which language? Did they say "refund"? Did the speaker change? How many speakers?

One frozen Qwen3-ASR-0.6B serves every question, with small readouts on top. Open weights, benchmark and code, Apache-2.0.

Everything is on the Hub:

Run it locally: pip install "marimba-audio[torch]"

  • Latency, p50 per question on an 8 s window:

  • Turn end: ~100 ms on a Mac M4 Pro CPU, ~12 ms on an NVIDIA L4

  • Voice activity: ~105 ms CPU, ~14 ms L4

  • Language: ~1.3 s CPU, ~150 ms L4

On 1,000 paired public questions it matches or beats Qwen3-Omni-30B-A3B on 6 of 9 tests.
Full table on the model card.

Code: https://github.com/Charcot-AI/marimba-api

Happy to answer questions!


r/huggingface • • 4h ago

Small useful solo ML project. An open weights MIDI generative model from scratch on laptop GPU (5060) and big pretraining on rented (5090). It beats most of larger open models on continuation of the song. Size is 477M (caption version with adapter, 464M base)

Post image
1 Upvotes

r/huggingface • • 8h ago

built a tiny vision model from scratch just for learning and somehow it got 50+ downloads on huggingface πŸ˜…

2 Upvotes

so ik there are way better vision models out there yaar. i built this one just out of curiosity only, cause i wanted to learn how these things actually work from inside.

and to be honest its not entirely mine, i took a lot of help from claude to build it. like the architecture, debugging, training setup, figuring out why my mac was training so slow, all that. but i learned a lot in the process which was the main goal anyway.

its called tiny-vision. its around 24M params and trained completely from scratch, no pretrained weights or anything. i trained it on my macbook pro m5 and it took like 42 min on flickr8k (only 6k images).

basic idea is pretty simple:

image/video β†’ small ViT encoder β†’ adapter β†’ tiny decoder β†’ caption text

for videos it takes frames every few seconds, captions them and makes a timeline like [00:04–00:10] a dog runs on the beach. so any llm can kinda β€œwatch” a video even if it dont support video natively.

i also made an MCP server for it, and tbh i was surprised that it actually gives a little bit better context in claude when i throw images or videos at it. i even tested it with a 10-15 min youtube video (this one) and it still worked fine. not perfect at all, but chal raha hai so i’m happy.

i uploaded it to huggingface just to have it somewhere, and also cause i kinda like the open source thing. later when i checked it was at like 44 downloads, and now its 50+. ik thats not a lot bhai, but its my first time putting something open source out in this field so it felt pretty cool ngl. a lot of people have built way cooler stuff than this, i just made it to learn.

some honest stuff:

  • it gets the scene right most of time (snow, grass, water etc) but the subject is often wrong lol

  • it loves saying β€œa man in a blue shirt” for everything, pata nahi kyun πŸ˜‚

  • its v0.1 and trained on very small data, so dont expect too much

random thing i learned on mac (MPS):

  • swapping the conv2d patch embedding for reshape + linear was like 30x faster

  • manual attention was ~1.5x faster than pytorch’s sdpa

next i want to train on flickr30k and maybe try making it plug directly into qwen/llama through llama.cpp (mmproj). lets see kitna ho pata hai.

code: https://github.com/amar683/tiny-vision
model: https://huggingface.co/amartomar683/tiny-vision-flickr8k

would love any feedback or tips, still learning πŸ™

(post also cleaned up a bit with claude)😌


r/huggingface • • 11h ago

"Spaces using this model" is not sorted by likes β€” it's trendingScore, then creation date. Measured across 23 model pages.

1 Upvotes

I run a benchmark Space that's registered against 23 models via the `models:` list in its README, and I've been recording its position in each model page's "Spaces using this model" list every week since September. I went in assuming the list was ordered by likes. It isn't, and the actual rule is worth knowing if you care whether anyone ever finds your Space.

## The sort

`/api/spaces?models=<id>&limit=1000` returns `likes` and `trendingScore` per row, in list order. Against that:

  1. `trendingScore` descending β€” a small block at the top, typically 5–7 rows.

  2. Then `createdAt` **ascending β€” oldest Space first** β€” for everything with `trendingScore == 0`, which is the entire long tail.

Likes are not a sort key at any level. On every page I checked, the list is not monotonic in likes, but is strictly monotonic in `trendingScore`, and strictly ascending in `createdAt` within the zero-trending block.

Rows 21–27 of `zai-org/GLM-5.3`, as `(likes, trending, created)`:

* #21 β€” 0 likes, 0 trending, created 2026-09-16

* #22 β€” 1 like, 0 trending, created 2026-09-23

* #23 β€” 0 likes, 0 trending, created 2026-09-24

* #24 β€” 1 like, 0 trending, created 2026-09-24

* #25 β€” 0 likes, 0 trending, created 2026-09-30

* #26 β€” 1 like, 0 trending, created 2026-10-02

* #27 β€” 0 likes, 0 trending, created 2026-10-06

Created dates ascend strictly. Likes alternate 0,1,0,1,0,1,0. Creation date is the tiebreak.

## It's a cliff, not a slope

On `deepseek-ai/DeepSeek-V4.1-Flash`, two Spaces with **identical like counts** β€” one with `trendingScore` 0.5 sits at position ~7, mine with `trendingScore` 0 sits at position 37. Thirty places apart on the same number of likes.

The lowest nonzero trending score I've seen anywhere is 0.5, and the top blocks are tiny: 1–2 rows on a 6-item list, 6 on a 32-item list, 7 on a 60-item list. So the gap between "trending at all" and "not trending" is most of the list.

HF doesn't publish the trending formula, but it decays to zero for anything not recently active, so it's recency-weighted. Practical read: **a burst beats a drip.** Ten likes in two days is worth something; ten likes over five months is worth approximately nothing.

## One Space, 23 model pages (measured 2026-11-16)

Same Space, same README, wildly different visibility. Position of the Space / length of that page's list:

**Visible β€” small, recent model pages**

* `ibm-granite/granite-4.2-3b` β€” #5 of 11

* `ibm-granite/granite-4.2-8b` β€” #6 of 9

* `zai-org/GLM-5.3` β€” #26 of 51

* `deepseek-ai/DeepSeek-V4.1-Flash` β€” #37 of 86

* `zai-org/GLM-5.3-Flash` β€” #47 of 66

* `MiniMaxAI/MiniMax-M3` β€” #54 of 69

* `Qwen/Qwen3.6-35B-A3B` β€” #78 of 96

**Buried β€” mid-size pages**

* `Qwen/Qwen3-Coder-30B-A3B-Instruct` β€” #132 of 152

* `google/gemma-3-1b-it` β€” #248 of 261

* `google/gemma-4-26B-A4B-it` β€” #280 of 321

* `Qwen/Qwen3-4B` β€” #378 of 455

* `Qwen/Qwen3.8-27B` β€” #489 of 811

* `Qwen/Qwen3-8B` β€” #537 of 636

* `Qwen/Qwen2.5-Coder-7B-Instruct` β€” #561 of 641

* `Qwen/Qwen3-0.6B` β€” #896 of 1000 (at the API cap)

**Invisible β€” past the 1000-row cap entirely**

`Qwen3.5-9B`, `gpt-oss-120b`, `gpt-oss-20b`, `Qwen2.5-7B-Instruct`, `Llama-3.1-8B-Instruct`, `Llama-3.2-3B-Instruct`, `DeepSeek-R1`, `TinyLlama-1.1B-Chat-v1.0`

That last group is eight of the 23 β€” and they're the popular models, exactly the ones you'd want to be listed on.

## Three things that follow

* **Don't delete and recreate a Space to "refresh" it.** `createdAt` ascending means a new Space starts at the *bottom* of the zero-trending block. An old creation date is an asset.

* **Registering against the biggest model pages buys you nothing.** On a 9-item list you're visible; on a 1000-item list you don't exist.

* **Accumulating likes slowly does nothing at all** β€” likes aren't a sort key, and a slow drip never builds trending.

The honest version: the ranking is a scoreboard for distribution, not a lever you can pull directly. Trending is downstream of reach. If your Space isn't getting in front of people, optimising its position in a list is solving the wrong problem.


r/huggingface • • 12h ago

Wiring a fixed-size recurrent state into transformers model.generate() without re-prefilling every token

1 Upvotes

We have a custom AutoModelForCausalLM port (trust_remote_code=True) for a 253M parameter linear attention/State Space Duality model. Our custom .chat() and .stream() methods hold a fixed 4.85 MB recurrent state across layers (S_prev, a 3 tap conv buffer, and a 1 token shift) and decode 1 token at a time

The problem is standard model.generate(). Right now it has no cache wired up, so generate() re-runs the full forward pass over the entire sequence on every new token. It matches the recurrent path within 1e-5 on logits, just way slower than it needs to be.

For anyone who has written a transformers wrapper for an RNN, Mamba or RWKV style model: what is the cleanest way to hook a non KV cache state tuple into GenerationMixin.generate() in current transformers? Should we subclass Cache and pass it through past_key_values or override prepare_inputs_for_generation and _update_model_kwargs_for_generation with custom model_kwargs?


r/huggingface • • 16h ago

Hierarchos-Native v0.2.0: RDNA3 WMMA, native multimodal training, and a lot of FP32 Vulkan optimization!

2 Upvotes

Hey everyone! It's been a little while since my last update, and I've been working pretty intensely on Hierarchos-Native, my Rust + Vulkan project for training, fine-tuning, and running language models without relying on Python, PyTorch, CUDA, ROCm, or oneAPI in the native runtime.

I wanted to share some of the bigger things that have landed over the past 18 days. There's been quite a bit of progress across GPU acceleration, multimodal training, numerical correctness, and general performance.

The new v0.2.0 release is now available!

GitHub: https://github.com/necat101/Hierarchos-Native
Release / Windows portable download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0

1. Native RDNA3 WMMA / cooperative matrix acceleration

One of the biggest additions is an opt-in Vulkan cooperative-matrix implementation for AMD RDNA3.

The new Mixed WMMA execution arm uses:

  • FP16 matrix operands.

  • FP32 matrix accumulation.

  • FP32 master weights and gradients.

  • FP32 AdamW optimizer moments and updates.

  • FP32 normalization, softmax, and loss calculations.

The important part is that this isn't simply converting an entire model to FP16. The optimizer and master parameters retain their FP32 representation.

I also added capability detection, CLI/GUI integration, and separate counters for actual cooperative-matrix dispatches and FP32 fallbacks.

Some of the isolated GEMM results on the qualified Ryzen Z1 Extreme / AMD Radeon Graphics device (1002:15bf) are pretty interesting:

WorkloadMeasured kernel speedup
QKV projection2.17Γ—
FFN up projection2.12Γ—
FFN down projection2.31Γ—
Vocabulary projection1.83Γ—

These are individual GEMM measurements, not whole-model speedups.

There is an important caveat: WMMA has a real crossover point. Small matrices can actually run slower because conversion, staging, and dispatch overhead outweigh the matrix-core benefit. I implemented a profitability policy that keeps smaller operations on their existing FP32 kernels.

In fact, the measured end-to-end training step was slower with Mixed WMMA in the initial 512-row fixture (14.0444 ms versus 12.2911 ms for FP32). That is why Mixed remains opt-in and disabled by default. I'm much more interested in making it genuinely faster than advertising misleading kernel-level numbers.

WMMA is currently physically qualified on the specific RDNA3 device and Windows driver configuration documented in the project. Other GPUs should not assume equivalent support.

Full FP16 accumulation is also available as a separate, narrowly qualified experimental mode.

Documentation: RDNA3 WMMA guide

2. Native multimodal image and text training

Another major development area has been the image-text pipeline.

The project now has native image processing and training paths that connect image inputs to language-model outputs through Vulkan.

Some of the implemented work includes:

  • Native RGB image input and preprocessing.

  • CLIP image encoders and LLaVA projector integration.

  • SigLIP / Gemma 3 vision training.

  • Gemma 4 vision integration.

  • Joint image/text training and encoder-decoder integration.

  • A dedicated DeepSeek V4.1 image-training path.

  • Backpropagation through the image tower and projector.

  • Export, reload, and checkpoint-related validation.

This is actual training work, not simply passing precomputed image embeddings into an otherwise frozen language model.

For the CLIP/LLaVA pipeline, the tests cover image sensitivity, gradients into image inputs, changes to image-tower/projector/text parameters, multiple AdamW updates, and export/reload behavior.

The qualification work has also expanded considerably:

  • 44 previously green text profiles now have joint image-training coverage at their frozen test geometries.

  • The Gen9 qualification retains 95 image cases and 166 text regression cases.

  • A separate physical RDNA3 replay reports 44/44 profiles and 85/85 image cases passing.

  • Native DeepSeek V4.1 image qualification includes 30 executions and 1,360 word-exact logits.

The maintained strict FP32 visible-logit gate remains max_abs < 2e-7, with rtol=0.

These are bounded qualification fixtures, not a claim that every production-size multimodal checkpoint or every architecture is fully supported. Audio and video pipelines are still outside the current qualified scope.

Documentation:

3. A lot of additional FP32 optimization

I haven't been focusing exclusively on reduced precision. A substantial amount of work over the same period has gone into improving the existing FP32 execution paths.

This includes vendor-specific shader work and scheduling optimizations for both Intel Gen9 integrated graphics and AMD RDNA3.

Some examples:

RDNA3 FP32 work

  • Specialized attention forward/backward paths.

  • Image-text attention and projector optimizations.

  • RMSNorm, RoPE, GELU, and other arithmetic specializations.

  • Dedicated AVX-512/RDNA3 shader variants where the measured hardware supports them.

  • More careful matching of CPU reference arithmetic and GPU execution order.

Intel Gen9 work

  • Additional native linear and gradient kernels.

  • Attention and activation specializations.

  • More efficient parameter upload and staging behavior.

  • Continued optimization of DeepSeek V4.1 training and execution.

One particularly interesting area has been reducing unnecessary Vulkan submissions and host/device transfer overhead in the DeepSeek V4.1 training pipeline.

On the measured Gen9 fixtures, upload batching reduced timed submissions from 1,102 to 982 per step, with paired throughput improvements of 1.052Γ— and 1.067Γ—.

On the separate RDNA3 V4.1 measurements, sharing the compute session and batching uploads reduced submissions from 970 to 509 per step, with measured improvements of 1.295Γ— and 1.217Γ— on two fixture geometries. A later session-wide upload optimization reduced that count further to 241 submissions per step, with measured improvements of 1.464Γ— and 1.366Γ—.

These are specific, paired benchmark results rather than promises of equivalent production-model speedups. The output comparisons remained word-identical in the recorded tests.

4. Preserving the original FP32 path

An important part of introducing WMMA was making sure that the existing FP32 path didn't regress.

I added a dedicated seven-workload FP32 non-regression comparison against the previous main revision:

  • 12 alternating ABBA rounds per workload.

  • Five warm-up runs and 30 timed forwards per arm.

  • Separate, hashed baseline and candidate binaries.

  • Exact FP32 logit digest comparisons.

All seven workloads stayed within the Β±2% performance band, and the logits were bit-identical in every recorded round.

The worst paired performance ratio was 1.0007.

This is particularly important to me: I don't want acceleration features to silently make the existing default execution path worse.

5. Getting started

The new v0.2.0 GitHub Release includes a prebuilt Windows x64 portable package with the CLI, desktop GUI, Vulkan companion executables, runtime assets, and build metadata.

You don't need to install Rust or Python just to run the released package. You do need Windows 11 x64 and a Vulkan 1.x compute driver for GPU operations.

Model weights are downloaded separately or supplied through a local model directory.

After extracting the archive, you can run:

.\bin\hierarchos-native-cli.exe doctor
.\bin\hierarchos-native-cli.exe devices
.\bin\hierarchos-native-cli.exe precision-info

The final command is especially useful for checking which precision capabilities your GPU actually exposes.

Download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0

What's next?

There is still a lot I'd like to improve.

In particular, I want to investigate reducing WMMA conversion overhead, improving activation and gradient reuse, finding more workloads that genuinely benefit from cooperative matrices, and expanding the range of qualified multimodal architectures.

I'd also like to continue improving the FP32 kernels rather than treating lower precision as the only route to better performance.

If you're interested in Rust, Vulkan compute, AMD GPU optimization, native LLM training, or multimodal model execution, I'd genuinely appreciate people taking a look.

Feedback from people with different Vulkan devices is especially valuable. Please include your GPU, driver, model geometry, and relevant execution counters when reporting results.

Repository: https://github.com/necat101/Hierarchos-Native

Stars, issue reports, testing feedback, and contributions are all appreciated. There is also a voluntary developer-support link in the repository README for anyone interested in helping the project continue.


r/huggingface • • 1d ago

Llama 3.2 1B, Kokoro TTS and Whisper all running in the browser for a racing game's team radio

Thumbnail reddit.com
2 Upvotes

r/huggingface • • 1d ago

Marimba: A System One Model for Speech. [R]

Thumbnail gallery
2 Upvotes

r/huggingface • • 1d ago

Looking for recommendations on open-source OCR/HTR models for large volumes of scanned PDFs

4 Upvotes

Hey everyone!

I'm currently working on a document processing system that handles a large number of scanned PDFs. Many documents contain a mixture of printed text and semi-handwritten content.

I'm looking for a reliable open-source solution that can handle:

  • Scanned PDF processing at scale.

  • Printed and semi-handwritten text recognition.

  • Good accuracy, even with poor-quality scans.

  • Document layout and text preservation.

  • Local inference without relying on paid APIs or proprietary cloud services.

I'm open to models available on Hugging Face, Ollama, or other open-source frameworks.

My available hardware is an NVIDIA RTX PRO 4500 Blackwell GPU with 32 GB VRAM.

I've been exploring different approaches, but I'd appreciate suggestions from people who have actually worked on similar document processing problems.

A few things I'd love to know:

  1. Which models or OCR/HTR pipelines have worked well for you?

  2. Do you recommend using dedicated OCR models or vision-language models?

  3. What approach would you suggest for processing thousands of scanned documents?

  4. Are there any particular preprocessing techniques that significantly improve handwriting recognition?

I'm particularly interested in real-world experience rather than just benchmark results.

Thanks in advance!


r/huggingface • • 1d ago

Prompt wording and a frozen Qwen weight edit: 1,024 archived measurements, seeking replication

Thumbnail
reddit.com
1 Upvotes

r/huggingface • • 1d ago

How much can we actually trust open base model benchmarks anymore? New study ("Benchleaking") finds ERNIE-5.5-Flash-Base reciting unseen test sets in exact order.

Thumbnail
reddit.com
1 Upvotes

r/huggingface • • 1d ago

addiesam/Qwen3.8-27B_qvb8r_2e-3_fp16 Β· Hugging Face

Thumbnail
huggingface.co
1 Upvotes

This is designed to run in 16GB card with DFlash.

As error is controlled, unlike Q4 or below, I can run it in V100 16GB with VSCode + Cline over 600+ conversation to do system development.

qvb8 inference engine:

http://github.com/addiesam/qvb8


r/huggingface • • 2d ago

Help me understand Hugging Face

27 Upvotes

I’m using ChatGPT to translate this message.

Could someone explain what Hugging Face (HF) is and how it works?
I’ve signed up and started experimenting with Spaces, but I feel like I might be missing the bigger picture of what Hugging Face is really about.
Can I only use the models if I download them to my computer? Does it cost anything?
I’ve noticed that Spaces lets me use models online, but are there any usage limits?
I’d really appreciate any advice, suggestions, or ideas to help me better understand the platform and make the most of it.

Thanks in advance!


r/huggingface • • 2d ago

Hoarding GGUFs in case Hugging Face changes? Fit 10% more on the same drive, byte-for-byte lossless

Thumbnail
gallery
8 Upvotes

With everyone archiving their favourite models lately, storage adds up fast. A few weeks ago I posted foldz 0.9 here. It's now tenz 0.25: same idea, better compression, much faster, and it runs fine on a phone. A 1 TB GGUF archive drops by roughly 80–130 GB, and every file comes back byte-for-byte identical, so llama.cpp, Ollama and LM Studio load it as before.

tenz is a lossless compressor for GGUF (and safetensors). Decompression gives back the identical file, byte for byte. It's one C file with no dependencies, and I develop and benchmark it on a Poco F5 in Termux.

General compressors barely touch quantized models, because the weights look random to them. tenz parses the GGUF, unpacks each block into its real quantized values and scales, and codes them with context models fitted to the file. It uses rANS from static tables, so decompression doesn't have to learn anything.

Results (images 1 and 2):

Qwen3.5-2B Q4_K_M: 8.53% saved, against 2.07% for zstd -19 --long and 1.78% for xz -9. Compress plus decompress takes 17 s on the phone; zstd took about 4 minutes just to compress.

Q4_K_M across 9 models: 7.6–9.2% saved, and 13.1% on Qwen3-4B.

Gemma-4-31B QAT Q4_0: 12.7%.

Q8_0: about 7.5%.

BF16: about 34%.

Against Aqueous, in a comparison someone posted here: 2–8 points more on every quantized model, and a tie on BF16.

Supported types: Q3_K, Q4_K, Q5_K, Q6_K, Q4_0, Q8_0, F32/F16/BF16, plus metadata (the tokenizer compresses to about 80% smaller). Anything else is stored as it is.

-q (quick) trades a fraction of a point for faster coding. For example, Qwen3.5-2B goes from 8.54% to 8.18% saved, and the Q4_K work is about 44% less.

The GPU experiment (image 3): I tried decoding Q4_K on the phone's Adreno 725 through OpenCL, to see whether a phone GPU could make decompression much faster. Three kernel designs later:

Its best kernel speed was about 150 MiB/s at full chunk size. It only matched the 3 big CPU cores (about 250 MiB/s) with tiny chunks that cost 2 points of size.

Byte stores to global memory are broken on this driver; it corrupted neighbouring bytes, so every write had to become an aligned 32-bit store.

Subgroup scans work, but are slow.

The option ships as -g (experimental). OpenCL is loaded at run time, the CPU decodes the same files, and the first GPU segment is always checked against the CPU. If you have a desktop GPU with OpenCL and subgroups, I'd love to see numbers.

Caveats:

It's a pre-release: until 1.0, a .tenz file only decompresses with the version that made it.

Decompression is far slower than zstd's, about 7 s for 1.2 GB on the phone, because every value is decoded from a model.

It only gains this much on model files.

Code, build line and full tables: tenz repo. Testers and results from other machines are very welcome.


r/huggingface • • 2d ago

GTA in a neural network: an open world model, and the Summaryformer

Thumbnail
huggingface.co
0 Upvotes

r/huggingface • • 2d ago

Jensen Fought for Open Modelsβ€”Now Open Source Fights for Him

Thumbnail
youtube.com
0 Upvotes

r/huggingface • • 2d ago

ShadeNet-5 35M (albedo/depth/normal) 31 MB quantised ONNX

Thumbnail gallery
2 Upvotes

r/huggingface • • 2d ago

Researchers: what would you study in a real science-centre "living lab"?

Thumbnail reddit.com
1 Upvotes

I run a research project inside a science centre in Bengaluru. Visitors play with puzzles, moving exhibits and an interactive AI room, and we capture anonymised data like how long people stay at each exhibit, how they interact with it, and staff observation notes.

Two asks:

  1. What psychological data would you collect here?
    2 Want to run your own study with us? Thesis, dissertation or side project, all welcome.

Ethics review + informed consent come first.

πŸ”— More about the project: https://project-octopus.iria.workers.dev
πŸ‘‰ Comment or DM


r/huggingface • • 2d ago

what is GUFF files

0 Upvotes

I want a deep explanation been using local AI for like 2 months now not in the hype or anything I don't have a good enough rig am just daily trying stuff out and exploring more of local AI lightly and I keep on finding this problem where llama.cpp says "guff files not found" or smth like that, so what are they? I'd rather ask reddit than just search it because it's more fun lol


r/huggingface • • 3d ago

Not sure how to build a local multimodal RAG? Here is a repo!! Clone it, use it, Own it!

Thumbnail
reddit.com
3 Upvotes

r/huggingface • • 3d ago

Project Genesis

Thumbnail
zenodo.org
0 Upvotes

I built Project Genesis, a cache-resident C++/CUDA geometric co-processor (libgenesis_core.so, ~0.51 MB) that attaches inline to existing causal language models during inference to reduce latency and boost token throughput without altering the host model's weights.

​In local testing on consumer NVIDIA Blackwell hardware across Qwen, Gemma 4, Llama 3.2, Nemotron, and Mistral checkpoints, compiled kernel execution cut end-to-end latency by 16% to 30% with commensurate gains in token throughput and O(1) memory overhead. It ships as a sealed runtime bundle supporting 20 common hidden dimensions (d_model from 576 to 8192) on 64-bit Linux or Windows WSL2 with NVIDIA GPUs (Compute Capability 8.0+: RTX 30/40/50-series, A100, L40S, H100).

​It is available now for non-commercial and research use on Hugging Face: https://huggingface.co/Grounded-Entropy/project-genesis

​Rather than taking my word for it, I'd love for people to pull it, run enable_telemetry=True on their own supported models, and share their latency, throughput, and ring-buffer telemetry results in the comments.

If you wish to speak privately on the matter, please use the contact info included on the paper.


r/huggingface • • 3d ago

Qwen4 27B Spoiler

Post image
0 Upvotes