r/speechtech • • 14h ago

Using Rumble-2-Flash's built-in CTC head for forced alignment instead of a separate aligner – some numbers

5 Upvotes

I've been building a transcription pipeline on Reson8's Rumble-2-Flash and tried using its CTC head for word timestamps. Haven't seen anyone write about it, so figured I'd share.

The checkpoint has a CTC head next to the decoder, just one linear layer on the encoder's ln_post output, and its classes are the decoder vocab plus a blank. Since the encoder already runs during generate(), you can hook ln_post, Viterbi the decoder's own tokens against the head and get word times without a second model or a second pass. On 20-60 min files that took 0.3-0.9 s per file, vs 7-17 s for WhisperX's alignment step on the same files.

The problem is the head is really peaky. 60-76% of frames are blank, a token is about 1.2 frames, and it's only ~13 frames/s. Starts are fine, but the last spike of a word says almost nothing about where it ends. I tried a few end rules against MFA alignments on LibriSpeech (picked on dev-clean, reported on test-clean):

  • end at the last spike: 115 ms MAE

  • end at the next word's start: 75 ms

  • next word's start, but cut at the first silence (30 ms, 25 dB below speech level): 57 ms

Vs WhisperX on the same 200 utterances: starts 58 vs 75 ms, ends 57 vs 34 ms. So better starts, worse ends, which makes sense with 13 vs 50 frames/s.

Also, padded batching changed the encoder output until I trimmed clips to whole mel hops, padded with a bit of silence and zeroed the masked frames. And my FLEURS Dutch WER was 9.2% vs 6.05% on the model card, which turned out to be almost all the normaliser. With the Open ASR Leaderboard's scoring the same transcripts give 6.24% (WhisperX large-v3 goes 7.05% -> 4.43%).

MFA isn't human labels btw, it's another forced aligner.

Code and raw results: https://github.com/EdwardGlockner/rumblex

Has anyone found a better way to get word ends out of a peaky CTC head? "Next start, cut at silence" works but feels hacky.

Edit: tried a blank penalty before Viterbi after a suggestion in the comments. Starts went 57 -> 52 ms on test-clean, ends unchanged.


r/speechtech • • 1d ago

Reverb-TTS: open-source zero-shot voice cloning running in the browser on WebGPU, 22x realtime on M4 Pro (MLX)

8 Upvotes

r/speechtech • • 1d ago

KaleComputer/kale-tts-1-nano a very small English text-to-speech model (14Mb)

Thumbnail
huggingface.co
12 Upvotes

r/speechtech • • 2d ago

Weight-Space Programming for Audio Models (very good ASR leaderboard WER, librispeech test-clean 0.83% WER)

Thumbnail
huggingface.co
25 Upvotes

r/speechtech • • 2d ago

Technology Voice AI engineers, you can drop your WER by up to 80% [open benchmark on Hugging Face]

19 Upvotes

Krisp team here.

We ran 11 STT setups on 265 real recordings on from offices, homes call centers and cars, with and without voice isolation. Nothing was mixed in afterwards, and the transcripts are hand-written.

With a second person talking, WER drops 71 to 80% with Krisp Voice Isolation.

Full test: https://krisp.ai/benchmarks/voice-isolation-benchmark/
Dataset with transcripts and labels: https://huggingface.co/datasets/Krisp-AI/VoiceIsolation-Benchmark-Dataset


r/speechtech • • 2d ago

We open-sourced a small model (41M) that splits a mono 2-person conversation into separate tracks per speaker

14 Upvotes

Hey everyone,

We’ve released Kezek, a lightweight speech separation model designed specifically for conversations.

Given a mono recording of two speakers — a podcast, interview, or phone call — Kezek separates it into two individual audio tracks, one for each speaker.

The important part is that it preserves the original timing, overlapping speech, and short backchannels like “yeah” or “mm-hm”. The separated tracks also reconstruct the original mixture exactly.

We built Kezek to address a practical problem in speech research: most real-world conversational audio is available as a single mixed recording, while training full-duplex speech models often requires separate speaker channels.

We benchmarked Kezek against pyannote + DialogueSidon on 121 real conversations from Open Yap:

  • Short replies preserved: 0.83 vs 0.69

  • Speaker leakage: −13.9 dB vs −10.1 dB

  • Correct channel assignment: 0.93 vs 0.79

  • Processing speed: 231 vs 51 hours of audio per GPU hour

We also tested on CallHome, a dataset of phone conversations not seen during training, achieving 0.86 correct channel assignment.

Kezek has just 41M parameters and combines speaker fingerprinting, 20ms speaker activity tracking, and a Conv-TasNet-style separation network.

GitHub: https://github.com/Manifestro/kezek

We're particularly interested in testing it on challenging real-world recordings. If you find cases involving heavy overlap, interruptions, background acknowledgements, or speaker confusion, we'd love to hear about them.

Feedback and failure cases are very welcome.


r/speechtech • • 3d ago

Kokoro Voice Mixer

Post image
34 Upvotes

I’ve created a little tool to play with different Kokoro mixes. In Kokoro TTS, you can blend different voice weights together to create new voices. We use Kokoro to play guided meditations, so I made this little “Kokoro Voice Mixer” where you can load the voice mixes we use in the app, try them out, and customize them by changing the mix.


r/speechtech • • 3d ago

Technology Fish Audio voice library, blind paired preference (72 listeners, 998 judgments, loudness-matched): usage predicts preference, the quality-check badge doesn't, wider spread than ElevenLabs

6 Upvotes

Follow-up to the ElevenLabs library test posted here last week. Two things changed because of that thread: Fish Audio's koko asked for the same study on their library, and the loudness criticism was fair, so every clip was level-matched before listening this time.

Design. 32 English narration voices, stratified rather than random, so that gender (16/16), age (11/11/10), usage band (lifetime generations, four bands), and Fish's source-audio quality check (14 passed / 18 unchecked) are each represented (marginally balanced, not fully crossed). Pool: 4,992 voices listed via the model API, 843 after gating (English, gender and age tagged, narration tag, not quality-failed, a title/description screen for real-person and character clones), 136 oversampled and checked by hand to verify the machine-generated gender and age tags and drop the clones the screen missed. Same 73-word passage as the ElevenLabs study, s2.1-pro, defaults, one render per voice (no seed exposed; stated as a limitation). Level: one static gain per clip to a common −26.1 LUFS, true peak ≤ −1 dBTP, no compression; shipped levels spanned 19 dB (−29.9 to −11.0 LUFS). Circulant design, offsets 1/3/5/9, 128 pairs. 72 paid US listeners, max 16 judgments each, no voice more than twice per listener; 998 judgments after excluding non-US sign-ins.

Analysis. Win rate with Wilson intervals sets the ranking; Bradley-Terry (MM) strengths alongside; 3,000 listener-resampled replicates for rank probabilities and coefficient intervals; logistic choice model on property differences (female, young, old, passed, expressive, log10 usage) with a side intercept. Level is not a covariate this time; residual level differences are reported as a separate check.

Results.

  • Top: "Old" 76% (CI 63–85), first in 47% of replicates. "Warm Royal" 74%, first in 32%. Bottom: "Young Narrative Storyteller" 21%. Observed range 21–76% vs 24–66% in the ElevenLabs sample; SD of per-voice win rates 0.14 vs 0.09 (different samples, different level treatment, so descriptive).

  • log10 usage: +0.12 (CI +0.04 to +0.19), positive in 99.97% of the 3,000 listener-resampled fits. Spearman usage vs win rate 0.39, computed from the published CSV (ElevenLabs: 0.26, coefficient ≈ 0).

  • Quality check passed vs unchecked: 41% of 570 cross pairs (CI 37–45); coefficient −0.09 (CI −0.40 to +0.21) once usage is in the model, so the adjusted estimate is compatible with zero. The likely confound, documented in the report with upload dates: the check is new, and the passed voices in the sample are recent, low-usage uploads.

  • Female: 54% of 568 mixed pairs (CI 50–58), coefficient +0.24 (CI +0.03 to +0.47). Young −0.08, old +0.13, expressive tag −0.12, all straddling zero. ElevenLabs had old at −0.19.

  • Residual level: louder clip 53% (CI 50–56) over a 1.8 dB residual spread; r(level, strength) = 0.23 vs 0.34 unmatched on ElevenLabs.

  • Side bias −0.12 (CI −0.31 to +0.06). Win-rate intervals are Wilson (per voice); coefficient and rank intervals are from the listener bootstrap.

Interim: 998 of a specified 1,152; the page refreshes when it completes. Per-judgment CSV: https://woodstitch.com/research/data/study007_judgments.csv. Report: https://woodstitch.com/research/which-fish-audio-voice-is-best-for-audiobooks

Disclosure: Woodstitch is our platform. We paid for the Fish renders ourselves and took nothing from Fish Audio; they saw nothing before publication. One correction to the ElevenLabs post, flagged by a reader: that library only carries professional clones, so its "professional vs instant" split was actually Studio-Quality-badged vs standard; badged voices won 54% (CI 50–58). The page is corrected.


r/speechtech • • 3d ago

We evaluated 26 streaming STT systems on voice-agent entity accuracy. Here’s how our new model compares.

15 Upvotes

I work at AssemblyAI. We released STT-3.7: models for recorded audio, live conversations and voice agents, supporting 32 languages.

Voice-agent entity error on 12,460 scripted caller clips, lower is better:

• STT-3.7-Realtime-Agent: 13.2%

• Soniox Realtime v6: 15.2%

• Reson8 Rumble-2: 16.2%

• ElevenLabs Scribe v3 Live: 16.9%

On long-form English, Meta scored lower WER: 7.37% versus our Agent model’s 8.35%.

These are our benchmarks. Systems were tested between August and November. Tables include methodology, confidence intervals and run dates.

Results: https://www.assemblyai.com/benchmarks

Release: https://www.assemblyai.com/blog/introducing-assemblyai-stt-3-7


r/speechtech • • 3d ago

We open-sourced a small model (41M) that splits a mono 2-person conversation into separate tracks per speaker

4 Upvotes

Hey everyone,

We’ve released Kezek, a lightweight speech separation model designed specifically for conversations.

Given a mono recording of two speakers — a podcast, interview, or phone call — Kezek separates it into two individual audio tracks, one for each speaker.

The important part is that it preserves the original timing, overlapping speech, and short backchannels like “yeah” or “mm-hm”. The separated tracks also reconstruct the original mixture exactly.

We built Kezek to address a practical problem in speech research: most real-world conversational audio is available as a single mixed recording, while training full-duplex speech models often requires separate speaker channels.

We benchmarked Kezek against pyannote + DialogueSidon on 121 real conversations from Open Yap:

  • Short replies preserved: 0.83 vs 0.69

  • Speaker leakage: −13.9 dB vs −10.1 dB

  • Correct channel assignment: 0.93 vs 0.79

  • Processing speed: 231 vs 51 hours of audio per GPU hour

We also tested on CallHome, a dataset of phone conversations not seen during training, achieving 0.86 correct channel assignment.

Kezek has just 41M parameters and combines speaker fingerprinting, 20ms speaker activity tracking, and a Conv-TasNet-style separation network.

GitHub: https://github.com/Manifestro/kezek

We're particularly interested in testing it on challenging real-world recordings. If you find cases involving heavy overlap, interruptions, background acknowledgements, or speaker confusion, we'd love to hear about them.

Feedback and failure cases are very welcome.


r/speechtech • • 4d ago

Interact-0 Full Duplex model of the day November 17th

Thumbnail
interact.inc
14 Upvotes

r/speechtech • • 5d ago

We ran nyra's speech timing benchmark on our forced aligner, found where it failed, and re-measured after fixing it

7 Upvotes

Disclosure: I work at Olewave. We make Olign, a hosted forced aligner, and we maintain FA-Bench. This post is about our own system.

The benchmark

nyra labs released the Nyra Speech Timing Benchmark this month. It scores 22 systems on word timing against hand-labeled boundaries, on three test sets. The scorer and every prediction are public, so anyone can rerun it.

Where we started

From nyra's own table (Olign v1.0.11, mean word timing error):

Test setOlignnyra's aligner
TIMIT, read speech15.9 ms (lowest of 22)21.3 ms
Buckeye, conversation35.7 ms20.2 ms
FluencyBank, stuttered speech203.4 ms75.3 ms

Good on clean read speech. Clearly behind on real conversation.

What went wrong on Buckeye

nyra cuts Buckeye conversations into chunks of up to 20 seconds and leaves in laughter, long pauses and interviewer speech. When we looked at our errors, most of them sat in two places:

  • the edges of a chunk, the first and last words

  • words next to a pause, where the word before the pause ran into the silence

What we changed

  • v1.1.1 and v1.1.2 handle the start and end of recordings longer than 15 s differently.

  • v1.1.3 moves where a word next to a pause begins or ends, so the pause comes back closer to its real length.

After the fix

Same scorer, same chunks, Olign v1.1.3:

Test setBeforeAfter
Buckeye35.3 ms23.6 ms
TIMIT15.9 ms14.2 ms

Caveats

  • These are our own runs, not leaderboard results. We have asked nyra to re-run Olign, and their number is the one that counts.

  • The Buckeye gain is in-sample. We found the problems by looking at errors on these same chunks.

  • We did not run FluencyBank. Stuttered speech is still our weakest case.

  • Non-speech is still open. We are working on robustness to laughter and other non-speech sounds in the recording.

Tables, method and commands to reproduce: olewave.com

nyra's benchmark: github.com/nyrahealth/nyra_speech_timing_benchmark

Question for the sub: what do you use to test word timing when the audio has laughter, overlap or long non-speech stretches? Hand-labeled boundaries for that kind of audio are hard to find.


r/speechtech • • 6d ago

Nazeem on X: "I trained a text-to-speech model that has never heard a real human voice. 4.5M parameters. 18 MB. Runs on a CPU. Meet tiny-voca" / X

Thumbnail x.com
9 Upvotes

r/speechtech • • 6d ago

Marimba: a System One model for speech [R]

Thumbnail huggingface.co
6 Upvotes

r/speechtech • • 7d ago

Tontaube V1.1: open-weights TTS with voice cloning for long-form generation. 4-bit version runs about real time on a desktop CPU

Thumbnail reddit.com
4 Upvotes

r/speechtech • • 8d ago

Haru-4B: A Low-Latency and Multilingual Full-Duplex Continuous Spoken Dialogue Foundation Model (40m hours, 9 languages)

Thumbnail
blog.higurashi.ai
46 Upvotes

r/speechtech • • 8d ago

Adapting Whisper and a Nemotron to medical speech with 12 hours of audio

Thumbnail
nanosamur.ai
14 Upvotes

Hi folks, just ran some experimental fine tuning on approx. 10hrs of medical recordings (Czech language) this week and was surprised to see how big improvement it made - in case of Whisper large v3 this was a LoRA, in case of Nemotron it was a full fine-tune (I also compared it with Nemotron LoRA arm). Interesting thing is that the transcription I trained on was full of mistakes; I corrected approx 4hrs of that, but used part of it for test for obvious reasons.


r/speechtech • • 9d ago

Finetune of Gemma-2-26B-A4B for ASR from Salesforce: English, French, German, Spanish, Italiain, Portuguese, Dutch

Thumbnail
huggingface.co
18 Upvotes

r/speechtech • • 9d ago

LiquidCodec: High-Fidelity Audio Tokenization Optimized for Speed

Thumbnail
liquid.ai
19 Upvotes

r/speechtech • • 9d ago

Silero ~100ms late on end-of-speech

9 Upvotes

Tried to hunt down latency bottlenecks in my local voice assistant recently. I looked at the waveform and saw silero's probability lagging quite much behind the actual end of speech.

I use a voice pipeline that does speculative work and cancels fast, so some false positives would not matter in my case.

So I had this idea: once silero says I'm in speech, what I need to detect is when the speech energy drops back to room noise level. Turned out a naive 10 ms energy check with a threshold tracking room noise does that quite well.

Gave me a median 100 ms earlier in the current version than silero over ~3200 real turns from my assistant on my voice, my mic, browser noise suppression on, mostly quiet room - with silero threshold at 0,5. Someone on Twitter got 92 ms on his data, roughly what I saw in my noisier recordings.

The downside is way more false positives. It fires on breaths and small pauses between words over 10x as often as silero. So it's really only useful if the pipeline can throw away speculative work very fast.

CPU only, MIT, drop-in for silero VADIterator, pip install realtimevad

benchmark script is in the repo, would love to hear what numbers you get on your own recordings

https://github.com/KoljaB/RealtimeVAD


r/speechtech • • 9d ago

#1 Joint diarization+ASR model -- AirCaps

5 Upvotes

AirCaps A5S just ranked #1 on https://voicearena.com/diarization-bench (beating all the other models by a MILE). API access available on: https://console.aircaps.ai/ with 5 hours free

I work @ AirCaps so if you have any feedback / problems just let me know!

We also have an open-source benchmark across 13 datasets that you can rerun yourself (NOTSOFAR-1 16.65, AMI 15.83, macro 21.51): https://github.com/aircaps/a5s-async-v1-benchmark


r/speechtech • • 9d ago

C++ port of pyannote overalapped speech detection

6 Upvotes

I ported pyannote-audio overlapped speech detection pipeline(python) to C++ so that my C++ project can use it and also speed up processing.
https://github.com/leohuang2013/pyannote-audio_overlapped-speech-detection_cpp

The segmentation model is exported as ONNX which run with onnx runtime in C++, and binarization stage is rewritten in C++.

Result: on a 7 minutes, 4 speaker audio file, all 62 detected overlap regions match pyannote output extactly.

Any question or contribution to the project welcome.


r/speechtech • • 10d ago

Technology I built a small streaming text tagger that reads text word by word, so a voice agent can react mid sentence. upto 4096 previous words as context, 1000 list in-stream name verification

9 Upvotes

We listen and type word by word, yet most AI only acts per turn. That wastes time and tokens, and it misses intent that was already clear halfway through the sentence.

So I built Arc Tagger, a small model that runs on the words as they arrive from an ASR API or from typed text. It tags intent, dialog act, entities, numbers, PII, profanity, toxicity and safety risks while the sentence is still being spoken.

In the video you can see a bank impersonation scam get flagged mid sentence, a voice assistant tracking a customer who changes their mind, and a legal sentence tagged live.

What it is today:

  • A small model (about 11M parameters) that runs on CPU

  • English only for now

  • Tracks up to 4096 previous words

  • Can check up to 1,000 list items in the stream, which you preload before a session

  • Useful for voice and text agents, voice assistant, call center monitoring, and trust and safety

Limits I want to be upfront about:

  • The eval results on the model page are internal evals, not independent benchmarks. Will share them soon.

  • The timings shown in the demo include tunnel and browser overhead, so please don’t read them as the model’s latency

  • The license is CC BY NC 4.0, so non commercial for now

I would love honest feedback, positive or negative. Especially: where does it break, what tags would you want that are missing, and would you use this in a real voice pipeline?

Model: https://huggingface.co/WhissleAI/arc-tagger
C library and CLI: https://github.com/WhissleAI/arc-tagger-c


r/speechtech • • 10d ago

Technology Blind test of 32 ElevenLabs Voice Library voices for audiobooks (125 paid listeners, 1,122 judgments): "professional" voices no better than instant clones, popularity predicts nothing, louder voices win

35 Upvotes

We needed a defensible way to pick narrator voices from the ElevenLabs Voice Library (5,000+ English narration voices, no quality signal), so we ran a paid blind test.

Setup: 32 voices stratified across gender, age, accent, tier (professional vs instant clone) and usage; the same 30-second first-person nonfiction passage; identical eleven_v4 settings for every voice; 128 pairs in a connected circulant design; 125 paid US listeners, 1,122 forced-choice judgments; no listener heard any voice more than twice. Bradley-Terry ranking with listener bootstrap, plus a logistic property model on the voice tags.

Results:

  • Marie - Audiobook Narrator and Measured finished first (66% of pairings won, 95% CI 55–76). Alder - Warm, Raspy and Comforting last (24%). The top eight overlap and can't be ordered.

  • The "professional" badge told listeners nothing. Studio-recorded PVC voices won 46% of 649 head-to-head pairs against instant clones.

  • Popularity predicts nothing. Usage from under 1M to nearly 1B characters/year; coefficient zero.

  • Loudness matters. Voices ship up to 15 dB apart; we deliberately did not normalise. The louder clip won 54% of pairs; the quietest voice finished last.

  • "Old"-tagged voices lose, mostly female ones: young female voices won 58% of pairings, old female 43%. Male voices barely move with age.

  • Gender itself: 49/51.

All 32 clips are playable on the page; the per-judgment CSV is published (https://woodstitch.com/research/data/study004_judgments.csv). Chart of all 32: https://woodstitch.com/research/og/study004.png

Full report with method and limitations (one passage, one setting; sample raised from 3 to 9 judgments per pair after a first look, stated as a data-dependent continuation): https://woodstitch.com/research/which-elevenlabs-voice-is-best-for-audiobooks

Disclosure: we run Woodstitch, the paid listening-test platform this was run on. No relationship with ElevenLabs or any voice creator. HN thread: https://news.ycombinator.com/item?id=50458929


r/speechtech • • 10d ago

Flock Speech: open-source text-to-speech voices that fit in 44 MB and run on any CPU

20 Upvotes

Flock Speech brings natural-sounding voices to any device, with no GPU and no cloud.

Why it stands out

🪶 44 MB per voice: small enough to ship inside any app

⚡ 8.7× faster than real time on a single CPU thread

🔒 Fully offline: text never leaves the device

😂 Sound tags: \[laugh\] and \[breath\] bring the voice to life

📜 Apache-2.0: free for personal and commercial use

Meet the voices 🐦

Goldfinch: warm and conversational

Junco: clear and calm, made for reading

Get started

pip install flockspeech

🎧 Samples:

  1. https://huggingface.co/flockspeech/goldfinch

  2. https://huggingface.co/flockspeech/junco

📦 PyPI: https://pypi.org/project/flockspeech/