r/speechtech • • Oct 06 '26

Technology Speakrail - a local voice assistant that rivals GPT-Live on a single RTX 4090

339 Upvotes

I created a fully-local open-source full-duplex voice agent that rivals GPT-Live on some benchmarks. It uses Voxtral Realtime with a turn-taking head, a microturn-finetuned Gemma 4 12B and Breeze TTS 2 under the hood. Go try it out: https://github.com/speakrail/speakrail

Why I did it

I have always liked the idea of voice assistants, but there is always some non-local component in the pipeline, which increases latency and introduces privacy concerns. I tried many fully-local approaches, like HF speech-to-speech, Unmute and Pipecat, but they were all limited by either the Whisper model (hello, hallucinations!) or slow turn taking. The only fully-local pipeline that had some full-duplex capability with low latency was the DuplexCascade paper (code), but it's tuned on a Qwen 2 7B with a gpt-3.5-turbo generated dataset, and the dumbness of the model really showed. So I decided to recreate DuplexCascade with newer data, newer models and a better harness. I also wanted to add interruptions, interjections and other cool things to rival the Thinking Machines demo.

How it works

Here is what the pipeline consists of:

  • STT: Voxtral Realtime with an attached turn head (HF), running on our audio.cpp fork.

  • LLM: Gemma 4 12B QAT with microturn finetuning (HF). It is chosen because it fits the GPU quite well, has vision support (I want to test it soon), and in general, the Gemma models perform well in real-life tasks, general chatting, etc.

  • TTS: Breeze TTS 2, patched to run at int8 (GitHub fork), although it can be replaced by any streaming TTS.

  • The harness itself: it is the glue between all the components, and has many latency-saving measures, like speculative LLM+TTS firing (inspired by HF speech-to-speech).

I also took inspiration from several "think while talking" papers (e.g. SHANKS): while you are talking, a base Gemma 4 12B int4 writes thinking notes, which are then passed to the talker. It helps with harder tasks that require more reasoning.

I will release a longer technical report soon - it will have a better description of the entire pipeline.

Benchmarks

Full Duplex Bench v3:

SystemPass@1Tool selection F1Reply qualityTask done (mean)Weights
TML-Interaction-Small*0.68–0.828–closed preview
GPT-Realtime 1.50.600.8760.7926.89 sclosed
GPT-Live-1 + gpt-6-astra†0.550.8350.7178.47 sclosed
Gemini Live 3.10.540.8170.7184.25 sclosed
Speakrail0.520.9150.7553.92 sopen
Gemini Live 2.50.490.7860.5547.26 sclosed
NVIDIA frontend-backend (Qwen3-235B)0.48–0.670–research
Cascaded (Whisper + GPT-4o + TTS)0.450.8030.60010.12 sclosed
Grok Voice0.430.7970.6176.65 sclosed
Ultravox v0.70.410.7940.5108.40 sclosed
Gander 9B (GPT-5.6 backend)0.400.7590.490–open + cloud
NemotronLabs VoiceChat 11B0.330.825––open

100 tool-use tasks with real disfluent speech; 95% CI on Pass@1 is about ±10 points. Speakrail: live audio through the full stack, GPT-4o judge. *Self-reported by Thinking Machines. †Our run. Others: FDB-v3 paper, arXiv 2609.19334, 2609.08977, 2609.21967.

Conversational dynamics (Artificial Analysis formula):

SystemScoreWeights
StepAudio 3 Realtime98.9closed
Qwen Audio 3.0 Realtime Plus98.4closed
GPT-Live-1 (Sol, low)97.3closed
Qwen Audio 3.0 Realtime Flash96.9closed
Gemini 3.8 Live96.1closed
GPT-Live-1 (Astra, medium)94.9closed
Speakrail94.0open
PersonaPlex 7B (NVIDIA)91.0open
Nemotron 3 VoiceChat V1 (NVIDIA)77.8open
FLM-Audio62.0open
Moshi (Kyutai)61.0open
Freeze-Omni58.7open

Score = mean of pause handling and turn-taking (FDB v1.0) plus interruption and backchannel (FDB v1.5). Speakrail's score is self-computed from live runs (pause 93.4, turn-taking 96.6, interruption 89.0, backchannel 96.9), not an official AA listing. Others: Artificial Analysis leaderboard (captured 09-28) and AA's open-weights comparison.

Limitations

  • Breeze TTS has a restrictive license, so if you need to use Speakrail commercially, you will need to change it.

  • 16k context length - the pipeline only supports 16k context length (~1 hr of speech), but you can get more easily by changing the Breeze TTS to a Pocket TTS and run the TTS on a CPU. I chose Breeze for the release because it's more expressive.

  • The turn-taking head is undertuned on non-assistant data. It may not fire on some basic chit-chat, but I will tune it harder later.

  • The model is certainly not the smartest one, and my finetune did dumb it down a little. Next time it will be smarter / better.

  • The model is kinda verbose sometimes, and the answers it provides are somewhere in the middle between real-life speech and the long text-based outputs of LLMs. I have a hypothesis on how to fix it, and will try it in the next release.

  • I tested it only on an RTX 4090, but I am sure it's easy to add support for any 24 GB+ NVIDIA card. Forks for AMD and MLX are welcome.

Final Notes

Feel free to try it out: https://github.com/speakrail/speakrail. If there is any capability you want the model to have, create a GitHub issue or write here in the comments, and I will gladly include it in the next dataset. Any feedback is welcome as well.


r/speechtech • • Nov 25 '25

Building a Voice-Activated POS: Wake Words Were the Hardest Part (Seriously)

106 Upvotes

Building a Voice-Activated POS: Wake Words Were the Hardest Part (Seriously)

I'm building a voice-activated POS system because, in a busy restaurant, nobody has time to wipe their hands and tap a screen. The goal is simple: the staff should just talk, and the order should appear.

In a Vietnamese kitchen, that sounds like this:

This isn't a clean, scripted user experience. It's shouting across a noisy room. When designing this, I fully expected the technical nightmare to be the Natural Language Processing (NLP), extracting the prices, quantities, and all the "less fat, no ice" modifiers.

I was dead wrong.

The hardest, most frustrating technical hurdle was the very first step: getting the system to accurately wake up.

Here’s a glimpse of the app in action:

The Fundamental Problem Wasn’t the Tech, It Was the Accent

We started by testing reputable wake word providers, including Picovoice. They are industry leaders for a reason: stable SDKs, excellent documentation, and predictable performance.

But stability and predictability broke down in a real Vietnamese environment:

  • Soft speech: The wake phrase was missed entirely.

  • Kitchen Noise: False triggers, or the system activated too late.

  • Regional Accents: Accuracy plummeted when a speaker used a different dialect (Hanoi vs. Hue vs. Saigon).

The reality is, Vietnamese pronunciation is not acoustically standardized. Even a simple, two-syllable phrase like "Vema ơi" has countless variations. An engine trained primarily on global, generalized English data will inherently struggle with the specific, messy nuances of a kitchen in Binh Thanh District.

It wasn't that the engine was bad; it's that it wasn't built for this specific acoustic environment. We tried to force it, and we paid for that mismatch in time and frustration.

Why DaVoice Became Our Practical Choice

My team started looking for hyper-specialized solutions. We connected with DaVoice, a team focused on solving wake word challenges in non-English, high variation languages.

Their pitch wasn't about platform scale; it was about precision:

That approach resonated deeply. We shifted our focus from platform integration to data collection:

  • 14 different Vietnamese speakers.

  • 3–4 variations from each (different tone, speed, noise).

  • Sent the dataset, and they delivered a custom model in under 48 hours.

We put it straight into a real restaurant during peak rush hour (plates, hissing, shouting, fans). The result?

  • 97% real-world wake word accuracy.

For those curious about their wake word technology, here’s their site:

https://davoice.io/

This wasn't theoretical lab accuracy. This was the level of reliability needed to make a voice-activated POS actually viable.

Practical Comparison: No "Winner," Just the Right Fit

In the real world of building products, you choose the tool that fits the constraint.

ApproachThe ProThe Real World Constraint
Build In-HouseTotal technical control.Requires huge datasets of local, diverse voices (too slow, too costly).
Use Big VendorsStable, scalable, documented (Excellent tools like Picovoice).Optimized for generalized, global languages; local accents become expensive edge cases.
Use DaVoiceTrained exactly on our user voices; fast iteration and response.We are reliant on a small, niche vendor for ongoing support.

That dependency turned out to be a major advantage. They treated our unique accent challenge as a core problem to solve, not a ticket in a queue. Most vendors give you a model; DaVoice gave us a responsive partnership.

When you build voice tech for real-world applications, the "best" tool isn't the biggest, it's the one that adapts fastest to how people really talk.

Final Thought: Wake Words are Foundation, Not Feature

A voice product dies at the wake word. It doesn't fail during the complex NLP phase.

If the system doesn't activate precisely when the user says the command, the entire pipeline is useless:

  • Not the intent parser

  • Not the entity extraction

  • Not the UX

  • Not the demo video

All of it collapses.

For our restaurant POS, that foundation had to be robust, noise-resistant, and hyperlocal. In this case, that foundation was built with DaVoice. Not because of marketing hype, but because that bowl of phở needs to get into the cart the second someone shouts the order

If You’re Building Voice Tech, Let's Connect.

I'm keen to share insights on:

  • Accent modeling and dataset creation.

  • NLP challenges in informal/slang-heavy speech.

  • Solving high noise environmental constraints.

If we keep building voice tech outside the English-first bubble, the next wave of AI might actually start listening to how we talk, not just how we're told to. Drop a comment.


r/speechtech • • May 02 '25

I benchmarked 12+ speech-to-text APIs under various real-world conditions

105 Upvotes

Hi all, I recently ran a benchmark comparing a bunch of speech-to-text APIs and models under real-world conditions like noise robustness, non-native accents, and technical vocab, etc.

It includes all the big players like Google, AWS, MS Azure, open source models like Whisper (small and large), speech recognition startups like AssemblyAI / Deepgram / Speechmatics, and newer LLM-based models like Gemini 2.0 Flash/Pro and GPT-4o. I've benchmarked the real time streaming versions of some of the APIs as well.

I mostly did this to decide the best API to use for an app I'm building but figured this might be helpful for other builders too. Would love to know what other cases would be useful to include too.

Link here: https://voicewriter.io/speech-recognition-leaderboard

TLDR if you don't want to click on the link: the best model right now seems to be GPT-4o-transcribe, followed by Eleven Labs, Whisper-large, and the Gemini models. All the startups and AWS/Microsoft are decent with varying performance in different situations. Google (the original, not Gemini) is extremely bad.


r/speechtech • • Sep 26 '26

Yodas3 · 55TB, 1.1M hours of speech data

Thumbnail
huggingface.co
70 Upvotes

r/speechtech • • 23d ago

Technology Arc: a ~70 MB streaming speech model that transcribes links names and metadata to your own list as they're spoken, with no second model

69 Upvotes

I built Arc-Transcriber, which does ASR-NL in 1-step. Every 100 ms, one model returns the transcript plus:

- Names linked to a list you give it (a roster, contacts, product names); new names are flagged "not on the list"
- Numbers with their values (scores, clocks, money, dates)
- Speakers and roles;
- Sound and channel events (telephone audio, clipping, poor audio)
- Whether the speech was addressed to an assistant.

The usual ASR, then NER, then entity-linking and then LLM chain is collapsed into the model's own heads, so live "fact-checking" against your list happens inside the stream, not in a second pass.

Currently it has two variants Arc mini (which uses moonshot pretrained encoder) and second Arc-v1 (which uses parakeet pretrained encoder).

Weights: huggingface.co/WhissleAI/arc-v1.1
Code and CLI: github.com/WhissleAI/arc_transcriber

We use it on-device in our Mac app. Voice search over 61k books primes Arc with matching titles while you're still talking. Here is a video to give a glimpse on how it works.


r/speechtech • • Nov 18 '25

NVidia release realtme model Parakeet-Realtime-EOU-120m

62 Upvotes

Real-Time Speech AI just got faster with Parakeet-Realtime-EOU-120m.

This NVIDIA streaming ASR model is designed specifically for Voice AI agents requiring low-latency interactions.

* Ultra-Low Latency: Achieves streaming recognition with latency as low as 80ms.

* Smart EOU Detection: Automatically signals "End-of-Utterance" with a dedicated <EOU> token, allowing agents to know exactly when a user stops speaking without long pauses.

* Efficient Architecture: Built on the cache-aware FastConformer-RNNT architecture with 120M parameters, optimized for edge deployment.

🤗 Try the model on Hugging Face: https://huggingface.co/nvidia/parakeet_realtime_eou_120m-v1


r/speechtech • • Oct 18 '26

Releasing SpecSpeech: 2.29x faster ASR Inference with Speculative Decoding

Post image
54 Upvotes

Github: https://github.com/yaya-sy/specspeech

The best performing open-weight models on the Open ASR Leaderboard (https://huggingface.co/spaces/hf-audio/open_asr_leaderboard) are autoregressive speech models, such as Qwen ASR, Cohere Transcribe, and Canary. Their language modeling capabilities help them transcribe speech accurately, but they are often much slower than transducer and CTC models, such as Parakeet and Granite CTC.

I open-source SpecSpeech (Speculative Speech), an inference engine that accelerates speech language models for ASR by using transducer or CTC models as drafters.

The drafter quickly transcribes batches of short audio pieces while the Speech LM prefills the full audio in its context in parallel. The Speech LM then verifies the drafts incrementally. At each step, a lookup algorithm uses the verified output to select up to k candidate tokens from the draft transcription, which are verified by the Speech LM.

Compared to approaches such as WhisperX, SpecSpeech approach is lossless: the Speech LM uses the full audio context and retains control over token acceptance.

Across seven English benchmark datasets from Open-ASR-Leaderbord, SpecSpeech achieves a 2.29x speedup over native vLLM inference, with per-dataset speedups of up to 2.48x, using Parakeet v3 as the drafter and Qwen3-ASR-1.7B as the verifier.

Results below were measured on a single NVIDIA A40, with continuous batching for both conditions.

DatasetClipsAudio hoursNative vLLM RTFx ↑SpecSpeech RTFx ↑Speedup
AMI Cleaned7,8058.02104.55187.421.79×
Earnings22 Cleaned (AA chunked)3411.92214.38347.591.62x
GigaSpeech Cleaned18,76835.15126.74280.352.21x
LibriSpeech test-clean2,6205.40139.03345.462.48x
LibriSpeech test-other2,9395.34133.43288.702.16x
SPGISpeech39,341100.00163.41394.692.42x
VoxPopuli Cleaned (AA)6281.98182.70417.782.29x
Full suite72,442157.81148.22338.712.29x

There is room for further improvement by reducing tokenization and verification overhead and developing better lookup algorithms.

Planned additions include:

  • More verifyers soon: Cohere Transcribe, Canary, etc.

  • Word-level alignment using timing information from CTC drafters.

  • Speaker diarization.

  • Streaming inference.

  • Speculative decoding for autoregressive TTS models.


r/speechtech • • 24d ago

NVIDIA Nemotron 3.5 Transcribe and Granary v2: Intelligent Multilingual Transcription You Can Prompt and Adapt

Thumbnail
huggingface.co
47 Upvotes

r/speechtech • • 8d ago

Haru-4B: A Low-Latency and Multilingual Full-Duplex Continuous Spoken Dialogue Foundation Model (40m hours, 9 languages)

Thumbnail
blog.higurashi.ai
46 Upvotes

r/speechtech • • 15d ago

Scylla's Band v4, open source TTS with code to run everywhere

45 Upvotes

This has been a long one for me: almost a year as a solo project, learning how to build a TTS model from the ground up. I wanted something fast, small enough to run on a phone, multilingual, and (the part I cared about most) able to do something a little different from the other small TTS models. So it sings.

The demo video speech and singing are all entirely from the model. The instrumental/fx are not and were added in afterwards.

What it is:

- ~131M parameters (plus a 30M phonemizer), 14 voices, English (US/UK), Spanish, French, German, Italian and Vietnamese. All voices can speak all languages supported.

- Intensity, pace and pitch controls per line, multi-voice dialogue with inline tags, streaming

- Sings from a JSON score (each word with its note and beats), pop or ballad. English only for singing format for now.

- Warm first audio around 30 ms on an M5 Max, 70 ms on an iPhone 17 Pro Max, 120 ms on a Galaxy Z Fold 8 (CPU only); over 30x real time on a desktop CPU

- English WER 0.85%, UTMOS 4.35 on the Open TTS Leaderboard eval (submission pending)

- Apache 2.0

The GitHub and Hugging Face repos have the original PyTorch weights plus ports for LiteRT, Core ML, Core AI, ONNX Runtime and WebGPU, and a C++ runtime with a C API that handles text normalization and phonemes, so numbers, dates and odd words come out right. There's sample code for Python, iOS and Android (both apps have a Sing tab), and benchmarks from real devices. The sample apps from both allow for music score and full control over text to speech generation. Python examples for combining both and doing things like singing in unison are provided as samples.

Limitations, up front: it only speaks in its own 14 voices (no voice cloning), it still trips on the odd word, and some voices get rough on fast, wide pitch jumps when singing. This is literally the first usable version of it singing, and it's something that I'm planning on improving over time... v1 was released a few months ago, and I wasn't happy with the improvements made to it until now.

GitHub: https://github.com/lowkeytea/scyllasband

Hugging Face: https://huggingface.co/spybyscript/scyllasband

Try it in your browser: https://lowkeytea.dev/scyllasband/demo/

Happy to answer questions about training it, the singing, or getting it running on a device.


r/speechtech • • 16d ago

Technology LinnetTTS: 41M streaming TTS with a Rust runtime, measured on everything from an Apple M5 to a Raspberry Pi Zero 2 W (3.3 ms to 59 ms per 80 ms of audio)

41 Upvotes

Author here. We just released LinnetTTS, a small on-device English TTS: an autoregressive flow LM over a 25 Hz continuous-latent codec with a Vocos-style decoder, 41 M parameters end to end, one flow evaluation per step, no diffusion loop at inference. Each step produces 80 ms of audio, so playback starts after the prime pass plus one step and never waits for the whole sentence. It speaks in a voice given as a short voice-prompt latent; two built-in voices ship with the weights.

The runtime is Rust with its own kernels (AVX2+FMA on x86-64, NEON on ARM), no PyTorch or ONNX at inference. fp16 weights are 83 MB; peak memory about 160 MB on Linux. (The "35m" in the model name is a label from the original parameter budget; the count is 41.4 M.) Same text, voice, parameters and seed give bit-identical audio on a given platform, at any thread count.

This post is about where it actually runs. All numbers are the Rust runtime with the fp16 weights, measured without real-time pacing (synthesis as fast as possible). One step = 80 ms of audio, real-time factor = step time / 80 ms. "First audio" = prime pass + first step.

devicethreadsper 80 ms stepRTFfirst audio
Apple M5 (macOS)13.3 ms4 %–
iPhone 13 Pro Max, A151 / 25.5 / 4.6 ms7 / 6 %85 / 51 ms
iPhone 11, A1316.7–7.1 ms8–9 %107–116 ms
Core i9-13900K, AVX21 / 47.0–7.4 / 2.5 ms9 / 3 %75–103 / 25 ms
Snapdragon 662 (Redmi 9T), Cortex-A731 / 442.4 / 14.3 ms53 / 18 %≈250 ms (sample app, warm, 4 threads)
Raspberry Pi Zero 2 W, Cortex-A53 @ 1 GHz1 / 4158 / 59–60 ms198 / 75 %–

Things we learned on the way:

  • The A-series phones do not want a second thread. A step is 5–7 ms and splits into ~80 tiny regions, so waking a worker on another performance core costs about as much as the work (A13: 2 threads 11.4 ms vs 6.7 ms, 4 threads 47 ms; A15 gains only 5.5 → 4.6 ms). The default on iOS is one thread. The M5 is different: two threads take it from 3.3 to 1.9 ms.

  • Android scheduling puts the synthesis on the small cores unless you pin it. On the Snapdragon 662 the sample app measured 22–33 ms per step before pinning the pool to the big cluster and 16–18 ms after (14.3 ms in a bare harness).

  • Pacing to playback makes the phone power manager lift the step time 3–4×. That is not the runtime getting slower; it is the CPU clocking down between steps. Measure unpaced, then keep a deep playback queue on weak devices (the bindings' players default to 8 s).

  • Pi Zero 2 W is the floor: four A53 cores at 1 GHz give 59–60 ms per 80 ms step, real time with a small margin, 160 MB peak RSS. (The synth example prints 88 % there because it also counts the prime pass over a 7 s sentence.) A Pi 4 or 5 has several times the margin.

  • The output is bit-identical per platform but differs across architectures below 2e-3 absolute (SNR above 60 dB, inaudible), so a golden-file test should be pinned per target.

Versus other small CPU models on one desktop core (same machine, each through its Python package, same three English texts, sentence-level RTF incl. text processing; method and script in docs/benchmarks.md): Linnet 10.6 % / 187 MB RSS, cloning, streams every 80 ms; pocket-tts (default 100M model; also clones and streams every 80 ms) 23.8 % on two cores / 968 MB; Kokoro-82M 51.7 % / 1.2 GB, whole utterance per call, but 8 languages; Piper medium 5.8 % / 503 MB, no cloning, streams per sentence. Speed only; quality was not measured there.

Licensing, upfront: the code is Apache-2.0. The weights are under our own Linnet Weights License: free to use, unmodified, inside your own products, commercial included, on device or on your own server. Custom voice creation (the encoder is not published), fine-tuning, saving quantised or converted copies of the weights (quantising in memory at load time is fine), training models on the weights or their outputs, and running a general-purpose TTS API need a separate license. So the model is not open source, and the README says so.

Honest limits: English only; no text normalisation (write numbers as words); occasional misreads on dense raw text; a slight roughness audible on headphones in some utterances; one session is capped at 24 s by default.

Try it: browser demo on https://voxrt.com (a WASM SIMD128 build of the same runtime, runs on your CPU, downloads the 83 MB weights once; the browser SDK itself is not released yet, which is why the README says it is deferred), Hugging Face Space https://huggingface.co/spaces/VoxRT/linnet-tts Code, bindings (Linux C, Python pip install linnet-tts on Linux and Apple-silicon macOS, Android AAR, iOS SwiftPM) and the benchmark script: https://github.com/VoxRT/LinnetTTS Weights and model card: https://huggingface.co/VoxRT/linnet-en-35m

Happy to answer questions on the runtime, the kernels or the mobile numbers. If you run it on a device that is not in the table, post your step time and I will add it to the README.


r/speechtech • • Oct 14 '26

VoxLock voice isolation

Post image
40 Upvotes

r/speechtech • • Sep 25 '26

Technology I compared diarization models on 15 clinical conversations: Nemotron 3, Pyannote, Sortformer, VibeVoice and Muse

Thumbnail
gallery
40 Upvotes

I've been working on clinical speaker attribution at Omi and wanted to compare the current diarization models on the same audio.

I used 15 mock doctor–patient consultations from PriMock57, about 2.4 hours. Full recordings, automatic speaker counts, without telling the models there are two people.

Batch

Diarization error rate (DER), with ±250 ms boundary tolerance. Lower is better.

ModelDERMedian processing time
Pyannote Precision-32.891%18.9 s / recording (API)
Nemotron 34.803%0.688 s / recording
Pyannote Community-16.620%18.691 s / recording
Sortformer v16.778%3.869 s / recording
Sortformer v2.17.974%1.077 s / recording
VibeVoice-ASR8.233%123 s / recording
Meta Muse Voice Transcribe †13.042%92 s / request (API)

Local models ran on one NVIDIA L4. API times include round-trip overhead; VibeVoice-ASR also performs transcription.

† Muse used 20 separate clips because of its 10-minute request limit, so its result isn't a whole-recording comparison.

Pyannote Precision-3 had the lowest error. Nemotron came next and was the fastest local model.

Streaming

ModelDER
Pyannote live API3.959%
Nemotron 3 †4.971%
Sortformer v2.1 †6.958%
VibeVoice 1.5B17.210%
VibeVoice 7B18.032%

† Native streaming presets evaluated through unpaced, completed-file replay. The other rows use paced, delivered speaker outputs. These scores don't establish live latency. I didn't evaluate Muse for streaming.

Same weights, different runtime

I also tried optimizing Nemotron and Community-1 with our proprietary runtime, without changing the weights:

  • Nemotron: 4.803% → 3.174% DER. 34% lower error, 2.13× faster.

  • Community-1: 6.620% → 5.435% DER. 18% lower error, 24× faster.

With zero boundary tolerance, Nemotron's runtime result gets slightly worse: 12.720% → 13.203%. Both scores are published. It's a small set with VAD-refined references, and we developed the runtime settings on it.

Audio, references, scorer, saved outputs and NVIDIA baseline runners are public. Our runtime code stays private, but its outputs are included for rescoring.

Repo and write-up in the comments. Any other diarization models worth adding?


r/speechtech • • Sep 23 '26

Technology NVIDIA Releases Nemotron 3 Diarization: A 100M-Parameter Open-Weight Model That Tracks 8 Speakers in Real Time

Post image
39 Upvotes

r/speechtech • • 10d ago

Technology Blind test of 32 ElevenLabs Voice Library voices for audiobooks (125 paid listeners, 1,122 judgments): "professional" voices no better than instant clones, popularity predicts nothing, louder voices win

35 Upvotes

We needed a defensible way to pick narrator voices from the ElevenLabs Voice Library (5,000+ English narration voices, no quality signal), so we ran a paid blind test.

Setup: 32 voices stratified across gender, age, accent, tier (professional vs instant clone) and usage; the same 30-second first-person nonfiction passage; identical eleven_v4 settings for every voice; 128 pairs in a connected circulant design; 125 paid US listeners, 1,122 forced-choice judgments; no listener heard any voice more than twice. Bradley-Terry ranking with listener bootstrap, plus a logistic property model on the voice tags.

Results:

  • Marie - Audiobook Narrator and Measured finished first (66% of pairings won, 95% CI 55–76). Alder - Warm, Raspy and Comforting last (24%). The top eight overlap and can't be ordered.

  • The "professional" badge told listeners nothing. Studio-recorded PVC voices won 46% of 649 head-to-head pairs against instant clones.

  • Popularity predicts nothing. Usage from under 1M to nearly 1B characters/year; coefficient zero.

  • Loudness matters. Voices ship up to 15 dB apart; we deliberately did not normalise. The louder clip won 54% of pairs; the quietest voice finished last.

  • "Old"-tagged voices lose, mostly female ones: young female voices won 58% of pairings, old female 43%. Male voices barely move with age.

  • Gender itself: 49/51.

All 32 clips are playable on the page; the per-judgment CSV is published (https://woodstitch.com/research/data/study004_judgments.csv). Chart of all 32: https://woodstitch.com/research/og/study004.png

Full report with method and limitations (one passage, one setting; sample raised from 3 to 9 judgments per pair after a first look, stated as a data-dependent continuation): https://woodstitch.com/research/which-elevenlabs-voice-is-best-for-audiobooks

Disclosure: we run Woodstitch, the paid listening-test platform this was run on. No relationship with ElevenLabs or any voice creator. HN thread: https://news.ycombinator.com/item?id=50458929


r/speechtech • • Jul 14 '25

I’m an ex-Googler — we built an AI voice agent that answers calls, books leads, and fixes a huge gap in service businesses

35 Upvotes

I used to lead Google Ads and AI projects during my 9+ years at Google. After leaving, I started a performance agency focused on law firms and home service businesses.

We were crushing it on the lead gen side but clients were still losing money because no one was answering the phone.

That pain led us to build Donna: an out-of-the-box AI voice assistant that picks up every call, handles intake, books appointments, and even processes cancellations. No call centers. No missed leads. It just works.

https://donnaio.ai/industry/home-service

Some early lessons from 1000+ calls: • Most leads are lost after the ad click • After-hours responsiveness = major revenue unlock • AI voice can work extremely well when it’s vertical-specific • SMBs don’t need dashboards they need outcomes

Curious if anyone else here has tackled this “lead leakage” problem, or is building similar vertical AI tools.


r/speechtech • • 3d ago

Kokoro Voice Mixer

Post image
34 Upvotes

I’ve created a little tool to play with different Kokoro mixes. In Kokoro TTS, you can blend different voice weights together to create new voices. We use Kokoro to play guided meditations, so I made this little “Kokoro Voice Mixer” where you can load the voice mixes we use in the app, try them out, and customize them by changing the mix.


r/speechtech • • Oct 22 '26

AuroraTTS-100K releases!

Thumbnail
huggingface.co
34 Upvotes

It is an english text to speech model, which achieves mostly understandable speech in under 100K parameters.

However, during more difficult words, or long numbers, it can slur them, and be not understandable.

Quantized to INT8 it only takes up 127KB.

72x real time on a CPU

1120x real time on a GPU

Check out the model card for more info or to try it out yourself!

Questions, feedback, or criticism is welcome!


r/speechtech • • Aug 30 '26

Extremely Fast and Accurate Transcription with Granite Speech 5.0 Turbo CTC

Thumbnail
huggingface.co
34 Upvotes

r/speechtech • • Apr 02 '26

OmniVoice: High-Quality Voice Cloning TTS for 600+ Languages

Thumbnail
github.com
30 Upvotes

r/speechtech • • Oct 20 '26

Phonon is now open-source, 31M and 40M TTS models designed to run on the edge

29 Upvotes

r/speechtech • • Mar 02 '26

I built a CPU-only speaker diarization library: it is ~7× faster than pyannote with comparable DER

29 Upvotes

Hi all,

I'd like to share a technical write-up about diarize - an open-source speaker diarization library I’ve been working on and released last weekend. (honestly, I hope you had more fun this weekend than I did).

diarize is focused specifically on CPU-only performance.

https://github.com/FoxNoseTech/diarize - Code (Apache 2.0)

https://foxnosetech.github.io/diarize/ - docs

Benchmark setup

  • Dataset: VoxConverse (216 recordings, 1–20 speakers)

  • Hardware: Apple M2 Max

  • CPU only, models preloaded (warm start)

  • Same evaluation protocol for both systems

Results

  • DER (VoxConverse):

    • This library: ~10.8%

    • pyannote (free models): ~11.2%

  • Speed (RTF):

    • This library: 0.12 (~8× faster than real time)

    • pyannote (free models): 0.86

  • 10-minute recording:

    • ~1.2 min vs ~8.6 min (pyannote)

Speaker count estimation accuracy (VoxConverse)

  • 1–5 speakers: 87–97% within ±1

  • Degrades significantly for 8+ speakers (tends to underestimate)

Pipeline

  • VAD: Silero VAD

  • Speaker embeddings: WeSpeaker ResNet34 (256-dim, ONNX Runtime)

  • Speaker count estimation:

    • fast single-speaker check

    • GMM + BIC model selection

    • local refinement around the selected hypothesis

  • Clustering: spectral clustering

  • Post-processing: short-segment reassignment, temporal merging

Limitations

  • No overlap handling (single speaker per frame)

  • Short segments (<0.4s) don’t get embeddings

  • Speaker count estimation is the main weak point for large groups

I also published a full article on Medium where I described full methodology & benchmarks.

I would appreciate any feedback, stars on GH and I hope it will be helpful for anyone.


r/speechtech • • Oct 08 '26

Neuphonic releases Neudecide, JEV for speech

Thumbnail
neuphonic.com
28 Upvotes

r/speechtech • • Jun 14 '26

Technology Diarization should be benchmarked separately from transcription accuracy

27 Upvotes

I think a lot of STT comparisons mix two different things:

  1. Did it transcribe words correctly?

  2. Did it know who said what?

For meeting notes, sales calls, interviews, support calls, etc, the second one can destroy the whole product even if the transcript is decent.

I've seen transcripts where the text is mostly fine, but the speaker labels drift after 20 minutes. Then the summary says the wrong person promised something. That is worse than a few missed filler words.

What I’d test separately:

  • speaker count detection

  • speaker swaps

  • late speaker joining

  • overlapping speech

  • two similar voices

  • short interjections

  • diarization speed

  • diarization + transcript alignment

  • whether it works live or only async

Tools I’d include in a test:

Deepgram, AssemblyAI, ElevenLabs Scribe, Speechmatics, pyannote, Gladia, Soniox, Smallest AI Pulse.

Smallest includes diarization in its STT feature set, so I’d be curious how it behaves on real multi-speaker calls. But I would not trust any vendor here without my own audio.

Anyone here running diarization in production? What failed first?


r/speechtech • • Jan 07 '26 Bravo

LFM2.5 Audio LLM released

Thumbnail
huggingface.co
27 Upvotes

LFM2.5-Audio is an end-to-end multimodal speech and text language model, and as such does not require separate ASR and TTS components. Designed with low latency and real time conversation in mind, at only 1.5 billion parameters LFM2.5-Audio enables seamless conversational interaction, achieving capabilities on par with much larger models. Our model consists of a pretrained LFM2.5 model as its multimodal backbone, along with a FastConformer based audio encoder to handle continuous audio inputs, and an RQ-transformer generating discrete tokens coupled with a lightweight audio detokenizer for audio output.