r/speechtech • u/EdwardGlockner • 14h ago
Using Rumble-2-Flash's built-in CTC head for forced alignment instead of a separate aligner – some numbers
I've been building a transcription pipeline on Reson8's Rumble-2-Flash and tried using its CTC head for word timestamps. Haven't seen anyone write about it, so figured I'd share.
The checkpoint has a CTC head next to the decoder, just one linear layer on the encoder's ln_post output, and its classes are the decoder vocab plus a blank. Since the encoder already runs during generate(), you can hook ln_post, Viterbi the decoder's own tokens against the head and get word times without a second model or a second pass. On 20-60 min files that took 0.3-0.9 s per file, vs 7-17 s for WhisperX's alignment step on the same files.
The problem is the head is really peaky. 60-76% of frames are blank, a token is about 1.2 frames, and it's only ~13 frames/s. Starts are fine, but the last spike of a word says almost nothing about where it ends. I tried a few end rules against MFA alignments on LibriSpeech (picked on dev-clean, reported on test-clean):
end at the last spike: 115 ms MAE
end at the next word's start: 75 ms
next word's start, but cut at the first silence (30 ms, 25 dB below speech level): 57 ms
Vs WhisperX on the same 200 utterances: starts 58 vs 75 ms, ends 57 vs 34 ms. So better starts, worse ends, which makes sense with 13 vs 50 frames/s.
Also, padded batching changed the encoder output until I trimmed clips to whole mel hops, padded with a bit of silence and zeroed the masked frames. And my FLEURS Dutch WER was 9.2% vs 6.05% on the model card, which turned out to be almost all the normaliser. With the Open ASR Leaderboard's scoring the same transcripts give 6.24% (WhisperX large-v3 goes 7.05% -> 4.43%).
MFA isn't human labels btw, it's another forced aligner.
Code and raw results: https://github.com/EdwardGlockner/rumblex
Has anyone found a better way to get word ends out of a peaky CTC head? "Next start, cut at silence" works but feels hacky.
Edit: tried a blank penalty before Viterbi after a suggestion in the comments. Starts went 57 -> 52 ms on test-clean, ends unchanged.
