Fade Aligner

Listen to every phoneme

Pick a clip, then click any phoneme or word to hear exactly that segment. The boundaries are the aligner's output, drawn at their exact positions on the spectrogram.

Demo clips are added at release. Come back soon, or run tools/make_demo.py on your own alignments to build this viewer.

How it works

A CTC acoustic model on mel spectrograms is refined with Viterbi training rounds, then decoded with an HMM that uses label priors to give every phone a duration.

Figure 1. Training and decoding pipeline of Fade Aligner.

GPU Viterbi

Alignment runs as batched Viterbi on the GPU, so large corpora are labelled quickly.

GPU mel spectrograms

Features are computed on the GPU during training and alignment, with no feature cache to build.

Any espeak-ng language

Text is phonemized with espeak-ng, so any language it supports can be trained on your own data.

TextGrid and JSON

Export per-token durations as JSON or Praat TextGrids for annotation and TTS.

Built on Stylish-TTS (MIT) with CTC label priors from Huang et al., ICASSP 2024.

Quick start

Install, prepare a folder of audio with transcripts, train, align and export.

Install

git clone https://github.com/FashionFlora/fade-aligner && cd fade-aligner
pip install -r requirements.txt
sudo apt install espeak-ng

Prepare data

python tools/prepare_data.py --format folder --lang en-us --out data/my_corpus ...

Train

python aligner/stylish_aligner_train.py --epochs 100 --out runs/my_aligner ...

Align

python aligner/stylish_aligner_align.py \
    --model-path runs/my_aligner/model.pth \
    --decode hmm --min-dur 3 --hmm-prior-scale 1.0 \
    --output-dir alignments/ ...

Export TextGrid

python tools/export_alignments.py --format textgrid --output-dir alignments/textgrid ...

The dots stand for the input paths; see --help of each script and the README for the full set of options.

Output format

Alignments are computed on 10 ms frames. Every token of the phoneme string gets one duration, a whole number of frames, and the durations add up to the length of the audio. The symbol _ is a silence or word boundary. With --min-dur 3 every token lasts at least 3 frames (30 ms).

JSON

{
  "text": "Hello world.",
  "ipa": "həlˈoʊ wˈɜːld.",
  "duration_s": 1.12,
  "tokens": [
    ["_", 0.00, 0.13],
    ["h", 0.13, 0.18],
    ["ə", 0.18, 0.22],
    ["l", 0.22, 0.30],
    ...
  ]
}

TextGrid

File type = "ooTextFile"
Object class = "TextGrid"
xmin = 0
xmax = 1.12
tiers? <exists>
size = 2
item []:
  item [1]:
    class = "IntervalTier"
    name = "words"
  item [2]:
    class = "IntervalTier"
    name = "phones"
    intervals [2]:
      xmin = 0.13
      xmax = 0.18
      text = "h"

The TextGrid above is abbreviated. Words are the runs of tokens between _ tokens.

Limitations

The demo model is trained on English read speech only. Other languages need training on your own data.
Time resolution is one 10 ms frame, and very short phones are held to the minimum duration you set.
Text is phonemized by espeak-ng. Errors in the phonemization, such as a wrong reading of a name or number, carry over into the alignment.
Noisy recordings, music, overlapping speakers and transcripts that do not match the audio can degrade the result. Check the score and listen to a sample of the output.
Boundaries inside fast, coarticulated speech are inherently fuzzy, so expect disagreement with manual annotation on the order of a frame or two.

Credits & license

Code: MIT. Built on Stylish-TTS (MIT) and StyleTTS 2, with k2 for decoding and espeak-ng for phonemization. CTC label priors follow Huang et al., ICASSP 2024.

The demo model is trained on LibriTTS-R (Koizumi et al., CC BY 4.0), English. Please attribute it when you reuse the demo model or its outputs.