PEASYV

From videos to TextGrids

Adrien Méli and Nicolas Ballier

2024-03-28

Basic concept

Phonetic

Extraction and

Alignment of

Subtitled

YouTube

Videos

flowchart LR
  A(Link to video) --> B(Praat TextGrid)

Today I’m going to speak about PEASYV.

PEASYV stands for…

The basic concept was originally very simple: to be able to collect data from videos with subtitles, most of which, at the time when I began working on the project, could be found on YouTube.

Now with Artificial Intelligence taking the world by storm, it has become easy to generate subtitles from any recording, a case in point being of course OpenAI’s Whisper.

Please note that PEASYV does work with AI-generated subtitles, as you will see.

Key tools

• #! /bin/sh

• SPPAS (Bigi 2012; Bigi and Hirst 2012)

• P2FA (Yuan and Liberman 2008)

• PRAAT (Boersma and Weenink 2019)

• R (R Core Team 2023)

• yt-dlp (yt-dlp 2022)

• ffmpeg (Developers 2021)

• the Longman Pronunciation Dictionary (Wells 2008)

As noted with the shebang on top of the list here, PEASYV is fundamentally a shell script, easily deployable on any Linux distribution. It relies on the following softwares:

• two python-based forced aligners. These are also fundamentally wrappers for two speech recognition engines, Julius for SPPAS, and HTK for P2FA.

• the almighty and omniscient PRAAT, which probably bears no introduction here.

• the not less infamous R, the statistical software

• yt-dlp is a python-based application to download YouTube videos along with their metadata

• ffmpeg is the Swiss Army knife of media files manipulations, a command line tool to convert transcode decode encode all media files

• and finally the LPD, the digital version of which has enabled me to align the recordings at the syllabic level

Detailed flowchart

flowchart LR

subgraph S1[Step 1]
direction TB
A(fa:fa-file-lines list of links) --> |yt-dlp| B(fa:fa-video video)
A --> |yt-dlp| C(fa:fa-closed-captioning subtitles)
B -->|ffmpeg| D(fa:fa-file-audio fa:fa-regular far:fa-square Main Audio TG)
C -->|praat| D
D -->|praat| E1(fa:fa-file-audio far:fa-square)
D -->|praat| E2(fa:fa-file-audio far:fa-square)
D -->|praat| E3(fa:fa-file-audio far:fa-square)
end

subgraph S2[Step 2]
direction LR
F1(fa:fa-file-audio far:fa-square) -->|SPPAS| G1(fa:fa-table-cells-large Segm TG)
F2(fa:fa-file-audio far:fa-square) -->|SPPAS| G2(fa:fa-table-cells-large Segm TG)
F3(fa:fa-file-audio far:fa-square) -->|SPPAS| G3(fa:fa-table-cells-large Segm TG)
F1(fa:fa-file-audio far:fa-square) -->|fa:fa-align P2FA| H1(fa:fa-table-cells-large Segm TG)
F2(fa:fa-file-audio far:fa-square) -->|fa:fa-align P2FA| H2(fa:fa-table-cells-large Segm TG)
F3(fa:fa-file-audio far:fa-square) -->|fa:fa-align P2FA| H3(fa:fa-table-cells-large Segm TG)
G1--> GH(fa:fa-file-audio fa:fa-table-cells-large)
H1--> GH
G2--> GH
H2--> GH
G3--> GH
H3--> GH
end

subgraph S3[Step 3]
direction TB
I(fa:fa-file-audio fa:fa-table-cells-large) -->|praat| K(fa:fa-table-cells Segm Syll TG)
J(fa:fa-book LPD) -->|praat|K
K -->|R| L(fa:fa-file-csv spreadsheets)
L -->|R| M(fa:fa-chart-line vocalic diagnoses)

end

S1 --> S2
S2 --> S3

Let us now have a look at the workflow in detail.

First, a simple text file containing the links to the YouTube videos is created, one link per line. Gender and variety of English can be added at the end of each line.

PEASYV uses that text file to download the videos, their subtitles and their metadata using yt-dlp. A folder named after the videos is created for each link.

The video is converted to a sound file with ffmpeg, and the timestamps in the subtitles are used to create intervals on a tier in a first main TextGrid in Praat. The tier features the content of the subtitles at the corresponding interval.

Then the main recording and the main TextGrid are split into subfiles along the intervals created from the timestamps in the subtitles: the audio is extracted, along with its transcription in the interval.

Let us now have a look at step 2.

Here the subfiles are fed into SPPAS first, then P2FA. Each aligner returns TextGrids aligned at the lexical and segmental level. SPPAS also returns MOMEL and INTSINT tiers for analyzing intonation. All the sub-TextGrids are then merged together in one intermediary main TextGrid.

In step 3, a praat script taps into the LPD and adds syllabic tiers for the two alignments. Two other tiers are added, one for each aligner, in an attempt to measure the potential gaps between the alignments—more on this later.

Two optional outputs are also generated, spreadsheets and vocalic diagnoses.

About the two aligners

• SPPAS:
  • Julius (Lee and Kawahara 2019)
  • CMU (Weide 1994) in SAMPA
• P2FA:
  • htk (Young et al. 2006)
  • CMU in ArpaBet

Both SPPAS and P2FA resort to the Carnegie Mellon university dictionary, the CMU. However they use different transcription systems, i.e. SAMPA for SPPAS, and ArpaBet for P2FA. This makes establishing correspondences between the two aligners to compare their outcomes a non-trivial matter.

Their greatest difference is arguably the fact that they resort to two different speech recognition engines: Julius for SPPAS, and htk for P2FA…

Claims

• 2 aligners are better than just one

  • SPPAS
  • P2FA

• Step 2 prevents cascading alignment errors

• Added values:

  • low-tech
  • syllabic tiers based on the LPD (Wells 2008)
  • (MIS)MATCHES on the TextGrid

… and this fact probably PEASYV’s grandest claim:

• 2 aligners are better, as they use two different speech recognition engines.

• step 2 is a key element of how PEASYV was designed. I was, am, obsessed by the fear that the collected data might be inaccurate. By splitting the main recording into subfiles of a few seconds each, aligners only have to align small chunks of speech at a time. This decreases the likelihood of misalignments but also prevents potential “domino effects” of such misalignments, to the rest of the file.

• of, in my view, not uninteresting value are also the following points: PEASYV is low-tech, easy to install and easy to run.

it is unique in the syllabic alignments it offers, along with its built-in assessment of differences between the two aligners (I’m getting there, I promise! as we’re now going to have a look at the outputs PEASYV returns.

Outputs

the PRAAT TextGrid

The main piece of interest is the concatenated main TextGrid returned for each video.

Tiers

1. Transcription
2. Momel
3. INTSINT

4. SPPAS Word
5. SPPAS Phoneme
6. SPPAS LPD Word
7. SPPAS Syllable
8. SPPAS LPD Syllable

9. P2FA Word
10. P2FA Phoneme
11. P2FA LPD Word
12. P2FA Syllable
13. P2FA LPD Syllable

14. SPPAS MISMATCH
15. P2FA MISMATCH

Before showing you a screenshot of a final TextGrid, let me break down the fifteen tiers it features.

The first tier contains the intervals created from the timestamped subtitles.

Tiers 2 and 3 were aligned by SPPAS, and are dedicated to intonational analysis.

Tiers 4 through 8 were generated by SPPAS, and are mirrored by tiers 9 through 13, which were generated by P2FA.

Tiers 14 and 15 will be explained in the next slide.

Screenshot of a TextGrid

So this is a screenshot of a final TextGrid.

As you can see, the cursor is located at roughly the middle of the SPPAS-aligned syllable /@t/ in the word “clarity”.

If you trace the cursor down from the highlighted tier to five tiers below (the antepenultimate tier), you can see that at the time indicated by the cursor, P2FA has returned the last syllable of “clarity”. The second syllable, /@t/, can be found a few milliseconds earlier, on the left.

This is flagged on tier 14, called “SPPAS MISMATCH”, as a mismatch.

Should the reverse situation happen, ie the midtemporal value of a P2FA-aligned syllable not being comprised in the corresponding SPPAS-aligned syllabic interval, tier 15 will feature “MISMATCH” as well.

This procedure, I contend, makes it possible to identify potentially problematic passages, because they might be mis-aligned.

Secondary outputs

PEASYV generates optional outputs, which were first implemented in order (once again) to check that the collected data was accurate.

Spreadsheets

• .csv format
• one per aligner
• one row per vowel
• formant readings at each centile of a vowel’s duration
• = 300 + 24 columns

Columns and data

Diagnoses

An example: English Like A Native

SPPAS

Data aligned by SPPAS.

General information

Nb of TextGrids: 453

Total length of the videos: 172:39:22

Monophthongs

Data on monophthongs.

Distribution

Distribution of monophthongs.

Durations

Per-monophthong boxplots of durations.

Vocalic Trapezoids

Scatterplots

Per-monophthong mean F1/F2 values with error bars (1 SE)

Deterding

Dotted grey line: reported native values — Black line: speaker

References: Deterding (1997)

Hillenbrand

Dotted grey line: reported native values — Black line: speaker

References: Hillenbrand et al. (1995)

Density plots

F1

Per-monophthong F1 density plots

F2

Per-monophthong F2 density plots

Formant tracking

Next are the formant tracks for monophthongs.

F1

F2

Diphthongs

Data on diphthongs.

Distribution

Distribution of diphthongs

Durations

Boxplot of diphthongs durations

Vocalic Trapezoids

Diphthong ɪə

Vector of the mean F1/F2 values from 20% to 80% percent of the diphthong’s duration

Diphthong eɪ

Vector of the mean F1/F2 values from 20% to 80% percent of the diphthong’s duration

Diphthong aɪ

Vector of the mean F1/F2 values from 20% to 80% percent of the diphthong’s duration

Diphthong eə

Vector of the mean F1/F2 values from 20% to 80% percent of the diphthong’s duration

Diphthong əʊ

Vector of the mean F1/F2 values from 20% to 80% percent of the diphthong’s duration

Diphthong aʊ

Vector of the mean F1/F2 values from 20% to 80% percent of the diphthong’s duration

Diphthong ɔɪ

Vector of the mean F1/F2 values from 20% to 80% percent of the diphthong’s duration

Diphthong ʊə

Vector of the mean F1/F2 values from 20% to 80% percent of the diphthong’s duration

Overview

Overview of all diphthongs

Formant tracking

Next are the formant tracks for diphthongs.

F1

F2

P2F

Data aligned by P2FA.

Monophthongs

Data on monophthongs.

Distribution

Distribution of monophthongs.

Durations

Per-monophthong boxplots of durations.

Vocalic Trapezoids

Scatterplots

Per-monophthong mean F1/F2 values with error bars (1 SE)

Deterding

Dotted grey line: reported native values — Black line: speaker

References: Deterding (1997)

Hillenbrand

Dotted grey line: reported native values — Black line: speaker

References: Hillenbrand et al. (1995)

Density plots

F1

Per-monophthong F1 density plots

F2

Per-monophthong F2 density plots

Formant tracking

Next are the formant tracks for monophthongs.

F1

F2

Diphthongs

Data on diphthongs.

Distribution

Distribution of diphthongs

Durations

Boxplot of diphthongs durations

Vocalic Trapezoids

Diphthong ɪə

Vector of the mean F1/F2 values from 20% to 80% percent of the diphthong’s duration

Diphthong eɪ

Vector of the mean F1/F2 values from 20% to 80% percent of the diphthong’s duration

Diphthong aɪ

Vector of the mean F1/F2 values from 20% to 80% percent of the diphthong’s duration

Diphthong eə

Vector of the mean F1/F2 values from 20% to 80% percent of the diphthong’s duration

Diphthong əʊ

Vector of the mean F1/F2 values from 20% to 80% percent of the diphthong’s duration

Diphthong aʊ

Vector of the mean F1/F2 values from 20% to 80% percent of the diphthong’s duration

Diphthong ɔɪ

Vector of the mean F1/F2 values from 20% to 80% percent of the diphthong’s duration

Diphthong ʊə

Vector of the mean F1/F2 values from 20% to 80% percent of the diphthong’s duration

Overview

Overview of all diphthongs

Formant tracking

Next are the formant tracks for diphthongs.

F1

F2

Comparison between the two aligners

Per-diphthong

Let’s now compare the data obtained with the two aligners.

Diphthong ɪə

Diphthong eɪ

Diphthong aɪ

Diphthong eə

Diphthong əʊ

Diphthong aʊ

Diphthong ɔɪ

Diphthong ʊə

Overview

What next?

• make PEASYV installable

but potential issues with Wells (2008)

• create interactive website

• upload interactive diagnoses

• adrienmeli@gmail.com

References

Bigi, B. 2012. “SPPAS: a tool for the phonetic segmentations of Speech.” Istanbul.

Bigi, B., and D. Hirst. 2012. “SPeech Phonetization Alignment and Syllabification (SPPAS):a Tool for the Automatic Analysis of Speech Prosody.” Shanghai.

Boersma, Paul, and David Weenink. 2019. “Praat: Doing Phonetics by Computer [Computer Program]. Version 6.1.07, Retrieved 26 November 2019 from Http://Www.praat.org/.” 2019.

Deterding, David. 1997. “The Formants of Monophthong Vowels in Standard Southern British English Pronunciation.” Journal of the International Phonetic Association 27 (1-2): 47–55. https://doi.org/10.1017/s0025100300005417.

Developers, FFmpeg. 2021. “Ffmpeg Tool (Version Be1d324)[software].” URL: Http://Ffmpeg. Org.

Hillenbrand, J., L. A. Getty, M. J. Clark, and K. Wheeler. 1995. “Acoustic Characteristics of American English Vowels.” The Journal of the Acoustical Society of America 97 (5): 3099–3111.

Lee, A., and T. Kawahara. 2019. https://doi.org/10.5281/zenodo.2530395.

R Core Team. 2023. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/.

Weide, R. L. 1994. “CMU Pronouncing Dictionary.” http://www.speech.cs.cmu.edu/cgi-bin/cmudict.

Wells, J. C. 2008. Longman Pronunciation Dictionary. London: Pearson Longman.

Young, S. J., G. Evermann, M. J. F. Gales, T. Hain, D. Kershaw, G. Moore, J. Odell, et al. 2006. The HTK Book, Version 3.4. Cambridge, UK: Cambridge University Engineering Department.

yt-dlp, Developers. 2022. “Yt-Dlp.” GitHub Repository. https://github.com/yt-dlp/yt-dlp; GitHub.

Yuan, J., and M. Liberman. 2008. “Speaker Identification on the SCOTUS Corpus.” Journal of the Acoustical Society of America, 123(5): 5687.