Abstract

Automatic syllable stress detection is an essential component in Computer assisted language learning (CALL) systems to guide non-native language learners. In English, each word typically contains only one primary stressed syllable. However, standard loss functions, such as Binary Cross-Entropy (BCE), often result in predictions where multiple syllables may be stressed or none at all. As a result, automatic syllable stress detection models frequently require an additional post-processing step to ensure that only one syllable is stressed per word. This reliance on post-processing suggests that the model is not fully capturing the stress patterns accurately. To address this issue, we propose an adaptive weighted loss function that builds upon the Stress Intensity Modulation Loss proposed in our recent work of Post-Net. This adaptive weighted loss function is designed to enforce the constraint of a single primary stressed syllable directly during model training. We integrate this loss function into the previously proposed Post-Net (PN_DNN) and on a new architecture which is a hybrid of Post-Net and LSTM (PN_DLSTM). Their performance is compared against the state-of-the-art models trained with standard BCE loss. Experiments conducted on the ISLE corpus reveal that both the models trained only with BCE loss show a significant accuracy gap between with and without post-processing. In contrast, when these models are trained on the proposed adaptive weighted loss function, the gap is narrowed in both the models. Between the two models, the highest reduction is observed in PN_DNN with a decrease from 3.87% to 2.45% & 4.6% to 3.75% for GER & ITA respectively. This indicates that the adaptive weighted loss function effectively captures the linguistic constraint during training, reducing the need for post-processing.

Abstract

Automatic syllable stress detection is a crucial component in Computer-Assisted Language Learning (CALL) systems for language learners. Current stress detection models are typically trained on clean speech, which may not be robust in real-world scenarios where background noise is prevalent. To address this, speech enhancement (SE) models, designed to enhance speech by removing noise, might be employed, but their impact on preserving syllable stress patterns is not well studied. This study examines how different SE models, representing discriminative and generative modeling approaches, affect syllable stress detection under noisy conditions. We assess these models by applying them to speech data with varying signal-to-noise ratios (SNRs) from 0 to 20 dB, and evaluating their effectiveness in maintaining stress patterns. Additionally, we explore different feature sets to determine which ones are most effective for capturing stress patterns amidst noise. To further understand the impact of SE models, a human-based perceptual study is conducted to compare the perceived stress patterns in SE-enhanced speech with those in clean speech, providing insights into how well these models preserve syllable stress as perceived by listeners. Experiments are performed on English speech data from non-native speakers of German and Italian. And the results reveal that the stress detection performance is robust with the generative SE models when heuristic features are used. Also, the observations from the perceptual study are consistent with the stress detection outcomes under all SE models.

Abstract

Automatic detection of prominence at the word and syllable-levels is critical for building computer-assisted language learning systems. It has been shown that prosody embeddings learned by the current state-of-the-art (SOTA) text-to-speech (TTS) systems could generate word- and syllable-level prominence in the synthesized speech as natural as in native speech. To understand the effectiveness of prosody embeddings from TTS for prominence detection under nonnative context, a comparative analysis is conducted on the embeddings extracted from native and non-native speech considering the prominence-related embeddings: duration, energy, and pitch from a SOTA TTS named FastSpeech2. These embeddings are extracted under two conditions considering: 1) only text, 2) both speech and text. For the first condition, the embeddings are extracted directly from the TTS inference mode, whereas for the second condition, we propose to extract from the TTS under training mode. Experiments are conducted on native speech corpus: Tatoeba, and non-native speech corpus: ISLE. For experimentation, word-level prominence locations are manually annotated for both corpora. The highest relative improvement on word & syllable-level prominence detection accuracies with the TTS embeddings are found to be 13.7% & 5.9% and 16.2% & 6.9% compared to those with the heuristic-based features and self-supervised Wav2Vec-2.0 representations, respectively.

Abstract

The language acquisition skills are predominant during childhood. Due to the lack of resources, a gap often arises in the language communication abilities of students. To bridge this gap, it is necessary to consider automatic methods for improving the children language skills. Our IIITH Ucchar e-Sudharak web based tool, deployed in a real environment, helps schoolgoing children to practice English reading skills with a teacher in the loop. The choice of English as the target language is strategic, given its status as a global lingua franca. Within the tool, children are provided with class and chapter-specific English sentences for practice tailored by their teachers. During practice sessions, the tool provides real-time feedback in the form of sentence and word-level scores, which are computed by comparing against teacher (expert) audio recordings. These scores offer insights into word-level pronunciation correctness and sentence-level quality by measuring fluency. Moreover, the tool provides a key feature where teachers can assign specific sentences to individual students and assess their performance. This functionality empowers teachers to personalize their instruction, catering to the specific needs of each student. As far as we know, this is a one-of-a-kind tool specifically designed to meet the needs of school-going children.

Abstract

The people with Major Depressive Disorder (MDD) exhibit the symptoms of tonal variations in their speech compared to the healthy counterparts. However, these tonal variations not only confine to the state of MDD but also on the language, which has unique tonal patterns. This work analyzes automatic speech-based depression detection across two languages, English and Malayalam, which exhibits distinctive prosodic and phonemic characteristics. We propose an approach that utilizes speech data collected along with self-reported labels from participants reading sentences from IViE corpus, in both English and Malayalam. The IViE corpus consists of five sets of sentences: simple sentences, WH-questions, questions without morphosyntactic markers, inversion questions and coordinations, that can naturally prompt speakers to speak in different tonal patterns. Convolutional Neural Networks (CNNs) are employed for detecting depression from speech. The CNN model is trained to identify acoustic features associated with depression in speech, focusing on both languages. The model's performance is evaluated on the collected dataset containing recordings from both depressed and non-depressed speakers, analyzing its effectiveness in detecting depression across the two languages. Our findings and collected data could contribute to the development of language-agnostic speech-based depression detection systems, thereby enhancing accessibility for diverse populations.

Abstract

Mental health conditions such as depression and anxiety are pervasive issues that affect millions of individuals worldwide. Understanding the intricate relationship between emotional perception and mental health is crucial. This work develops a database named IIITSaint-EmoMDB containing Malayalam language speech samples annotated with emotion ratings at valence and arousal scale using three annotators considering four emotion labels: happy, sad, angry and neutral. In addition to emotion ratings, emotional perception is rated from 150 participants according to the circumplex model of va-lence and arousal. The database also consists of self-reported mental health scores of those participants based on PHQ-9 and GAD-7 questionnaire. Our preliminary analysis of comparing the participants' emotional perception based PHQ-9 and GAD-7 scores revealed that the depressed category showed the highest improvement.

Abstract

Depression, a pervasive mental health issue, highlights the critical need for early detection and effective intervention. A database (InStant-EMDB) is developed to analyze varia tions in the spoken responses of individuals with depression compared to healthy counterparts. Spoken responses are ob tained from English and Malayalam bilingual speakers who respond spontaneously to a set of 15 emotionally evocative words. These words are sourced from the Affective Norms for English Words (ANEW) dataset, which includes valence and arousal ratings for each word. The speech in both English and Malayalam is manually transcribed to accurately reflect the spoken content. The dataset also contains self-reported af fective ratings and data from a mental health survey (PHQ-9) collected from the participants to determine their mental state, which we considered the self-reported depression labels. A preliminary analysis is conducted on the collected speech us ing current state-of-the-art deep learning models such as Con volutional Neural Networks (CNN), Long Short-Term Mem ory networks (LSTM) and Bi-directional Long Short-Term Memory networks (Bi-LSTM). Although distinct linguistic patterns exhibited by individuals struggling with depression are successfully identified by all three models in both Malay alam and English spoken responses, the highest accuracy is achieved by the LSTM models. Our findings and dataset em phasize the potential of linguistic patterns as valuable cues for the early identification and intervention of depression and could contribute to enhancing accessibility for diverse popu lations.

Abstract

The demand for high-quality parallel speech data has been increasing as deep learning-based Speech to Speech Machine Translation (SSMT) and automatic dubbing approaches gain popularity in speech applications. Traditional and well-established speech applications such as Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) heavily rely on large corpus of monolingual speech and the corresponding text. While there is a wealth of parallel text data available for both English and Indic languages, parallel speech data is available only for English and other European languages, yet it often lacks natural prosody and semantic alignment between the languages. For achieving cross-lingual prosody transfer, end-to-end SSMT models, and high-quality dubbing from English to Hindi, in this work, an English-Hindi parallel bilingual speech-text corpus named lIlT-Speech Twins 1.0 is created. This data contains twin-like English and Hindi speech-text pairs obtained from publicly available children's stories in both the languages, through manual and automatic processing. Starting with 8 stories in each language, totaling around 4 hours of audio, the final outcome was a 2-hour dataset. This was achieved through systematic segmentation, re-moval of non-speech background audio, and sentence-by-sentence alignment to ensure accurate meaning in both languages. In addition to ensuring proper alignment and transcription, this dataset offers a rich source of natural prosody, expressions, and emotions, due to the narrative diversity within the stories. The dataset also provides sig-nificant speaker variability, with different characters being voiced by various speakers, enhancing the richness of the lIlT-Speech Twins 1.0 corpus.

Abstract

Automatic syllable stress detection methods typically consider syllable-level features as independent. However, as per linguistic studies, there is a dependency among the syllables within a word. In this work, we address this issue by proposing a Post-Net approach using Time-Delay Neural Networks to exploit the syllable dependency in a word for stress detection task. For this, we propose a loss function to incorporate the dependency by ensuring only one stressed syllable in a word. The proposed Post-Net leverages the existing SOTA sequence-independent stress detection models and learns in both supervised and unsupervised settings. We compare the Post-Net with three existing SOTA sequence-independent models and also with sequential model (LSTMs). Experiments conducted on ISLE corpus show the highest relative accuracy improvement of 2.1% and 20.28% with the proposed Post-Net compared to the best sequenceindependent SOTA model in supervised and unsupervised manners, respectively.

Abstract

Automatic syllable stress detection is typically operated at syllable level with stress-related acoustic features. The stress placed on a syllable is influenced not only by its own characteristics but also by its context in the word. However, traditional methods for stress detection overlook the contextual acoustic factors that influence stress placement. By addressing this issue, we study sequential modeling approaches by integrating the syllable dependency for automatic syllable stress detection using a masking strategy. This approach considers a sequence of syllables at the word level and identifies its stress label sequence. We explore various sequential models, such as RNNs, LSTMs, GRUs, and Attention networks. We conduct experiments on the ISLE corpus comprising non-native speakers speaking English. From the experiments, we observe a significant improvement in the performance with all sequential models compared to the state-of-the-art non-sequential baseline (DNN).