Abstract

The language acquisition skills are predominant during childhood. Due to the lack of resources, a gap often arises in the language communication abilities of students. To bridge this gap, it is necessary to consider automatic methods for improving the children language skills. Our IIITH Ucchar e-Sudharak web based tool, deployed in a real environment, helps schoolgoing children to practice English reading skills with a teacher in the loop. The choice of English as the target language is strategic, given its status as a global lingua franca. Within the tool, children are provided with class and chapter-specific English sentences for practice tailored by their teachers. During practice sessions, the tool provides real-time feedback in the form of sentence and word-level scores, which are computed by comparing against teacher (expert) audio recordings. These scores offer insights into word-level pronunciation correctness and sentence-level quality by measuring fluency. Moreover, the tool provides a key feature where teachers can assign specific sentences to individual students and assess their performance. This functionality empowers teachers to personalize their instruction, catering to the specific needs of each student. As far as we know, this is a one-of-a-kind tool specifically designed to meet the needs of school-going children.

Abstract

The language acquisition skills are predominant during childhood. Due to the lack of resources, a gap often arises in the language communication abilities of students. To bridge this gap, it is necessary to consider automatic methods for improving the children language skills. Our IIITH Ucchar e-Sudharak web based tool, deployed in a real environment, helps schoolgoing children to practice English reading skills with a teacher in the loop. The choice of English as the target language is strategic, given its status as a global lingua franca. Within the tool, children are provided with class and chapter-specific English sentences for practice tailored by their teachers. During practice sessions, the tool provides real-time feedback in the form of sentence and word-level scores, which are computed by comparing against teacher (expert) audio recordings. These scores offer insights into word-level pronunciation correctness and sentence-level quality by measuring fluency. Moreover, the tool provides a key feature where teachers can assign specific sentences to individual students and assess their performance. This functionality empowers teachers to personalize their instruction, catering to the specific needs of each student. As far as we know, this is a one-of-a-kind tool specifically designed to meet the needs of school-going children.

Abstract

The people with Major Depressive Disorder (MDD) exhibit the symptoms of tonal variations in their speech compared to the healthy counterparts. However, these tonal variations not only confine to the state of MDD but also on the language, which has unique tonal patterns. This work analyzes automatic speech-based depression detection across two languages, English and Malayalam, which exhibits distinctive prosodic and phonemic characteristics. We propose an approach that utilizes speech data collected along with self-reported labels from participants reading sentences from IViE corpus, in both English and Malayalam. The IViE corpus consists of five sets of sentences: simple sentences, WH-questions, questions without morphosyntactic markers, inversion questions and coordinations, that can naturally prompt speakers to speak in different tonal patterns. Convolutional Neural Networks (CNNs) are employed for detecting depression from speech. The CNN model is trained to identify acoustic features associated with depression in speech, focusing on both languages. The model's performance is evaluated on the collected dataset containing recordings from both depressed and non-depressed speakers, analyzing its effectiveness in detecting depression across the two languages. Our findings and collected data could contribute to the development of language-agnostic speech-based depression detection systems, thereby enhancing accessibility for diverse populations.

Abstract

Mental health conditions such as depression and anxiety are pervasive issues that affect millions of individuals worldwide. Understanding the intricate relationship between emotional perception and mental health is crucial. This work develops a database named IIITSaint-EmoMDB containing Malayalam language speech samples annotated with emotion ratings at valence and arousal scale using three annotators considering four emotion labels: happy, sad, angry and neutral. In addition to emotion ratings, emotional perception is rated from 150 participants according to the circumplex model of va-lence and arousal. The database also consists of self-reported mental health scores of those participants based on PHQ-9 and GAD-7 questionnaire. Our preliminary analysis of comparing the participants' emotional perception based PHQ-9 and GAD-7 scores revealed that the depressed category showed the highest improvement.

Abstract

Depression, a pervasive mental health issue, highlights the critical need for early detection and effective intervention. A database (InStant-EMDB) is developed to analyze varia tions in the spoken responses of individuals with depression compared to healthy counterparts. Spoken responses are ob tained from English and Malayalam bilingual speakers who respond spontaneously to a set of 15 emotionally evocative words. These words are sourced from the Affective Norms for English Words (ANEW) dataset, which includes valence and arousal ratings for each word. The speech in both English and Malayalam is manually transcribed to accurately reflect the spoken content. The dataset also contains self-reported af fective ratings and data from a mental health survey (PHQ-9) collected from the participants to determine their mental state, which we considered the self-reported depression labels. A preliminary analysis is conducted on the collected speech us ing current state-of-the-art deep learning models such as Con volutional Neural Networks (CNN), Long Short-Term Mem ory networks (LSTM) and Bi-directional Long Short-Term Memory networks (Bi-LSTM). Although distinct linguistic patterns exhibited by individuals struggling with depression are successfully identified by all three models in both Malay alam and English spoken responses, the highest accuracy is achieved by the LSTM models. Our findings and dataset em phasize the potential of linguistic patterns as valuable cues for the early identification and intervention of depression and could contribute to enhancing accessibility for diverse popu lations.

Abstract

The demand for high-quality parallel speech data has been increasing as deep learning-based Speech to Speech Machine Translation (SSMT) and automatic dubbing approaches gain popularity in speech applications. Traditional and well-established speech applications such as Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) heavily rely on large corpus of monolingual speech and the corresponding text. While there is a wealth of parallel text data available for both English and Indic languages, parallel speech data is available only for English and other European languages, yet it often lacks natural prosody and semantic alignment between the languages. For achieving cross-lingual prosody transfer, end-to-end SSMT models, and high-quality dubbing from English to Hindi, in this work, an English-Hindi parallel bilingual speech-text corpus named lIlT-Speech Twins 1.0 is created. This data contains twin-like English and Hindi speech-text pairs obtained from publicly available children's stories in both the languages, through manual and automatic processing. Starting with 8 stories in each language, totaling around 4 hours of audio, the final outcome was a 2-hour dataset. This was achieved through systematic segmentation, re-moval of non-speech background audio, and sentence-by-sentence alignment to ensure accurate meaning in both languages. In addition to ensuring proper alignment and transcription, this dataset offers a rich source of natural prosody, expressions, and emotions, due to the narrative diversity within the stories. The dataset also provides sig-nificant speaker variability, with different characters being voiced by various speakers, enhancing the richness of the lIlT-Speech Twins 1.0 corpus.

Abstract

Automatic syllable stress detection methods typically consider syllable-level features as independent. However, as per linguistic studies, there is a dependency among the syllables within a word. In this work, we address this issue by proposing a Post-Net approach using Time-Delay Neural Networks to exploit the syllable dependency in a word for stress detection task. For this, we propose a loss function to incorporate the dependency by ensuring only one stressed syllable in a word. The proposed Post-Net leverages the existing SOTA sequence-independent stress detection models and learns in both supervised and unsupervised settings. We compare the Post-Net with three existing SOTA sequence-independent models and also with sequential model (LSTMs). Experiments conducted on ISLE corpus show the highest relative accuracy improvement of 2.1% and 20.28% with the proposed Post-Net compared to the best sequenceindependent SOTA model in supervised and unsupervised manners, respectively.

Abstract

Automatic syllable stress detection is typically operated at syllable level with stress-related acoustic features. The stress placed on a syllable is influenced not only by its own characteristics but also by its context in the word. However, traditional methods for stress detection overlook the contextual acoustic factors that influence stress placement. By addressing this issue, we study sequential modeling approaches by integrating the syllable dependency for automatic syllable stress detection using a masking strategy. This approach considers a sequence of syllables at the word level and identifies its stress label sequence. We explore various sequential models, such as RNNs, LSTMs, GRUs, and Attention networks. We conduct experiments on the ISLE corpus comprising non-native speakers speaking English. From the experiments, we observe a significant improvement in the performance with all sequential models compared to the state-of-the-art non-sequential baseline (DNN).

Abstract

The task of automatically detecting syllable stress is a key module in computer-assisted language learning systems. There are numerous studies proposed in the literature for automatic syllable stress detection by using different knowledge-based prosodic features. Also, different statistical machine learning and deep learning models are explored for this task using knowledge-based features. However, the acoustic parameters considered to compute knowledge-based features might not always represent the stress phenomena, hence the knowledge based features are not always suitable for generalization and scalability. Recently, the rapidly emerging self-supervised learning based representations are outperforming the existing state-of-the-art knowledge-based features in all speech applications. Also, these representations allow the models to be built in an end-to-end fashion. In this work, we explore the use of self-supervised representations (Wav2Vec-2.0), for syllable stress detection and compare the performance with state-of-the-art knowledge-based features. Further, we use our recently proposed explicit representation learning framework, modeled by jointly optimizing variational autoencoder (VAE) and DNN for stress detection. We analyze the performance of representation learning framework with two different state-of-the-art classifiers, support vector machines (SVM) and simple deep neural network (DNN). We conduct experiments on two non-native English speakers’ datasets from ISLE corpus i.e., German (GER), and Italian (ITA). From the analysis study, it is observed that the classification accuracy for syllable stress detection using self-supervised representations significantly improved by 3.2% and 2.7% over knowledge-based features in GER and ITA, respectively. From the t-SNE plots, it is observed that the representations learned by explicit representation learning framework with VAE show better discrimination among stressed and unstressed syllables compared with representations learned implicitly with simple DNN.

Abstract

Speech systems are sensitive to accent variations. This is a challenge in India, which has numerous languages but few linguistic studies on pronunciation variation. The growing number of L2 English speakers reinforces the need to study accents and L1-L2 interactions. We investigate Indian English (IE) accents and report our observations on regional and shared features. Specifically, we observe phonemic variations and phonotactics in speakers’ native languages and apply this to their English pronunciations. We demonstrate the influence of 18 Indian languages on IE by comparing native language features with IE pronunciations obtained from literature studies and phonetically annotated speech. Hence, we validate Indian language influences on IE by justifying pronunciation rules from the perspective of Indian language phonology. We obtain a comprehensive description of generalised and region-specific IE characteristics, which facilitates accent adaptation of existing speech systems.