Abstract

In this paper, we introduce a novel approach to address the task of synthesizing speech from silent videos of any in-the-wild speaker solely based on lip movements. The traditional approach of directly generating speech from lip videos faces the challenge of not being able to learn a robust language model from speech alone, resulting in unsatisfactory outcomes. To overcome this issue, we propose incorporating noisy text supervision using a state-of-the-art lip-to-text network that instills language information into our model. The noisy text is generated using a pre-trained lip-to-text model, enabling our approach to work without text annotations during inference. We design a visual text-to-speech network that utilizes the visual stream to generate accurate speech, which is in-sync with the silent input video. We perform extensive experiments and ablation studies, demonstrating our approach's superiority over the current state-of-the-art methods on various benchmark datasets. Further, we demonstrate an essential practical application of our method in assistive technology by generating speech for an ALS patient who has lost the voice but can make mouth movements. Our demo video, code, and additional details can be found at http://cvit.iiit.ac.in/research/projects/cvit-projects/ms-l2s-itw.

Abstract

Visual servoing has been gaining popularity in various real-world vision-centric robotic applications. Autonomous robotic grasping often deals with unseen and unstructured environments, and in this task, Visual Servoing has been able to generate improved end-effector control by providing visual feedback. However, existing Servoing-aided grasping methods tend to fail at the task of grasping in dynamic environments i.e. - moving objects. In this paper, we introduce DynGraspVS, a novel Image-based Visual Servoing-aided Grasping approach that models the motion of moving objects in its interaction matrix. Leveraging a single-step rollout strategy, our approach achieves a remarkable increase in success rate, while converging faster and achieving a smoother trajectory, while maintaining precise alignments in six degrees of freedom. By integrating the velocity information into the interaction matrix, our method is able to successfully complete the challenging task of robotic grasping in the case of dynamic objects, while outperforming existing deep Model Predictive Control (MPC) based methods in the PyBullet simulation environment. We test it with a range of objects in the YCB dataset with varying range of shapes, sizes, and material properties. We report various evaluation metrics such as photometric error, success rate, time taken, and trajectory length.

Abstract

Deployment of Face Recognition systems on the edge has seen significant growth due to advancements in hardware design and efficient neural architectures. However, tailoring SOTA Face Recognition solutions to a specific edge device is still not easy and is vastly unexplored. Although, benchmark data is available for some combinations of model, device, and framework, it is neither comprehensive nor scalable. We propose an approximation to determine the relationship between a model and its inference time in an edge deployment scenario. Using a small number of data points, we are able to predict the throughput of custom models in an explainable manner. The prediction errors are small enough to be considered noise in observations. We also analyze which approaches are most efficient and make better use of hardware in terms of accuracy and error rates to gain a better understanding of their behaviour. Related & necessary modules such as Face Anti-Spoofing are also analyzed. To the best of our knowledge, we are the first to tackle this issue directly. The data and code along with future updates to the models and hardware will be made available at https://github.com/AyanBiswas19/Resource_Efficient_FR.

Abstract

Visual Place Recognition (VPR) is vital for robot localization. To date, the most performant VPR approaches are environment- and task-specific : while they exhibit strong performance in structured environments (predominantly urban driving), their performance degrades severely in unstructured environments, rendering most approaches brittle to robust real-world deployment. In this work, we develop a universal solution to VPR – a technique that works across a broad range of structured and unstructured environments (urban, outdoors, indoors, aerial, underwater, and subterranean environments) without any re-training or finetuning. We demonstrate that general-purpose feature representations derived from off-the-shelf self-supervised models with no VPR-specific training are the right substrate upon which to build such a universal VPR solution. Combining these derived features with unsupervised feature aggregation enables our suite of methods, AnyLoc , to achieve up to 4× significantly higher performance than existing approaches. We further obtain a 6% improvement in performance by characterizing the semantic properties of these features, uncovering unique domains which encapsulate datasets from similar environments. Our detailed experiments and analysis lay a foundation for building VPR solutions that may be deployed anywhere , anytime , and across anyview . We encourage the readers to explore our project page and interactive demos: https://anyloc.github.io/ .

Abstract

Biomimetic technologies are an important part of the soft robotics research area because of their ability to replicate naturally occurring structures. Researchers have been developing and improving many such technologies ranging from prosthetics to artificial skin. In this work, we present the fabrication and characterization of an artificial muscle flapping wing using dielectric elastomer actuators (DEAs). The actuator was fabricated using a urethane acrylate polymer elastomer with carbon nanotube (CNT) based electrodes. The 40 mm × 20 mm rectangular actuator was powered through two electrodes at the top. It was subjected to different voltages in the range of 500 - 2500 V and the displacement of the bottom edge was observed. We report the actuator edge displacement to be 20 mm, i.e., 0.5 body lengths (BL) at static 2.5 kV. The dynamic behavior of the device was also analyzed by actuating it with a square wave of various frequencies (0 to 10 Hz). We report the resonant frequency of the actuator to be 4 Hz, with a sub-harmonic at 2 Hz

Abstract

Abstract The proliferation of speech technology applications has led to an unprecedented demand for effective speech enhancement techniques, particularly in low Signal-to-Noise Ratio (SNR) conditions. This research presents a novel approach to speech enhancement, specifically designed for very low SNR scenarios. Our technique focuses on speaker embedding at a granular level and highlights its consistent impact on enhancing speech quality and improving Automatic Speech Recognition (ASR) performance, a significant downstream task. Experimental findings demonstrate competitive speech quality and substantial enhancements in ASR accuracy compared to alternative methods in low SNR situations. The proposed technique offers promising advancements in addressing the challenges posed by low SNR conditions in speech technology applications.

Abstract

This paper presents a trim-free, low-power, and high-precision voltage/current reference for IoT and biomedical applications. The voltage reference is generated by the proper addition of CTAT and PTAT voltages, whereas, the current reference is generated using a MOSFET in the active region, which acts like a resistor and helps in lowering power consumption without increasing the form factor. Implemented in the 90nm CMOS process, the proposed voltage/current reference outputs a voltage and current of 820mV and 3.23nA, respectively. Without trimming, the process variation of the proposed voltage/current reference is 1.34%(σ/μ) / 1.75%(σ/μ) and the temperature coefficient of the voltage/current reference is 62ppm/oC / 332ppm/oC over the temperature range of -40oC to 100oC. The line sensitivity of the voltage/current reference is 0.296%/V / 0.414%/V for a wide supply range of 1V - 3.5V. The area occupied by the total circuit is 0.0112mm2, while the total power consumption of the design is 37nW at the typical corner of 27oC and 1V supply.

Abstract

The pronunciation quality of second language (L2) learners can be affected by different factors including the fol- lowing seven factors: Intelligibility, Intonation, Phoneme quality, mispronunciation, Mother tongue influence, Correct placement of pause and Correct stress placement. An automatic assessment of these seven factors could be helpful for developing computer- assisted language learning systems. In this work, we assess the quality of all seven factors considering an unsupervised approach using DTW based utterance level alignment distance between expert’s and learner’s speech. Unlike the existing works that consider factor specific heuristic based features, we explore Wav2Vec-2.0 based self-supervised representations as a feature for assessing all the seven factors. The distance is computed using the following three distance metrics: Mean absolute error (MAE), Mean squared error (MSE) and Cosine distance (CD). Experiments are conducted on voisTUTOR corpus containing spoken English speech samples from 16 Indian L2 learners annotated with binary ratings (1 and 0) for all the factors. Using each distance metric, for each learner’s speech, four distance values are computed with a set of four expert samples. In the assessment, to circumvent the need for parallel expert data, we consider two (out of four) expert samples synthesized from the state-of-the-art text-to-speech (TTS) systems. We observe that the performance with the considered distance metric based unsupervised assessment approach is significantly more than that with the baseline for six out of seven factors under all three distance metrics and all four experts samples.

Abstract

The demand for high-quality speech data has been increasing as deep-learning approaches gain popularity in speech applications. Among these, automatic speech recognition (ASR) and text-to-speech (TTS) require large amount of data contain- ing speech and the corresponding text. For these applications, high-quality data is often obtained through manual validation, which ensures matching between speech and text. The manual validation is not scalable as per the demand due to the cost and time involved. In order to cater to the high-quality data demand, validating the data automatically could be useful. In this work, for automatic data validation, a spoken English corpus named IIITH MM2 Speech-Text is created, containing matched and mismatched speech-text pairs under read speech conditions from Indian speakers with different nativities. For the creation, we consider 100 unique stimuli selected from the TIMIT corpus, ensuring phonetic richness, for which a joint entropy maximization is proposed. These stimuli are recorded from 50 speakers, resulting in matched and mismatched sets containing 5000 and 764 utterances with a total duration of 6 hours and 1 hour, respectively. The mismatched set contains speech from the instances where the speakers naturally made spoken errors while reading the reference text. It also contains two stimuli per utterance, one stimulus is the reference text, and the other is manually annotated text that reflects the erroneous speech. Thus, the reference and the annotated text are used for building the models of speech-text mismatch detection and correction, respectively. To the best of our knowledge, no such corpora exist containing both matched and mismatched speech-text. As a preliminary analysis for speech-text mismatch detection, a baseline considering Wav2Vec-2.0 representations and DTW results in the detection F1-score of 0.87

Abstract

—This paper presents a sub-1V, 593 pA current reference without using resistors and amplifiers. Temperature compensation is achieved by incorporating a CTAT gate to source voltage and a PTAT drain to source voltage in an NMOSbased sub-threshold biased circuit. The proposed architecture achieves ± 3σ/µ process variation of ± 0.75%, which enables seamless trim-free operation. Implemented in the 90nm CMOS process, the temperature-compensated current reference achieves a temperature coefficient of 378 ppm/oC over a high-temperature range of -40 to 100 oC and a line sensitivity of 0.198%/V in a supply voltage range from 0.8 to 2.5 V. Duty cycling technique is adopted in the CTAT generator to reduce the total average power consumption. The proposed circuit occupies an area of 0.074 mm2 while consuming only 3.3 nW at a typical corner of 27 oC and 0.8 V supply.