Abstract

This paper presents our contributions to the WMT2023 shared metrics task, consisting of two distinct evaluation approaches: a) Unsu- pervised Metric (MEE4) and b) Supervised Metric (XLSim). MEE4 represents an unsu- pervised, reference-based assessment metric that quantifies linguistic features, encompass- ing lexical, syntactic, semantic, morphologi- cal, and contextual similarities, leveraging em- beddings. In contrast, XLsim is a supervised reference-based evaluation metric, employing a Siamese Architecture, which regresses on Di- rect Assessments (DA) from previous WMT News Translation shared tasks from 2017-2022. XLsim is trained using XLM-RoBERTa (base) on English-German reference and mt pairs with human scores. Here are the links for MEE4 1 and XLsim2 metrics.

Abstract

Proceedings of the Eighth Conference on Machine Translation (WMT), pages 246–251 December 6–7, 2023. ©2023 Association for Computational Linguistics 246IIIT HYD’s Submission for WMT23 Test-suite task Ananya Mukherjee and Manish Shrivastava Machine Translation - Natural Language Processing Lab Language Technologies Research Centre International Institute of Information Technology - Hyderabad ananya.mukherjee@research.iiit.ac.in m.shrivastava@iiit.ac.in Abstract This paper summarizes the results of our test suite evaluation on 12 machine translation systems submitted at the Shared Task of the 8th Conference of Machine Translation (WMT23) for English-German (en-de) language pair. Our test suite covers five specific domains (entertainment, environ- ment, health, science, legal) and spans five distinct writing styles (descriptive, judgments, narrative, reporting, technical-writing). We present our analysis through automatic evaluation methods, conducted with a focus on domain-specific and writing style-specific evaluations.

Abstract

Lately, instruction-based techniques have made significant strides in improving performance in few-shot learning scenarios. They achieve this by bridging the gap between pre-trained language models and fine-tuning for specific downstream tasks. Despite these advancements, the performance of Large Language Models (LLMs) in information extraction tasks like Named Entity Recognition (NER), using prompts or instructions, still falls short of supervised baselines. The reason for this performance gap can be attributed to the fundamental disparity between NER and LLMs. NER is inherently a sequence labeling task, where the model must assign entity-type labels to individual tokens within a sentence. In contrast, LLMs are designed as a text generation task. This distinction between semantic labeling and text generation leads to subpar performance. In this paper, we transform the NER task into a text-generation task that can be readily adapted by LLMs. This involves enhancing source sentences with task-specific instructions and answer choices, allowing for the identification of entities and their types within natural language. We harness the strength of LLMs by integrating supervised learning within them. The goal of this combined strategy is to boost the performance of LLMs in extraction tasks like NER while simultaneously addressing hallucination issues often observed in LLM-generated content. A novel corpus Contract NER comprising seven frequently observed contract categories, encompassing named entities associated with 18 distinct legal entity types is released along with our baseline models. Our models and dataset are available to the community for future research

Abstract

Semantic segmentation provides a pixel-level understanding of an image essential for various scene-understanding vision tasks. However, semantic segmentation models demand significant computational resources during training and inference. These requirements pose a challenge in resource-constraint scenarios. To address this issue, we present a compression algorithm based on differentiable meta-pruning through hypernetwork: MPHyp. Our proposed method MPHyp utilizes hypernetworks that take latent vectors as input and output weight matrices for the segmentation model. L1 sparsification follows the proximal gradient optimizer, updates the latent vectors and introduces sparsity leading to automatic model pruning. The proposed method offers the benefit of achieving controllable compression during the training and significantly reducing the training time. We compare our methodology with a popular pruning approach and demonstrate its efficacy by reducing the number of parameters and floating point operations while maintaining the mean Intersection over Union (mIoU) metric. We conduct experiments on two widely accepted semantic segmentation architectures: UNet and ERFNet. Our experiments and ablation study demonstrate the effectiveness of our proposed methodology by achieving efficient and reasonable segmentation results.

Abstract

Water quality modelling has proved to be effective method for managing river water quality. But the most effective and comprehensive approach involving integration of river water quality simulation and pollution visualization with the objective of pollution reduction and maintenance of environmental flow strategies has gained less attention. Thus, the objective of this study was to employ an integrated approach for mapping and analysing river water quality under various hydro-climatic and pollution scenarios. Specifically, this approach involved the integration of a river water quality simulation model, QUAL2K, Global Environmental Flow Calculator (GEFC), and Geographical Information System (GIS) to develop water quality index (WQI) based map charts of water quality. The calibrated QUAL2K model was utilized to simulate WQI parameters including water temperature, pH, electrical conductivity, dissolved oxygen (DO), biological oxygen demand (BOD), nitrates (NO3), ammonia (NH4), and alkalinity. To analyse the WQI, the Weighted Arithmetic-Water Quality Index (WA-WQI) method was employed for various individual and combined pollution scenarios, environmental flow (Eflow), and climate change scenarios. The developed integrated approach was applied to the Bhadravati segment of Bhadra River, India. The findings revealed that the prevailing WQI status of the study stretch ranged from poor to unsuitable for drinking purposes. This deterioration can be attributed to the impact of both industrial and municipal effluents. By maintaining the effective Environmental Management Class (EMC) flow rates (class C flowrate of EMC (40.32 m3/s)) in conjunction with appropriate Pollution Reduction (PR) level (10% PR) at headwater and incoming drains, the stream self-purification capacity was enhanced resulting in the Bhadravati River stretch water quality transitioning to favourable water quality condition.

Abstract

This research was carried out along the Bhadra River, which is one of the tributaries of the Tungabhadra and originates near Gangamula in Karnataka’s the Western Ghats. The study stretch is around 27 km divided into three reaches with elements of 1 km as 3, 4, and 20 for each reach, respectively. The objectives of the study are to assess the effects of wastewater discharges on the water quality of the Bhadra River stretch and to simulate the dissolved oxygen (DO) by varying the biochemical oxygen demand (BOD) loads coming from different pollutant sources within the study stretch using the QUAL2K river water quality model. The study period considered is about 24 months (April 2014–March 2016) for the simulation of temperature, pH, DO, and nitrate parameters with the help of observed data at three monitoring stations. The investigation period was separated into two parts: the model was calibrated using monthly average data from April 2014 to March 2015 (12 months), and the model was validated using monthly average data from April 2015 to March 2016 (12 months). The R-square (R2) value for temperature and pH ranges from 0.87–0.56 to 0.64–0.62, respectively. The Root Mean Square Error (RMSE) for nitrate and DO ranges from 0.06–0.02 to 0.47–0.34, respectively. The results revealed that the Bhadra river stretch was highly polluted due to effluents coming from the industries present in the study stretch. It was also discovered that river flow rate, point source discharge, fast Carbonaceous Biochemical oxygen demand (CBOD) oxidation rate, and nitrification rate as the highly sensitive quality parameters defining the Bhadra river water quality. There must be a reduction of 25% of BOD effluent to reach the minimum standards set by the Central Pollution Control Board (CPCB). It is noted that a 75% reduction of BOD effluent from point sources will lead to an increase of 15% average DO throughout the study stretch.

Abstract

In an effort to address the training instabilities of GANs, we introduce a class of dual-objective GANs with different value functions (objectives) for the generator (G) and discriminator (D). In particular, we model each objective using α-loss, a tunable classification loss, to obtain (αD,αG)-GANs, parameterized by (αD,αG)∈(0,∞]2. For sufficiently large number of samples and capacities for G and D, we show that the resulting non-zero sum game simplifies to minimizing an f-divergence under appropriate conditions on (αD,αG). In the finite sample and capacity setting, we define estimation error to quantify the gap in the generator's performance relative to the optimal setting with infinite samples and obtain upper bounds on this error, showing it to be order optimal under certain conditions. Finally, we highlight the value of tuning (αD,αG) in alleviating training instabilities for the synthetic 2D Gaussian mixture ring and the Stacked MNIST datasets.

Abstract

In this report, we share our contribution to the Eval4NLP Shared Task titled "Prompting Large Language Models as Explainable Metrics." We build our prompts with a primary focus on effective prompting strategies, score-aggregation, and explainability for LLM-based metrics. We participated in the track for smaller models by submitting the scores along with their explanations. According to the Kendall correlation scores on the leaderboard, our MT evaluation submission ranks second-best, while our summarization evaluation submission ranks fourth, with only a 0.06 difference from the leading submission.

Abstract

No Results Found

Abstract

No Results Found