In the realm of data-to-text generation tasks, the use of large language models (LLMs) has become common practice, yielding fluent and coherent outputs. Existing literature highlights that the quality of in-context examples significantly influences the empirical performance of these models, making the efficient selection of high-quality examples crucial. We hypothesize that the quality of these examples is primarily determined by two properties: their similarity to the input data and their diversity from one another. Based on this insight, we introduce a novel approach, Double Clustering-based In-Context Example Selection, specifically designed for data-to-text generation tasks. Our method involves two distinct clustering stages. The first stage aims to maximize the similarity between the in-context examples and the input data. The second stage ensures diversity among the selected in-context examples. Additionally, we have developed a batched generation method to enhance the token usage efficiency of LLMs. Experimental results demonstrate that, compared to traditional methods of selecting in-context learning samples, our approach significantly improves both time efficiency and token utilization while maintaining accuracy.

This survey provides an overview of the challenges of misspellings in natural language processing (NLP). Misspellings are ubiquitous in digital communication, and even if humans can generally interpret misspelt text, NLP models frequently struggle to handle it: this causes a decline in performance in common tasks like text classification and machine translation. In this paper, we reconstruct a history of misspellings as a scientific problem. We then discuss the latest advancements to address the challenge of misspellings in NLP. Main strategies to mitigate the effect of misspellings include data augmentation, double step, character-order agnostic, and tuple-based methods, among others. This survey also examines dedicated data challenges and competitions to spur progress in the field. Critical safety and ethical concerns are also examined, for example, the voluntary use of misspellings to inject malicious messages and hate speech on social networks. The survey also explores psycholinguistic perspectives on how humans process misspellings, potentially informing innovative computational techniques for text normalisation and representation. Additionally, the survey explores the challenges that misspellings pose in multilingual contexts. Finally, the misspelling-related challenges and opportunities associated with modern large language models are also analysed, including benchmarks, datasets and performances of the most prominent language models against misspellings. This survey provides a comprehensive review of recent research on misspellings and aims to serve as a valuable resource for researchers seeking to get up to speed on this problem within the rapidly evolving landscape of NLP.

Controllable summarization models are typically limited only to a short text, such as a topic mention, a keyword, or an entity, to control the output summary. At the same time, existing models for controllable summarization are prone to generate artificial content, resulting in unreliable summaries. In this work, we propose a method for controllable abstractive summarization that can exploit arbitrary textual context from a short text to a collection of documents to direct the focus of the generated summary. The proposed method incorporates a sentence BERT model to extract an embedding-based representation of the given context, which is then used to tag the most representative words of the input document towards this context. In addition, we propose an unsupervised metric to evaluate the faithfulness of the topic-oriented sentences of the generated summaries with respect to the input document. Experimental results under different zero-shot setups demonstrate that the proposed method surpasses both state-of-the-art large language models (LLMs) and controllable summarization methods. The generated summaries are both reliable and relevant with respect to the input document.

Results obtained by transformer-based token classification models are now considered to be a benchmark for the Automatic Terminology Extraction (ATE) task. However, the unsatisfactory results (they rarely exceed 0.7 of the F1 value) raise the question of whether this approach is correct and of what text features are being remembered or inferred by the model trained on this type of annotation. In the paper, we describe a number of experiments using the fine-tuned RoBERTa base model on the ACTER data, RD-TEC, and three Wikipedia articles, which proved that the results of the ATE task obtained by such models depend considerably on the type of texts being processed and their relationship to the training data. While the results are relatively good for some texts with highly specialized vocabulary, the poor results seem to correlate with the high frequency (in general English texts) of tokens that are part of terms in a particular domain. Another property that affects the results is the degree of overlap between the vocabulary of the test data and the vocabulary of terms from the training data. Words that have been labeled as terms in the training data are usually labeled as terms in other, unrelated domains as well. Moreover, we show that the results obtained by these models are unstable—models trained on more data do not include all the items identified by models trained on a smaller dataset and can present substantially lower performance.

In the digital information age, artificial intelligence is increasingly being applied to national governance and judicial decision-making assistance. Existing studies lack case studies and empirical analyses of the effectiveness of large models in aiding judicial decisions. To address this research gap, this study designs a comprehensive evaluation framework encompassing five core task dimensions: Task-oriented Information Extraction, Legal Article Citation, Event Extraction, Judicial Decision Generation, and Legal Opinion Generation. By using carefully crafted prompts to activate the legal reasoning capabilities of the models, we conducted extensive testing on 13 mainstream large language models (LLMs). The experimental results demonstrate that large models perform excellently in processing legal texts and providing preliminary legal opinions, but still exhibit shortcomings in complex legal reasoning and precise decision-making. On this basis, we applied a weakly supervised learning strategy to fine-tune the LLMs for targeted improvements. The results indicate that introducing a small amount of task-specific learning can significantly enhance the performance of LLMs in judicial tasks. This further underscores the critical role of data and the acquisition of domain-specific knowledge in applying AI technology to judicial tasks. Additionally, this study briefly discusses the issue of the boundaries of AI’s involvement in judicial activities, aiming to provide theoretical foundations and practical guidance for the deep integration of AI technology with legal practice.

Named Entity Recognition (NER) is a natural language processing task that traditionally relies on supervised learning and annotated data. Acquiring such data is often a challenge, particularly in specialized fields like medical, legal, and financial sectors. Those are commonly referred to as low-resource domains, which comprise long-tail entities, due to the scarcity of available data. To address this, data augmentation techniques are increasingly being employed to generate additional training instances from the original dataset. In this study, we evaluate the effectiveness of two prominent text augmentation techniques, Mention Replacement and Contextual Word Replacement, on two widely used NER models, Bi-LSTM + CRF and BERT. We conduct experiments on three datasets from low-resource domains, and we explore the impact of various combinations of training subset sizes and the number of augmented examples. We not only confirm that data augmentation is particularly beneficial for smaller datasets, but we also demonstrate that there is no universally optimal number of augmented examples, i.e., NER practitioners must experiment with different quantities in order to fine-tune their projects.

We propose an unsupervised, corpus-independent method to extract keywords from a single text. It is based on the spatial distribution of words and the response of this distribution to a random permutation of words. Our method has three advantages over existing unsupervised methods (such as YAKE). First, it is significantly more effective at extracting keywords from long texts in terms of precision and recall. Second, it allows inference of two types of keywords: local and global. Third, it extracts basic topics from texts. Additionally, our method is language-independent and applies to short texts. The results are obtained via human annotators with previous knowledge of texts from our database of classical literary works. The agreement between annotators is moderate to substantial. Our results are supported via human-independent arguments based on the average length of extracted content words and on the average number of nouns in extracted words. We discuss relations of keywords with higher-order textual features and reveal a connection between keywords and chapter divisions.

Neural Machine Translation (NMT), a subfield of Natural Language Processing, has seen significant advancements with the emergence of transformer architectures and generative artificial intelligence, demonstrating remarkable performance in various languages. However, translating Arabic dialects remains a notable challenge that becomes very pronounced primarily due to their morphological complexity and divergence from standardised grammatical rules. In this paper, we present a hybrid approach for translating the Maghrebi dialects into/from Modern Standard Arabic (MSA). The approach takes advantage of the strengths of the transformer architecture and the BERT language model for transfer learning of representations. To achieve this, we incorporated BERT embeddings into the encoder and decoder stacks of the transformer architecture. The BERT architecture, which we utilised, was trained in a self-supervised manner on Maghrebi dialects and Arabic corpora. The resulting BLEU/BERTScore/ChrF/METEOR scores for the approach were 14.148/79.414/28.885/28.428 and 8.961/20.994/19.465 (BLEU/ChrF/METEOR) for the translation in both directions using the raw data, demonstrating competitive performance compared to ChatGPT and Gemini Large Language Models (LLMs). Furthermore, we evaluated the approach using an ablation study with fine-tuned NLLB-200 and against three combinations of tokeniser techniques used in conjunction with the transformer architecture: Byte-Pair Encoding (BPE) tokeniser, WordPiece tokeniser, and BERT tokeniser. Both evaluations, including human evaluation, confirm the efficacy of our method.

Over the next decade, I expect more and more computing on phones and small computers and less on clouds and supercomputers. Supercomputers are super impressive to engineers, but not to economists because economies of scale have more to do with the size of the market than the size of the machine. Clouds are like wire-wrapped Cray computers; they were never designed for mass production. There was never much supply or demand. Supercomputers and clouds are too expensive and burn too much power. The future is more promising for phones because the market is larger. In addition, there are a number of other advantages to computing on phones and small (commodity) machines designed for mass markets: privacy, power, size, weight, latency, bandwidth, and especially affordability.