Semantic Text Processing is a subfield of Natural Language Processing (NLP) that attempts to derive meaning from natural language and helps machines interpret textual data semantically. It involves developing algorithms and models that can understand and represent the semantics of words and sentences, and the relationships between them.

Natural Language Processing (NLP) is an interdisciplinary subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language, in particular how to program computers to process and analyze large amounts of natural language data. The goal is a computer capable of 'understanding' the contents of documents, including the contextual nuances of the language within them. The technology can then accurately extract information and insights contained in the documents as well as categorize and organize the documents themselves.

Representation Learning is a subfield of Semantic Text Processing that deals with the automatic learning and extraction of useful and compact representations of textual data, such as words, sentences, and documents, that can capture their semantic, syntactic, and pragmatic properties. It involves the development of algorithms and models that can transform natural language text into low-dimensional, dense, and informative vectors or embeddings that can be used in downstream tasks. This also includes learning vector representations of different modalities that can be used in NLP tasks.

Language Models are a subfield of Semantic Text Processing that deal with the probabilistic modeling and generation of natural language, based on a learned probability function of sequences of words. Language models generate probabilities by training on corpora of natural language texts and can be fine-tuned for a variety of different downstream tasks.

Transformers & Large Language Models is concerned with models based on the transformer neural network architecture that introduced the attention mechanism and has become foundational in various machine learning applications. In addition, this concept includes Large Language Models, a specific application of transformers that are pre-trained on large amounts of textual data and excel at various natural language processing tasks.

Understanding the similarity of the numerous released large language models (LLMs) has many uses, e.g., simplifying model selection, detecting illegal model reuse, and advancing our understanding of what makes LLMs perform well. In this work, we measure the similarity of representations of a set of LLMs with 7B parameters. Our results suggest that some LLMs are substantially different from others. We identify challenges of using representational similarity measures that suggest the need of careful study of similarity scores to avoid false conclusions.

Publication:

Towards Measuring Representational Similarity of Large Language Models

Related Fields of Study

Citations

References