Text Representations aim to capture the semantic meaning, context, or sentiment of texts so they can be understood and processed by machines.

Information Extraction & Text Mining is a subfield of Natural Language Processing (NLP) that focuses on automatically extracting structured information and knowledge from unstructured textual data.

Representation Learning is a subfield of Semantic Text Processing that deals with the automatic learning and extraction of useful and compact representations of textual data, such as words, sentences, and documents, that can capture their semantic, syntactic, and pragmatic properties. It involves the development of algorithms and models that can transform natural language text into low-dimensional, dense, and informative vectors or embeddings that can be used in downstream tasks. This also includes learning vector representations of different modalities that can be used in NLP tasks.

Paraphrasing is a subfield of Text Generation that deals with the automatic generation of natural language text that expresses the same semantic meaning as a given input text, but using different words or syntactic structures. The goal of paraphrasing is to retain the original meaning while avoiding plagiarism, simplifying complex sentences, or adapting the language to a different audience.

Information Retrieval is a subfield of Natural Language Processing (NLP) that focuses on the automatic retrieval of relevant information from large collections of unstructured textual data. It involves developing techniques and algorithms to search, retrieve, and present information in response to user queries. Information Retrieval is concerned with understanding the meaning of the user's query and matching it with relevant information in the collection, taking into account factors such as word meaning, word similarity, and relevance.

Text Generation is a subfield of Natural Language Processing (NLP) which attempts to teach machines to automatically generate texts that are indistinguishable from texts written by humans.

Text Classification is a subfield of Information Retrieval and Information Extraction & Text Mining that deals with automatically assigning a label or class to a natural language text. It involves the development of algorithms and models that can learn to recognize patterns in texts and classify them into predefined categories.

Knowledge Representation is a subfield of Semantic Text Processing that deals with the representation, organization, and management of knowledge and information in a form that can be processed and used by computers in conjunction with natural language texts. It involves developing formal languages and frameworks, such as ontologies, semantic networks, and knowledge graphs, that can capture and represent the semantics, structure, and relations of concepts and entities in a domain-specific or general knowledge base.

Reasoning is a subfield of Natural Language Processing (NLP) that enables machines to draw logical conclusions and derive new knowledge from natural language texts based on the information available to them, using techniques such as deduction and induction.

Natural Language Processing (NLP) is an interdisciplinary subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language, in particular how to program computers to process and analyze large amounts of natural language data. The goal is a computer capable of 'understanding' the contents of documents, including the contextual nuances of the language within them. The technology can then accurately extract information and insights contained in the documents as well as categorize and organize the documents themselves.

Textual Inference is a subfield of Reasoning that deals with the task of automatically determining whether a natural-language hypothesis can be inferred from a given premise, predicting their logical relationship, such as entailment, contradiction, or neutral.

Semantic Text Processing is a subfield of Natural Language Processing (NLP) that attempts to derive meaning from natural language and helps machines interpret textual data semantically. It involves developing algorithms and models that can understand and represent the semantics of words and sentences, and the relationships between them.

Text Embeddings refer to the representation of text in the form of vectors. The underlying idea is to convert text into a numerical form that can be easily processed by machines. These embeddings capture the semantic meaning or context of words or sentences, where similar words or sentences are closer in the vector space.

Word Embeddings refer to words or phrases that are mapped to vectors of real numbers. These vectors capture the semantic and syntactic meaning, relationship with other words, and context of the words in a compact way.

<h1>PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification</h1> <h2>Abstract</h2> <p>We present a new release of the Paraphrase Database. PPDB 2.0 includes a discriminatively re-ranked set of paraphrases that achieve a higher correlation with human judgments than PPDB 1.0’s heuristic rankings. Each paraphrase pair in the database now also includes finegrained entailment relations, word embedding similarities, and style annotations.</p> <h2>1 Introduction</h2> <p>The Paraphrase Database (PPDB) is a collection of over 100 million paraphrases that was automatically constructed by Ganitkevitch et al. (2013). Although it is relatively new, it has been adopted by a large number of researchers, who have demonstrated that it is useful for a variety of natural language processing tasks. It has been used for recognizing textual entailment (Beltagy et al., 2014; Bjerva et al., 2014), measuring the semantic similarity of texts (Han et al., 2013; Ji and Eisenstein, 2013; Sultan et al., 2014b), monolingual alignment (Yao et al., 2013; Sultan et al., 2014a), natural language generation (Ganitkevitch et al., 2011), and improved lexical embeddings (Yu and Dredze, 2014; Rastogi et al., 2015; Faruqui et al., 2015). For any given input phrase to PPDB, there are often dozens or hundreds of possible paraphrases. There are several interesting research questions that arise because of the number and variety of paraphrases in PPDB. How can we distinguish between correct and incorrect paraphrases? Within the paraphrase sets, are all of the paraphrases truly substitutable or do they sometimes exhibit other types of relationships (like directional entailment)? When the paraphrases share the same meaning, are there stylistic reasons why we should choose one versus another (e.g., is one paraphrase a less formal version of another)? ranked paraphrases of berries PPDB 1.0 PPDB 2.0 1. embayments 1. strawberries @ 2. strawberries 2. raspberries @ 3. racks 3. blueberries @ 4. grains 4. blackberries @ 5. raspberries 5. fruits A 6. blueberries 6. fruit A 7. fruits 7. beans # 8. fruit 8. grains ∼ 9. blackberries 9. seeds # 10. beans 10. kernels # In this paper we describe several improvements to PPDB that address these questions. We release PPDB version 2.0, incorporating the following improvements: • A completely re-ranked set of paraphrases that uses a regression model to fit the paraphrase scores to human judgments of paraphrase quality. Figure 1 shows the re-ranked paraphrases for the word berries . • Each paraphrase pair is automatically labeled with an explicit entailment relationship. Instead of assuming all paraphrases are perfectly equivalent, we label some as one directional entailments (or other entailment types). • Each paraphrase rule now has new features that indicate when its application is expected to result in a change in style. • Each paraphrase entry in the database now has an associated word embedding learned using Multiview Latent Semantic Analysis. 425 p ( e 2 | e 1 ) ( ρ = 0.4144) PPDB 1.0 ( ρ = 0.4074) W2V ( ρ = 0.4633) PPDB 2.0 ( ρ = 0.7130) Upon publication of this paper, we will release PPDB 2.0 along with a set of 26K phrase pairs annotated with human similarity judgments.</p> <h2>2 Improved rankings of paraphrases</h2> <p>The notion of ranking paraphrases goes back to the original method that PPDB is based on. Bannard and Callison-Burch (2005) introduced the bilingual pivoting method, which extracts incarcerated as a potential paraphrase of put in prison since they are both aligned to festgenommen in different sentence pairs in an English-German bitext. Since incarcerated aligns to many foreign words (in many languages) the list of potential paraphrases is long. Paraphrases vary in quality since the alignments are automatically produced and noisy. In order to rank the paraphrases, Bannard and CallisonBurch (2005) define a paraphrase probability in terms of the translation model probabilities p ( f | e ) and p ( e | f ) : Heuristic scoring in PPDB 1.0 Instead of ranking the paraphrases with a single score, Ganitkevitch et al. (2013) expanded the set of scores in PPDB. Each paraphrase rule in PPDB consists of four components: a phrase ( e 1 ), a paraphrase ( e 2 ), a syntactic category ( LHS 1 ), and a feature vector. This feature vector contains 33 scores of paraphrase quality, which are described in full in the supplementary material to this paper. The rules in PPDB 1.0 were scored using an ad-hoc weighting of seven of these features, given by the following equation: where − log p ( e 2 | e 1 ) is the paraphrase probability computed according to Equation 1 and RarityP enalty is a real-valued feature that indicates how frequently the paraphrase was observed in the training data. This heuristic linear combination of scores was used to divide PPDB into six increasingly large sizes– S, M, L, XL, XXL, and XXXL. PPDBXXXL contains all of the paraphrase rules and has the highest recall, but the lowest average precision. The smaller sizes contain better average scores but offer lower coverage. Ganitkevitch et al. (2013) performed a small-scale analysis of how their heuristic score correlated with human judgments by collecting < 2,000 judgments for PPDB paraphrases of verbs that occurred in Propbank.</p> <h2>Supervised scoring model</h2> <p>For this paper, we rank the paraphrases using a supervised scoring model. To train the model, we collected human judgements for 26,455 paraphrase pairs sampled from PPDB. Each paraphrase pair was judged by 5 people who each assigned a score on a 5-point Likert scale, as described in Callison-Burch (2008). These 5 scores were averaged. We used these human judgments to fit a regression to the 33 features available in the PPDB 1.0 feature vector, plus an additional 176 new features that we developed. Our features included the cosine similarity of the word embeddings that we generated for each PPDB phrase (described in Section 3.3), as well as lexical overlap features, features derived from WordNet, and distributional 426 similarity features. We weighted the contribution of these features using ridge regression with its regularization parameter tuned using cross validation on the training data. See the supplemental materials for a complete description of the features used in our model and our data collection methodology including interannotator agreement.</p> <h2>2.1 Evaluating the rankings</h2> <p>We evaluate the new rankings in two ways: • We calculate the correlation of the different ways of automatically ranking the paraphrases against the 26k human judgments that we collected. • We compute the goodness (in terms of mean reciprocal rank and averaged precision) of the ranked paraphrase lists for 100 phrases drawn randomly from Wikipedia.</p> <h2>Correlation</h2> <p>Figure 2 plots the different automatic paraphrase scores against the 5-point human judgments for four different ways of ranking the paraphrases: 1) the original paraphrase probability defined by Bannard and Callison-Burch (2005), 2) the heuristic ranking that Ganitkevitch et al. (2013) defined for PPDB 1.0, 3) the cosine similarity of word2vec 2 embeddings 3 , and 4) the new score predicted by our discriminative model. The paraphrase probability has a Spearman correlation of 0.41. The heuristic PPDB 1.0 ranking has a similar correlation of ρ = 0 . 41 . The word2vec similarity improves correlation slightly to 0.46. To test our supervised method, we use cross validation: in each fold, we hold out 200 phrases along with all of their associated paraphrases for testing. Our rankings for PPDB 2.0 dramatically improve correlation with human judgments to ρ = 0 . 71 . Goodness of the top-ranked paraphrases In addition to calculating the correlation over the sample of paraphrases (where the human judgments were taken evenly over the range of p ( e 2 | e 1 ) values), we also evaluated the full list of paraphrases as it is likely to be used by researchers who use PPDB. We took a sample of 100 unique phrase types from Wikipedia (constraining to types which appear in PPDB), and collected human judgments for their full list of paraphrases. Random 0.56 0.46 (16% of judgments) p ( e 2 | e 1 ) 0.84 0.61 W2V 0.85 0.64 PPDB 1.0 0.86 0.64 PPDB 2.0 0.95 0.72 human rating ≥ 4 Random 0.34 0.27 (4% of judgments) p ( e 2 | e 1 ) 0.69 0.46 W2V 0.69 0.49 PPDB 1.0 0.70 0.50 PPDB 2.0 0.80 0.59 human rating ≥ 4.5 Random 0.25 0.20 (1% of judgments) p ( e 2 | e 1 ) 0.46 0.37 W2V We compare the ranking produced by the proposed PPDB 2.0 model against the heuristic PPDB 1.0 ranking in terms of each one’s ability to put good paraphrases at the top of its list. Figure 3 shows precision curves for the ranked paraphrases in PPDB 1.0 compared to PPDB 2.0. PPDB 2.0 achieves consistently higher precision, improving P@1 by 17 points and P@5 by 9 points. We also analyzed the different rankings when we varied the criterion that we used for what constitutes a good paraphrase. Table 1 shows how the averaged precision (AP) and the mean reciprocal rank (MRR) change as we vary the human score for good paraphrases from ≥ 3 to ≥ 4.5. Depending on the threshold, our PPDB 2.0 ranking achieves a 9-12 point improvement in MRR over the PPDB 1.0 rankings. Similarly, it improves AP by 7-9 points.</p> <h2>3 Other Additions</h2> <p>In addition to dramatically improving the rankings of the paraphrases (novel to this publication), our PPDB 2.0 release adds several automatic annotations created in other research. Every paraphrase pair now has an entailment relation from Pavlick et al. (2015), style classifications from Pavlick and Nenkova (2015), and associated vector embedding from Rastogi et al. (2015). These are described briefly below.</p> <h2>3.1 Entailment relations</h2> <p>Although we typically think of paraphrases as equivalent or as bidirectionally entailing, a substantial fraction of the phrase pairs in PPDB exhibit different entailment relations. Figure 1 gives an example of how these relations capture the range or entailment present in the paraphrases of berries . We automatically annotate each paraphrase rule in PPDB with an explicit entailment relation based on natural logic (MacCartney, 2009). These relations include forward entailment/hyponym ( @ ), reverse entailment/hypernym ( A ), non-entailing topical relatedness ( ∼ ), unrelatedness ( # ), and even exclusion/contradiction ( ¬ ). For a complete evaluation of the entailment classifications, and the prevalence of each type in PPDB, see Pavlick et al. (2015).</p> <h2>3.2 Style scores</h2> <p>Some of the variation within paraphrase sets can be attributed to stylistic variations of language. We automatically induce style information on each rule in PPDB for two dimensions– complexity and formality. Table 2 shows some paraphrases of the end , sorted from most complex to most simple using these scores. These classifications could be useful for natural language generation tasks like text simplification (Xu et al., 2015). A complete evaluation of these scores is given in Pavlick and Nenkova (2015).</p> <h2>3.3 Multiview LSA vector embeddings</h2> <p>Recently there has been tremendous interest in representing words via vector embeddings (Dhillon et al., 2011; Mikolov et al., 2013; Pennington et al., 2014). Such representations can be used to measure word and phrase similarity, possibly to improve paraphrasing. Multiview Latent Semantic Analysis (MVLSA) is a state-of-the-art method for modeling word similarities. MVLSA can incorporate an arbitrary number of data views, such as monolingual signals, bilingual signals, and even signals from other embeddings. PPDB 2.0 contains new similarity features based on MVLSA embeddings for all phrases. A complete discussion is given in Rastogi et al. (2015).</p> <h2>4 Related Work</h2> <p>The most closely related work to our supervised re-ranking of PPDB is work by Zhao et al. (2008) and Malakasiotis and Androutsopoulos (2011). Zhao et al. (2008) improved Bannard and Callison-Burch (2005)’s paraphrase probability by converting it into log-linear model inspired by machine translation, allowing them to incorporate a variety of features. Malakasiotis and Androutsopoulos (2011) developed a similar model trained on human judgements. Both efforts apply their model to natural language generation by paraphrasing full sentences. We apply our model to the sub-sentential paraphrases directly, in order to improve the quality of the Paraphrase Database. Also related is work by Chan et al. (2011) which reranked bilingually-extracted paraphrases using monolingual distributional similarities, but did not use a supervised model. Work that is relevant to our classification of semantic entailment types to each paraphrase, includes learning directionality of inference rules (Bhagat et al., 2007; Berant et al., 2011) and learning hypernyms rather than paraphrases (Snow et al., 2004). Our style annotations are related to Xu et al. (2012)’s efforts at learning stylistic paraphrases. Our word embeddings additions to the paraphrase database are related to many current projects on that topic, including projects that attempt to customize embeddings to lexical resources (Faruqui et al., 2015). However, the Rastogi et al. (2015) embeddings included here were shown to be state-of-the art in 428 predicting human judgements.</p> <h2>5 Conclusion</h2> <p>We release PPDB 2.0 ( http://paraphrase. org/#/download ). The resource includes dramatically improved paraphrase rankings, explicit entailment relations, style information, and state-of-the-art distributional similarity measures for each paraphrase rule. The 2.0 release contains 100m+ paraphrases, and 26k manually rated phrase pairs, which will facilitate further research in modeling semantic similarity.</p> <h2>Acknowledgements</h2> <p>This research was supported by the Allen Institute for Artificial Intelligence (AI2), the Human Language Technology Center of Excellence (HLTCOE), and by gifts from the Alfred P. Sloan Foundation, Google, and Facebook. This material is based in part on research sponsored by the NSF under grant IIS1249516 and DARPA under agreement number FA8750-13-2-0017 (the DEFT program). The U.S. Government is authorized to reproduce and The views and conclusions contained in this publication are those of the authors and should not be interpreted as representing official policies or endorsements of DARPA or the U.S. Government. We would like to thank the anonymous reviewers for their thoughtful comments.</p> <h2>References</h2> <p>429 430</p>

We present a new release of the Paraphrase Database. PPDB 2.0 includes a discriminatively re-ranked set of paraphrases that achieve a higher correlation with human judgments than PPDB 1.0’s heuristic rankings. Each paraphrase pair in the database now also includes finegrained entailment relations, word embedding similarities, and style annotations.

Publication:

PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification

Related Fields of Study

Citations

References