Sentiment Analysis is a subfield of Natural Language Processing (NLP) that deals with the identification, extraction and quantification of affective states and subjective information from textual data.

Opinion Mining is a subfield of Sentiment Analysis that deals with the automatic extraction and analysis of opinions, attitudes, and positions regarding a particular topic expressed in natural language texts.

Natural Language Processing (NLP) is an interdisciplinary subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language, in particular how to program computers to process and analyze large amounts of natural language data. The goal is a computer capable of 'understanding' the contents of documents, including the contextual nuances of the language within them. The technology can then accurately extract information and insights contained in the documents as well as categorize and organize the documents themselves.

<h1>Bidirectional Inter-dependencies of Subjective Expressions and Targets and their Value for a Joint Model</h1> <h2>Abstract</h2> <p>Opinion mining is often regarded as a classification or segmentation task, involving the prediction of i) subjective expressions, ii) their target and iii) their polarity. Intuitively, these three variables are bidirectionally interdependent, but most work has either attempted to predict them in isolation or proposing pipeline-based approaches that cannot model the bidirectional interaction between these variables. Towards better understanding the interaction between these variables, we propose a model that allows for analyzing the relation of target and subjective phrases in both directions, thus providing an upper bound for the impact of a joint model in comparison to a pipeline model. We report results on two public datasets (cameras and cars), showing that our model outperforms state-ofthe-art models, as well as on a new dataset consisting of Twitter posts.</p> <h2>1 Introduction</h2> <p>Sentiment analysis or opinion mining is the task of identifying subjective statements about products, their polarity ( e. g. positive, negative or neutral) in addition to the particular aspect or feature of the entity that is under discussion, i. e. , the socalled target . Opinion analysis is thus typically approached as a classification (T¨ ackstr¨ om and McDonald, 2011; Sayeed et al., 2012; Pang and Lee, 2004) or segmentation (Choi et al., 2010; Johansson and Moschitti, 2011; Yang and Cardie, 2012) task by which fragments of the input are classified or labelled as representing a subjective phrase (Yang and Cardie, 2012), a polarity or a target (Hu and Liu, 2004; Li et al., 2010; Popescu and Etzioni, 2005; Jakob and Gurevych, 2010). As an example, the sentence “I like the low weight of the camera.” contains a subjective term “like”, and the target “low weight”, which can be classified as a positive statement. While the three key variables (subjective phrase, polarity and target) intuitively influence each other bidirectionally, most work in the area of opinion mining has concentrated on either predicting one of these variables in isolation ( e. g. subjective expressions by Yang and Cardie (2012)) or modeling the dependencies uni-directionally in a pipeline architecture, e. g. predicting targets on the basis of perfect and complete knowledge about subjective terms (Jakob and Gurevych, 2010). However, such pipeline models do not allow for inclusion of bidirectional interactions between the key variables. In this paper, we propose a model that can include bidirectional dependencies, attempting to answer the following questions which so far have not been addressed but provide the basis for a joint model: • What is the impact of the performance loss of a non-perfect subjective term extraction in comparison to perfect knowledge? • Further, how does perfect knowledge about targets influence the prediction of subjective terms? • How is the latter affected if the knowledge about targets is imperfect, i. e. predicted by a learned model? We study these questions using imperatively defined factor graphs (IDFs, McCallum et al. (2008), McCallum et al. (2009)) to show how these bidirectional dependencies can be modeled in an architecture which allows for further steps towards joint inference. IDFs are a convenient way to define probabilistic graphical models that make structured predictions based on complex dependencies. 848</p> <h2>2 A Model for the Extraction of Target Phrases and Subjective Expressions</h2> <p>This section gives a brief introduction to imperatively defined factor graphs and then introduces our model.</p> <h2>2.1 Imperatively Defined Factor Graphs</h2> <p>A factor graph (Kschischang et al., 2001) is a bipartite graph over factors and variables. Let factor graph G define a probability distribution over a set of output variables y conditioned on input variables x . A factor Ψ i computes a scalar value over the subset of variables x i and y i that are neighbors of Ψ i in the graph. Often this real-valued function is defined as the exponential of an inner product over sufficient statistics { f ik ( x i , y i ) } and parameters { θ ik } , where k ∈ [1 , K i ] and K i is the number of parameters for factor Ψ i . A factor template T j consists of parameters { θ jk } , sufficient statistic functions { f jk } , and a description of an arbitrary relationship between variables, yielding a set of tuples { ( x j , y j ) } . For each of these tuples, the factor template instantiates a factor that shares { θ jk } and { f jk } with all other instantiations of T j . Let T be the set of factor templates and Z ( x ) be the partition function for normalization. The probability distribution can then be written as p ( y | x ) = 1 Z ( x ) Q F ACTORIE 1 (McCallum et al., 2008; McCallum et al., 2009) is an implementation of imperatively defined factor graphs in the context of Markov</p> <h2>better than CCD shift systems</h2> <p>n chain Monte Carlo (MCMC) inference, a common approach for inference in very large graph structures (Culotta and McCallum, 2006; Richardson and Domingos, 2006; Milch et al., 2006). The term imperative is used to denote that actual code in an imperative programming language is written to describe templates and the relationship of tuples they yield. This flexibility is beneficial for modeling inter-dependencies as well as designing information flow in joint models.</p> <h2>2.2 Model</h2> <p>Our model is similar to a semi-Markov conditional random field (Sarawagi and Cohen, 2004). It predicts the offsets for target mentions and subjective phrases and can use the information of each other during inference. In contrast to a linear chain conditional random field (Lafferty et al., 2001), this allows for taking distant dependencies of unobserved variables into account and simplifies the design of features measuring characteristics of multi-token phrases. The relevant variables, i. e. target and subjective phrase, are modelled via complex span variables of the form s = ( l, r, c ) with a left and right offset l and r , and a class c ∈ { target , subjective } . These offsets denote the span on a token sequence t = ( t 1 , . . . , t n ) . We use two different templates to define factors between variables: a single span template and an inter-span template. The single span template defines factors with scores based on features of the tokens in the span and its vicinity. In our model, all features are boolean. As token-based features we use the POS tag, the lower-case representation of the token as well as both in combination. The actual span representation consists of these features prefixed with “I” for all tokens in the span, with “B” for the token at the beginning of the span, and with “E” for the token at the end of the span. In addition, the sequence of POS tags of all tokens in the span is included as a feature. The inter-span template takes three characteristics of spans into account: Firstly, we measure if a potential target span contains a noun which is the closest noun to a subjective expression. Secondly, we measure for each span if a span of the other class is in the same sentence. A third feature indicates whether there is only one edge in the dependency graph between the tokens contained in spans of a different class. These features are to a great extent inspired by Jakob and Gurevych 849 (2010). For parsing, we use the Stanford parser (Klein and Manning, 2003). The features described so far, however, cannot differentiate between a possible aspect mention which is a target of a subjective expression and one which is not. Therefore, the features of the inter-span template are actually built by taking the cross-product of the three described characteristics with all single-span features. Spans which are not in the context of a span of a different class are represented by a ‘negated’ feature (namely No-CloseNoun , No-Single-Edge , and Not-Both-In-Sentence ). The example in Figure 1 shows features for two spans which are in context of each other. All of these features representing the text are taken into account for each class, i. e. , target and subjective expression. Inference is performed via Markov Chain Monte Carlo (MCMC) sampling. In each sampling step, only the variables which actually change need to be evaluated, and therefore the sampler directs the process of unrolling the templates to factors. These world changes are necessary to find the maximum a posteriori (MAP) configuration as well as learning the parameters of the model. For each token in the sequence, a span of length one of each class is proposed if no span containing the token exists. For each existing span, it is proposed to change its label, shorten or extend it by one token if possible (all at the beginning and at the end of the span, respectively). Finally, a span can be removed completely. In order to learn the parameters of our model, we apply SampleRank (Wick et al., 2011). A crucial component in the framework is the objective function which gives feedback about the quality of a sample proposal during training. We use the following objective function f ( t ) to evaluate a proposed span t : where s is the set of all spans in the gold standard. Further, the function o calculates the overlap in terms of tokens of two spans and the function p returns the number of tokens in t that are not contained in g , i. e. , those which are outside the overlap (both functions taking into account the class of the span). Thus, the first part of the objective function represents the fraction of correctly proposed contiguous tokens, while the second part penalizes a span for containing too many tokens that are outside the best span. Here, α is a parameter which controls the penalty.</p> <h2>3 Results and Discussion 3.1 Experimental Setting</h2> <p>We report results on the J.D. Power and Associates Sentiment Corpora 2 , an annotated data set of blog posts in the car and in the camera domain (Kessler et al., 2010). From the rich annotation set, we use subjective terms and entity mentions which are in relation to them as targets. We do not consider comitter , negator , neutralizer , comparison , opo , or descriptor annotations to be subjective expressions. Results on these data sets are compared to Jakob and Gurevych (2010). In addition, we report results on a Twitter data set 3 for the first time (Spina et al., 2012). Here, we use a Twitter-specific tokenizer and POS tagger 4 (Owoputi et al., 2013) instead of the Stanford parser. Hence, the single-edge-based feature described in Section 2.2 is not used for this dataset. A short summary of the datasets is given in Table 1. As evaluation metric we use the F 1 measure, the harmonic mean between precision and recall. True positive spans are evaluated in a perfect match and approximate match mode, where the latter regards a span as positive if one token within it is included in a corresponding span in the gold standard. In this case, other predicted spans matching the same gold span do not count as false positives. In the objective function, α is set to 0 . 01 to prefer spans which are longer than the gold phrase over predicting no span. Four different experiments are performed (all via 10-fold cross validation): First, predicting subjectivity expressions followed by predicting targets while making use of the previous prediction. SecCar Camera Twitter Texts 457 178 9238 Targets 11966 4516 1418 Subjectives 15056 5128 1519 850 ond, predicting targets followed by predicting subjective expressions. Third, assuming perfect knowledge of subjective expressions when predicting targets, and fourth, assuming perfect knowledge of targets in predicting subjective expressions. This provides us with the information how good a prediction can be with perfect knowledge of the other variable as well as an estimate of how good the</p> <h2>3.2 Results</h2> <p>Figures 2, 3 and 4 show the results for the four different settings compared to the results by Jakob and Gurevych (2010) for cars and cameras. The darker bars correspond to perfect match, the lighter ones to the increase when taking partial matches into account. In the following we only discuss the perfect match. Comparing the results (for the car and camera data sets, Figure 2 and 3) for subjectivity prediction, one can observe a limited performance when targets are not known (0.54 F 1 for the camera set, 0.56 F 1 for the car set), an upper bound with perfect target information is much higher (0.65 F 1 , 0.7 F 1 ). When first predicting targets followed by subjective term prediction, we obtain results of 0.6 F 1 and 0.66 F 1 . The results for target prediction are much lower when not knowing subjective expressions in advance (0.32 F 1 , 0.33 F 1 ), and clearly increase with predicted subjective expressions (0.48 F 1 , 0.43 F 1 ) and outperform previous results when compared to Jakob and Gurevych (2010) (0.58 F 1 , 0.55 F 1 in comparison to their 0.5 F 1 on both sets). The results for the Twitter data set show the same characteristics (in Figure 4). However, they are generally much lower. In addition, the difference between exact and partial match evaluation modes is higher. This is due to the existence of many more phrases spanning multiple tokens. Exemplarily, the impact of the three features in the inter-span templates for the camera data set is depicted in Figure 5 for (a) given subjective terms (b) given targets, respectively. Detecting the closest noun is mainly of importance for target identification and only to a minor extent for detecting subjective phrases. A short path in the dependency graph and detecting if both phrases are in the same sentence have a high positive impact for both subjective and target phrases.</p> <h2>3.3 Conclusion and Discussion</h2> <p>The experiments in this paper show that target phrases and subjective terms are clearly interdependent. However, the impact of knowledge about one type of entity for the prediction of the other type of entity has been shown to be asymmetric. The results clearly suggest that the impact of subjective terms on target terms is higher than the other way round. Therefore, if a pipeline architecture is chosen, this order is to be preferred. However, the results with perfect knowledge of the counterpart entity show (in both directions) that the entities influence each other positively. Therefore, the challenge of extracting subjective expressions and their targets is a great candidate for applying supervised, joint inference.</p> <h2>Acknowledgments</h2> <p>Roman Klinger has been funded by the “It’s OWL” project (“Intelligent Technical Systems Ostwestfalen-Lippe”, http://www.its-owl. de/ ), a leading-edge cluster of the German Ministry of Education and Research. We thank the information extraction and synthesis laboratory (IESL) at the University of Massachusetts Amherst for their support.</p> <h2>References</h2> <p>853 854</p>

Opinion mining is often regarded as a classification or segmentation task, involving the prediction of i) subjective expressions, ii) their target and iii) their polarity. Intuitively, these three variables are bidirectionally interdependent, but most work has either attempted to predict them in isolation or proposing pipeline-based approaches that cannot model the bidirectional interaction between these variables. Towards better understanding the interaction between these variables, we propose a model that allows for analyzing the relation of target and subjective phrases in both directions, thus providing an upper bound for the impact of a joint model in comparison to a pipeline model. We report results on two public datasets (cameras and cars), showing that our model outperforms state-ofthe-art models, as well as on a new dataset consisting of Twitter posts.

Publication:

Bi-directional Inter-dependencies of Subjective Expressions and Targets and their Value for a Joint Model

Related Fields of Study

Citations

References