Semantic Text Processing is a subfield of Natural Language Processing (NLP) that attempts to derive meaning from natural language and helps machines interpret textual data semantically. It involves developing algorithms and models that can understand and represent the semantics of words and sentences, and the relationships between them.

Responsible & Trustworthy NLP is a subfield of Natural Language Processing (NLP) that is concerned with implementing methods that focus on fairness, transparency, trustworthiness, explainability, accountability, and ethical aspects at its core. It involves considering the societal impact of NLP applications and ensuring that they do not perpetuate or amplify biases or discrimination. Additionally, responsible NLP also involves developing methods to ensure the privacy and security of user data, and to mitigate the risks of unintended consequences or misuse of NLP systems.

Language Models are a subfield of Semantic Text Processing that deal with the probabilistic modeling and generation of natural language, based on a learned probability function of sequences of words. Language models generate probabilities by training on corpora of natural language texts and can be fine-tuned for a variety of different downstream tasks.

Natural Language Processing (NLP) is an interdisciplinary subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language, in particular how to program computers to process and analyze large amounts of natural language data. The goal is a computer capable of 'understanding' the contents of documents, including the contextual nuances of the language within them. The technology can then accurately extract information and insights contained in the documents as well as categorize and organize the documents themselves.

<h1>Neural Network Surgery with Sets</h1> <h2>Abstract</h2> <p>The cost to train machine learning models has been increasing exponentially [1], making exploration and research into the correct features and architecture a costly or intractable endeavor at scale. However, using a technique named “surgery” OpenAI Five was continuously trained to play the game DotA 2 over the course of 10 months through 20 major changes in features and architecture. Surgery transfers trained weights from one network to another after a selection process to determine which sections of the model are unchanged and which must be re-initialized. In the past, the selection process relied on heuristics, manual labor, or pre-existing boundaries in the structure of the model, limiting the ability to salvage experiments after modifications of the feature set or input reorderings. We propose a solution to automatically determine which components of a neural network model should be salvaged and which require retraining. We achieve this by allowing the model to operate over discrete sets of features and use set-based operations to determine the exact relationship between inputs and outputs, and how they change across tweaks in model architecture. In this paper, we introduce the methodology for enabling neural networks to operate on sets, derive two methods for detecting feature-parameter interaction maps, and show their equivalence. We empirically validate that we can surgery weights across feature and architecture changes to the OpenAI Five model.</p> <h2>1 Introduction</h2> <p>The computational cost of training neural networks has been shown to be growing at an exponential rate [1]. The ability to repurpose or fine-tune machine learning models for different tasks has in parallel been touted as an approach to reap the benefits of a costly investment in a high capacity model. Starting with word vectors [2], base layers of computer vision models, and lately with generative models, Transformers, and reinforcement learning agents, larger and larger components from a trained model can be used on a new task or modified to satisfy new requirements, and attempts have been made to understand how best to achieve this. For instance, [3] used linear classifier probes to try and determine the utility of layers within a network, while [4] looked at which portions of a model can be transferred. In simpler cases, it has been found that the output of a network is sufficiently similar to an existing task, and thus the majority of the network can be kept as is [5, 6, 7, 8]. While some transfer cases have natural boundaries along which parameters can be transferred, we find that domains with semantically rich and complex input features pose an important difficulty to overcome when attempting to gradually incorporate them into a growing model. In the development of OpenAI Five 3 [9], the input domain would periodically change to incorporate new heroes or a r X i v : 1912 . 06719v2 [ c s . C L ] 19 M a r 2020 [ ] additional observable features that were deemed necessary to master the game across changes in game versions, as depicted in Figure 1. These changes would pose significant challenges for continued training of the model, motivating the development and formalization of a methodology named “surgery” for automatically discovering which parameters to port over. The main contributions of this work are a formalization of the discovery of surgery steps for porting the parameters from one network to another across feature or architecture changes, two implementations using gradients or boolean logic operations, and empirical validation of surgery applied to feature changes in OpenAI Five.</p> <h2>2 Approach 2.1 Surgery Overview</h2> <p>For a change to a model F old with parameters Θ old , we would like to define a mapping M : Θ old → Θ new of model parameters for a new model F new such that where s is a state of the world S , X in old ( s ) and X in new ( s ) are the corresponding featurized inputs to each model at a state s . In order to define this mapping M , we require the following: 1. An ordered and named schema for all input features in X in old ( · ) and X in new ( · ) . 2. A mapping Φ old from each input feature x in i ( · ) ∈ X in old ( · ) to each parameter θ j ∈ Θ old indicating the interaction between the two within the network. A separate mapping Φ new is obtained for features in X in new and parameters in Θ new . 3. The difference between Φ old and Φ new is computed. Changes indicate areas where new parameters need to be inserted to preserve output equivalence between F old and F new .</p> <h2>2.2 Input feature schema</h2> <p>Let X in ( s ) denote the input features for a given state s that is fed into some model F , which takes the form X in ( s ) = [ x max health ( s ) , x attack damage ( s ) , . . . ] . We would then record an index mapping of: { max health → 0 , attack damage → 1 , . . . } . We call this snapshot of feature identifiers to their corresponding indices within the model input X in an input feature schema . For notational simplicity we will omit s from X in past this point.</p> <h2>2.3 Input feature to parameter mapping</h2> <p>Let Θ = { θ i } i ∈L be the set of learnable parameters within a model F , where L is the set of layers within a given model. We need to define a mapping Φ : x in i → θ j ∀ x in i ∈ X in , θ j ∈ Θ which 2 connects input features to the parameters in the model for which these features interact with. Binary operations between inputs 4 and parameters are defined as “interactions”, and we can derive this interaction mapping using either gradients or boolean logic.</p> <h2>2.3.1 Gradient mapping</h2> <p>One method of obtaining Φ is to look for cases where parameters receive a nonzero gradient. For example, in the case of a single fully-connected (FC) layer with an element-wise differentiable activation function f , let us consider the N × M matrix of weights W , a 1 × M vector of biases b , and a 1 × N input vector X . We now have Y = f ( X · W + b ) , where Y is the output of this layer. If we have a cost function C that takes Y as its input: C ( Y ) = C ( y 1 , y 2 , . . . , y M ) , then the gradient of the cost function with respect to the weights of this layer is: and since we have where y k = f (cid:16) (cid:80) N i =1 x i · w i,k (cid:17) , the matrix of partials ∂y k ∂W reduces down to all zeros except for the k -th column. Furthermore, if we were to set the input vector X to be all zeroes except for a 1 at index t , the matrix of partials ∂y k ∂W would consist of simply a single nonzero value at index ( t, k ) . Thus from feeding in inputs X t that is nonzero only at index t , and computing the gradient, we can back out all the elements in W that need to be modified if an element of X was changed, since the nonzero values would all reside on row t of ∂C ( Y ) ∂W . Using gradients to define Φ is practical as it relies on existing auto-differentiation tools [11] to collect information. However, poor initialization and saturating functions can mask interactions. We resolve these issues in practice by initializing (non-bias) weights to be strictly positive, normalizing the inputs into sigmoid/tanh to avoid saturation, and replacing max pool by avg pool . If we were to have more than two layers, we would not be able to distinguish changes to the weight matrix in the second layer under the assumptions given in our surgery setup. That is, given positive weight matrices in our network and the assumption that we do not saturate any of our activation functions, we cannot construct inputs to the second layer that are nonzero at a single index without solving for complex systems of equations. As a result, the gradient of the cost function at the output of the second layer with respect to the weight matrix in the second layer would be nonzero throughout, and we would not be able isolate changes to the second layer parameters as a function of input changes. While the limitation to a single FC layer may appear to be somewhat restrictive by design, we can still sufficiently cover all cases where input features to the model are added or removed. All input features into the model will be fed through some layer after which the output dimension is changed either through a matmul or a concat with another feature vector. In the case of matmul s, we have illustrated above that we can isolate the impact of individual inputs through the index of nonzero gradients. In the concat case, say with the output of two FC layers processed over two sets of inputs, we can similarly isolate changes in the ordering of the concat arguments by looking at the position of feature groups as shown in Figure 4.</p> <h2>2.3.2 Boolean logic mapping</h2> <p>Another method for defining Φ relies on tracking explicitly when and where a particular input feature is involved in a binary operation with a model parameter. This interaction is stored through a list of boolean flags 5 corresponding to each input feature. Every model parameter will maintain its own list 3 [ ] [ ] × Weights Out Weights set of features that affect this output A B Interaction Side effects Output features of boolean flags, where a true value indicates the parameter’s involvement in a binary operation with a given input feature. An illustration of the propagation and update of these feature flags is shown in Figure 3. To create these per-parameter feature flags, we create a copy of the computational graph where the original network operations (matmul, addition, sigmoid, etc...) are implemented using boolean logic rules to track feature interactions between inputs and parameters. The rules are defined as follows. Let F w denote the set of input features x in i ∈ X in that interact with a scalar parameter w , and let F x denote the set of input features that feed directly into an input x . Using the rules shown in Table 1, we can combine feature sets whenever parameters and inputs interact through addition or multiplication. At the end of this process, we obtain a parameter to feature mapping, which we can then do a reverse lookup to create Φ . This process is also illustrated in Figure 3.</p> <h2>2.3.3 Gradient and boolean logic mapping equivalence</h2> <p>We can show that both gradient and boolean logic mappings generate the same Φ by noticing that the existence of a nonzero gradient for a parameter (when a single input is nonzero) relies solely on the outputs of the forward pass up to this point being positive. From (2), we can see that: Since ∂C ∂y b is assumed to be nonzero for all y b , and f (cid:48) is similarly designed to be nonzero, we get: if there exists w a,b ∈ W that is multiplied with x a . It follows that if we were to do a forward pass with an input vector X that is zero except for a 1 at index t , any nonzero outputs of y k ∈ Y would indicate an interaction between x t and w t,k , which is equivalent to the location of nonzero gradients within ∂ C ( Y ) ∂W . In order for x a to be positive it must be either an input feature that was set to 1, or the output of some other layer in the network. In Table 2 we describe how a positive output relies either on any input being positive (in the case of matmuls), or a positive input element at the same index for element-wise operations. We can also observe that the truth table under the positive output conditional is identical to the truth table from conditional yielding the presence of a feature x in i in the output 6 , and thus 4 Interaction Positive output Input feature x in i present in output x 1 · x 2 x 1 > 0 ∧ x 2 > 0 ( x in i ∈ F x 1 ) ∨ ( x in i ∈ F x 2 ) x · w x > 0 x in i ∈ F x x 1 + x 2 x 1 > 0 ∨ x 2 > 0 ( x in i ∈ F x 1 ) ∨ ( x in i ∈ F x 2 ) x + w x > 0 x in i ∈ F x propagation of positivity and the propagation of features operate identically. Gradient and boolean</p> <h2>2.4 Mapping Differences</h2> <p>The final stage for obtaining surgery steps is to compare Φ old with Φ new . We construct a lookup table using Φ that is keyed by feature groups and maps to parameters that interact with them 7 . Whenever new features are introduced, they will alter the feature groups so that Φ old and Φ new will not share this key in their lookup tables. We can use this mismatch to identify parameters that need to be reinitialized. When new sets of features are introduced and concatenated with an existing input, the existing feature group keys will make it obvious that features have shifted location; this location shift information is useful for finding the parameters that should be copied from the old model. Figure 2 is an illustration new feature introduction and shift detection. However, after one FC layer or matmul , features can become mixed (see Out 1 and 2 in Figure 4). making it impossible to track propagation past this point using purely input features. Nonetheless, it is possible to track the movement of feature groups that have been processed in isolation. For instance, in the case of OpenAI Five, information about allied and enemy units was processed by separate network components, and only later concatenated and fed to the main LSTM [9]. This architectural pattern is described pictorially in Figure 4, where Input 1 and 2 go through separate FC layers. Their outputs are later concatenated and multiplied by the same matrix (box A), and we can see how feature group movement, insertion, or deletion is visible in the interaction with this deeper weight matrix.</p> <h2>3 Results</h2> <p>To test whether we can use gradient or boolean logic mapping to automatically obtain surgery steps for transferrings parameters from one model version to the next, we add a group of 12 per-hero features and verify that we are able to preserve most of the trained weights when we use proper initialization and avoidance of zero-gradient functions. 5 We conduct our experiments on the model that was used during the OG-OpenAI Five match. In Table 3 we report the percentage of total feature-parameter interactions detected, as well as the percentage of the original model’s parameters that can be transferred to the new model under different initializations and zero-gradient function replacements. We find that random positive initialization and zero-gradient function replacement achieves the highest number of transferred parameters, and detects all feature-parameter interactions. Random initialization on its own masks interactions (49.42%) and prevents the same degree of transfer. Unlike gradient mapping, boolean logic mapping has the desirable trait that it is agnostic to initialization or the presence of zero-gradient functions. In Section 2.3.3 we established equivalence between gradient and boolean logic mapping, and in Table 3 we empirically verify this holds and observe the the same number of parameters are transferred by both techniques. A further advantage of boolean logic mapping is the ability to trace all feature-parameter interactions simultaneously, which we hypothesize should accelerate the map generation process; we measure time taken by both techniques and notice boolean logic mapping is ≈ 24 times faster (Table 3). The underlying motivation for surgery is the ability to amortize training time while still being able to introduce new features. We find that surgery does indeed reduce the expected overall time to attain professional level play at DotA 2. Specifically, training from scratch (Rerun in [13]) took approximately 2 months, therefore retraining for each of the 20 major surgeries shown in Figure 1 would have taken 40 months, while training with surgery completed in only 10 months.</p> <h2>4 Conclusion</h2> <p>The exponential rise in computation found in machine learning experiments motivates the creation of tools that amortize the cost to train new models. Surgery has emerged as a powerful method for transferring trained weights across model changes and warm starting experiments without having to pay the price of starting from scratch. Knowing which parts of a model should be kept remains a complicated task that requires human labor and expertise. In this paper, we introduce a methodology for automatically suggesting surgical steps, and present how it was successfully used to continuously train OpenAI Five across 20 surgeries over the course of 10 months. We show how we can evaluate a neural network using sets to trace the interaction between features and parameters deep inside the network. Building upon this technique, we present two different methods to generate these interaction maps and show their equivalence. We empirically validate that we can detect 100% of the feature-parameter interactions present in the model when we use a positive initialization and remove zero-gradient functions. By observing the difference in the interaction maps across a feature change we can transfer 98.68% of the parameters automatically. We believe neural network surgery with sets can become a powerful tool to modify and reuse parameters in long running machine learning experiments. An important area for future work is determining how to robustly continue training surgeried models without scrambling transferred weights.</p> <h2>Acknowledgement</h2> <p>We wish to thank the anonymous reviewers for their valuable feedback and remarks, along with Jakub Pachocki and Szymon Sidor for their work on gradient-based surgery. 6</p> <h2>References</h2> <p>7</p>

The cost to train machine learning models has been increasing exponentially, making exploration and research into the correct features and architecture a costly or intractable endeavor at scale. However, using a technique named "surgery" OpenAI Five was continuously trained to play the game DotA 2 over the course of 10 months through 20 major changes in features and architecture. Surgery transfers trained weights from one network to another after a selection process to determine which sections of the model are unchanged and which must be re-initialized. In the past, the selection process relied on heuristics, manual labor, or pre-existing boundaries in the structure of the model, limiting the ability to salvage experiments after modifications of the feature set or input reorderings. 
We propose a solution to automatically determine which components of a neural network model should be salvaged and which require retraining. We achieve this by allowing the model to operate over discrete sets of features and use set-based operations to determine the exact relationship between inputs and outputs, and how they change across tweaks in model architecture. In this paper, we introduce the methodology for enabling neural networks to operate on sets, derive two methods for detecting feature-parameter interaction maps, and show their equivalence. We empirically validate that we can surgery weights across feature and architecture changes to the OpenAI Five model.

Publication:

Neural Network Surgery with Sets

Related Fields of Study

Citations

References