Natural Language Processing (NLP) is an interdisciplinary subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language, in particular how to program computers to process and analyze large amounts of natural language data. The goal is a computer capable of 'understanding' the contents of documents, including the contextual nuances of the language within them. The technology can then accurately extract information and insights contained in the documents as well as categorize and organize the documents themselves.

Text Generation is a subfield of Natural Language Processing (NLP) which attempts to teach machines to automatically generate texts that are indistinguishable from texts written by humans.

<p>a r X i v : 2 1 0 2 . 0 8 8 9 2 v 1 [ c s . C L ] 1 7 F e b 2 0 2 1</p> <h1>THEaiTRE 1.0: Interactive generation of theatre play scripts ⋆</h1> <h2>Abstract. We present the first version of a system for interactive gen-</h2> <p>eration of theatre play scripts. The system is based on a vanilla GPT-2 model with several adjustments, targeting specific issues we encountered in practice. We also list other issues we encountered but plan to only solve in a future version of the system. The presented system was used Keywords: Theatre plays · Natural language generation · GPT-2.</p> <h2>1 Introduction</h2> <p>The THEaiTRE project 5 [13] aims to produce and stage the first computergenerated theatre play on the occasion of the 100th anniversary of Karel ˇ Capek’s play R.U.R. [2], in which the word “robot” first appeared. In this paper, we describe the THEaiTRobot 1.0 tool, which allows the user to interactively generate scripts for individual theatre play scenes. The tool is based on the GPT-2 XL [11] generative language model, using the model without any fine-tuning, as we found that with a prompt formatted as a part of a theatre play script, the model usually generates continuations that fit the format well. However, we encountered numerous problems when generating the script in this way. We managed to tackle some of the problems with various adjustments, but some of them remain to be solved in a future version. Our tool was used to generate the script for a new play, AI: Kdyˇz robot p´ıˇse hru ( AI: When a robot writes a play ), which is planned for an online premiere on 26th February 2021. 6 Although there were various forms of human intervention when generating the script, we estimate that over 90% of the text comes from the automated tool; moreover, most of the interventions were similar to those a dramaturge and director would do in case of a human-written script (making cuts, rearranging lines, reassigning characters, adding scenic remarks, minor edits of the lines, etc.). The GPT-2 model was found unfit for generating long and complex texts such as a full play script; we therefore generated several individual scenes and then a dramaturge joined them into a full play. We have published a video showing the operation of THEaiTRobot 1.0, a sample of its outputs, and its source codes: – Video: https://youtu.be/ksrZouM7Wyg – Sample outputs: http://bit.ly/theaitre-samples – Source codes: http://hdl.handle.net/11234/1-3507 [12]</p> <h2>2 The Generation Process</h2> <p>The process of generating a theatre play scene script starts by the user (a theatre dramaturge in our case) defining the start of the scene, typically a setting and several initial lines of dialogue ; for the first play, we defined a set of inputs revolving around a common topic to ensure some basic coherence of the whole play. The THEaiTRobot tool then uses the vanilla GPT-2 XL model to generate continuing lines. The user has the option to discard any generated line (together with all subsequent lines), prompting the tool to generate a different continuation. The user can also manually enter a line into the script, which becomes part of the input for GPT-2 (this option was used only rarely). The tool itself is implemented as a web application with a server backend, using the Huggingface Transformers library [15].</p> <h2>2.1 Resolved Issues</h2> <p>Set of characters The model does not work with a limited set of characters naturally and tends to forget characters and invent new characters too often. We resolve this by modifying the next token probability distribution within the GPT-2 model, so that at the start of a new line, only tokens corresponding to character names present in the input prompt are allowed. We also boost probabilities of characters that have not spoken for a long time. Repetitiveness GPT-2’s generation may get stuck in a loop, generating one or several lines again and again. We managed to resolve this by modifying the hyperparameters of GPT-2, changing repetition penalty from 1.00 to 1.01. As a backup, we also automatically discard any generated repeated lines and prompt the model to generate another continuing line. Limited context The variant of the GPT-2 model which we are using has a limit of 1024 subword tokens, within which both the input prompt and the generated output must fit. The typical solution is to crop the input at the beginning so that it fits into the window with sufficient space for generating the output. However, this means forgetting potentially important information from the input prompt and the previously generated text, which can lead to an unwanted continual topic drift and also to generating contradictory text; the text is still locally consistent, but as a whole it may be inconsistent. To handle this issue, we introduce automated extractive summarization into the process, hoping that the summarization algorithm will identify the most important pieces of information to remember. Whenever the input for GPT2 (the input prompt + the so far generated script) exceeds a preset limit of M = 924 tokens, 7 we summarize the input using TextRank 8 [6] before feeding the input into the GPT-2 model: – We keep all lines within the last R = 250 tokens from the input 9 to ensure local consistency. – We summarize all the preceding lines into N = 5 lines (while keeping their original order) to ensure global consistency. – We concatenate the summary and the kept lines. – If the resulting text is still longer than M tokens, we crop it at the beginning to M tokens. Machine translation The GPT-2 model operates on English, while we want to generate a Czech script. We therefore automatically translate the generated script using the CUBBITT [10] neural translation model. As the translation tends to discard character names from the lines, we add them by identifying them in the input and translating them independently.</p> <h2>3 Unresolved issues and future plans</h2> <p>Generating a whole play The model is not able to generate a long and complex text such as a full theatre play script. To resolve this, we intend to generate the script hierarchically, first generating a synopsis for the whole play, then expanding it into synopses for individual scenes, and finally generating each scene individually based on its synopsis. This approach is inspired by the work of Fan et al. [3,4], who take a similar coarse-to-fine approach to story generation. Our situation is, however, more complex, as we plan to use one more step of the hierarchy. Character personalities The characters in the play do not seem to have independent personalities in the generated script; the model seems to simply ensure consistency with already generated text, not taking the character names into account. The character personalities thus appear to switch and merge. We intend to resolve this by learning theatre character embeddings and using them to condition the language model. We plan to resolve this by clustering our data into several basic character personality types [1], then train separate character-aware language models, either by finetuning the GPT-2 model, or by using adapter models [5,14]. Dramatic situations The text is generated word by word and line by line, whereas human authors of theatre plays typically operate on a more abstract level, such as dramatic situations [9]. 10 While there is some work on identifying dramatic turning points [8,7], it is too coarse-grained for our application. We are thus currently annotating a corpus of theatre play scripts with a modified set of dramatic situations, and plan to enhance the tool with this abstraction, either by adding one more layer in the hierarchical setup, or by using special tokens or embeddings to mark dramatic situations in the generated text. Machine translation issues The MT model we use is tuned for news text, not theatre scripts, and translates each sentence independently. This leads to various issues, including errors in morphological gender (which should pertain to the character), variance in the honorific T–V distinction (which may vary but should be consistent for each pair of characters), and erroneous sentence splitting. We intend to tackle these issues by using a document-level translation system which takes larger context into account, fine-tuning the model on a corpus of theatre play scripts, and adding various heuristic modifications where necessary.</p> <h2>4 Conclusion</h2> <p>We have developed THEaiTRobot 1.0, a tool for interactively generating theatre play scripts. The tools is based on GPT-2, with several modifications targeting encountered issues. We have also discussed persisting issues and suggested remedies for a future version. We used the tool to create the first predominantly machine-generated theatre play script, which is planned for a premiere on 26th February 2021. Another play, to be generated by an improved version of the tool, is planned for January 2022. 5</p> <h2>References</h2>

We present the first version of a system for interactive generation of theatre play scripts. The system is based on a vanilla GPT-2 model with several adjustments, targeting specific issues we encountered in practice. We also list other issues we encountered but plan to only solve in a future version of the system. The presented system was used to generate a theatre play script planned for premiere in February 2021.

Publication:

THEaiTRE 1.0: Interactive Generation of Theatre Play Scripts

Related Fields of Study

Citations

References