Natural Language Processing (NLP) is an interdisciplinary subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language, in particular how to program computers to process and analyze large amounts of natural language data. The goal is a computer capable of 'understanding' the contents of documents, including the contextual nuances of the language within them. The technology can then accurately extract information and insights contained in the documents as well as categorize and organize the documents themselves.

Text Generation is a subfield of Natural Language Processing (NLP) which attempts to teach machines to automatically generate texts that are indistinguishable from texts written by humans.

Machine Translation is a subfield of Multilinguality and Text Generation that deals with developing algorithms and models for automatically translating text or speech from one language to another. It involves analyzing the input text or speech, generating a representation of its meaning in the source language, and then producing a new text or speech in the target language that conveys the same meaning.

Multilinguality is a subfield of Natural Language Processing (NLP) that refers to the ability of machines to understand, process, and generate natural language text in multiple languages. Multilinguality is concerned with addressing the challenges that arise due to the linguistic and structural variations that exist across different languages. This includes variations in word order, syntax, grammar, or vocabulary.

<h1>BSC Participation in the WMT Translation of Biomedical Abstracts</h1> <h2>Abstract</h2> <p>This paper describes the machine translation systems developed by the Barcelona Supercomputing (BSC) team for the biomedical translation shared task of WMT19. Our system is based on Neural Machine Translation unsing the OpenNMT-py toolkit and Transformer architecture. We participated in four translation directions for the English/Spanish and English/Portuguese language pairs. To create our training data, we concatenated several parallel corpora, both from in-domain and out-of-domain sources, as well as terminological resources from UMLS.</p> <h2>1 Introduction</h2> <p>In this paper, we present the system developed at the Barcelona Supercomputing Center (BSC) for the Biomedical Translation shared task in the Fourth Conference on Machine Translation (WMT19), which consists in translating scientific texts from the biological and health domain. Our participation in this task considered the English/Portuguese and English/Spanish language pairs, with translations in both directions. For that matter, we developed a machine translation (MT) system based on neural machine translation (NMT), using OpenNMT-py (Klein et al., 2017).</p> <h2>2 Related Works</h2> <p>Previous participation in biomedical translation tasks include the works of Costa-Juss` a et al. (2016) which employed Moses Statistic Machine Translation (SMT) to perform automatic translation integrated with a neural character-based recurrent neural network for model re-ranking and bilingual word embeddings for out of vocabulary (OOV) resolution. Given the 1000-best list of SMT translations, the RNN performs a rescoring and selects the translation with the highest score. The OOV resolution module infers the word in the target language based on the bilingual word embedding trained on large monolingual corpora. Their reported results show that both approaches can improve BLEU scores, with the best results given by the combination of OOV resolution and RNN re-ranking. Similarly, Ive et al. (2016) also used the n-best output from Moses as input to a reranking model, which is based on a neural network that can handle vocabularies of arbitrary size. More recently, Tubay and Costa-juss ˜ A (2018) employed multi-source language translation using romance languages to translate from Spanish, French, and Portuguese to English. They used data from SciELO and Medline abstracts to train a Transformer model with individual languages to English and also with all languages concatenated to English. In the last WMT biomedical translation challenge (2018) (Neves et al., 2018), the submission that achieved the best BLEU scores for the ES/EN and PT/EN, in both directions, were the ones submited by the UFRGS team (Soares and Becker, 2018), followed by the TALP-UPC (Tubay and Costa-juss ˜ A, 2018) in the ES/EN direction and the UHH-DS in the EN/PT directions.</p> <h2>3 Resources</h2> <p>In this section, we describe the language resources used to train both models, which are from two main types: corpora and terminological resources.</p> <h2>3.1 Corpora</h2> <p>We used both in-domain and general domain corpora to train our systems. For general domain data, we used the books corpus (Tiedemann, 2012), which is available for several languages, included the ones we explored in our systems, and the JRCAcquis (Tiedemann, 2012). As for in-domain data, we included several different corpora: • The corpus of full-text scientific articles from Scielo (Soares et al., 2018a), which includes articles from several scientific domains in the desired language pairs, but predominantly from biomedical and health areas. • A subset of the UFAL medical corpus 1 , containing the Medical Web Crawl data for the English/Spanish language pair. • The EMEA corpus (Tiedemann, 2012), consisting of documents from the European Medicines Agency. • A corpus of theses and dissertations abstracts (BDTD) (Soares et al., 2018b) from CAPES, a Brazilian governmental agency responsible for overseeing post-graduate courses. This corpus contains data only for the English/Portuguese language pair. • A corpus from Virtual Health Library 2 (BVS), containing also parallel sentences for the language pairs explored in our systems. Table 1 depicts the original number of parallel segments according to each corpora source. In Section 4.1, we detail the pre-processing steps performed on the data to comply with the task evaluation.</p> <h2>3.2 Terminological Resources</h2> <p>Regarding terminological resources, we extracted parallel terminologies from the Unified Medical Language System 3 (UMLS). For that matter, we used the MetamorphoSys application provided by U.S. National Library of Medicine (NLM) to subset the language resources for our desired language pairs. Our approach is similar to what was proposed by Perez-de Vi˜ naspre and Labaka (2016). Once the resource was available, we imported the MRCONSO RRF file to an SQL database to split the data in a parallel format in the two language pairs. Table 2 shows the number of parallel concepts for each pair.</p> <h2>4 Experimental Settings</h2> <p>In this section, we detail the pre-processing steps employed as well as the architecture of the Transformer.</p> <h2>4.1 Pre-processing</h2> <p>As detailed in the description of the biomedical translation task, the evaluation is based on texts extracted from Medline. Since one of our corpora, the one comprised of full-text articles from Scielo, may contain a considerable overlap with Medline data, we decided to employ a filtering step in order to avoid including such data. The first step in our filter was to download metadata from Pubmed articles in Spanish and Portuguese. For that matter, we used the Ebot utility 4 provided by NLM using the queries POR[la] and ESP[la] , retrieving all results available. Once downloaded, we imported them to an SQL database which already contained the corpora metadata. To perform the filtering, we used the pii field from Pubmed to match the Scielo unique identifiers or the title of the papers, which would match documents not from Scielo. Once the documents were matched, we removed them from our database and partitioned the data in training and validation sets. Table 3 contains the final number of sentences for each language pair and partition.</p> <h2>4.2 NMT System</h2> <p>As for the NMT system, we employed the OpenNMT-py toolkit (Klein et al., 2017) to train three MT systems, one for (Spanish,Portuguese) → English, another one for (English,Spanish) → Portuguese and a third one for (English,Portuguese) → Spanish. Tokenization was performed using the SentecePiece 5 unsupervised tokenizer with a vocabulary size of 32,000. The tokenization was done for each MT system (e.g. concatenated English, Spanish and Portuguese to generate one of the models). The parameters of our network are as follows. Encoder and Decoder: Transformer; Word vector size: 1024; Layers for encoder and decoder: 6; Attention heads: 16; RNN size: 1024; Hidden transformer feed-forward: 4096; Batch size: 4096. To train our system, we used the an IBM cluster with 2 Power-9 CPUs and with four NVIDIA Tesla V100 GPUs. The models with the best perplexity value were chosen as final models. During translation, OOV words were replace by their original word in the source language, all other OpenNMT-py options for translation were kept as default.</p> <h2>5 Results</h2> <p>We now detail the results achieved by our Transformer systems on the official test data used in the shared task. Table 4 shows the BLEU scores (Papineni et al., 2002) for our systems and for the submissions made by other teams. For the ES/EN language pair, we figured in 5 out of 11, while for EN/ES in 4 out of 8. However, one should also take in account the confidence interval of the average of the results. By performing a t-test on the ES/EN results, we found out that the mean of the BLEU scores is 0.4366 ( p -value < 0.01 with confidence interval (95%) between 0.4145 and 0.4857. This means that only the submissions from UCAM can be said to be better than the average. Similarly, the team from UHH-DS is has statistically lower performance than the average. Meanwhile, all other teams, including ours, are statistically tied around the mean, meaning that there is no sufficient information to difference the performance from one system to another. Similarly, for the EN/ES language pair, we performed the same statistical test and achieved p value < 0.01. The reported mean is 0.4675, with confidence interval (95%) between 0.4489 and 0.4861. Thus, Only submissions 2 and 3 from UCAM can be said to be better than average, while the submission from MT-UOC-UPF performed worse than the average. All other teams, including ours, are statistically tied around the mean, without evidence that there is any significant difference among the systems. Unfortunately, no other team participated on the PT/EN and EN/PT language pairs.</p> <h2>6 Conclusions</h2> <p>We presented the BSC machine translation system for the biomedical translation shared task in WMT19. For our submission, we trained three Transformers NMT systems with multilingual implementation for the English/Spanish and English/Portuguese language pairs. For model building, we included several corpora from biomedical and health domain, and from out-of-domain data that we considered to have similar textual structure, such as JRC-Acquis and books. Prior training, we also pre-processed our corpora to ensure, or at least minimize the risk, of including Medline data in our training set, which could produce biased models, since the evaluation was carried out on texts extracted from Medline. Regarding future work, we are planning on optimizing our systems by studying the use of synthetic data from back-translation of monolingual to increase NMT performance (Sennrich et al., 2016) by providing additional training data.</p> <h2>Acknowledgements</h2> <p>This work was supported by the Encargo de Gestion SEAD-BSC of the Spanish National Plan for the Advancement of Language technologies, the ICTUSnet INTERREG Sudoe programme, the European Union Horizon2020 eTransafe (grant agreemed 777365) project, and the Amazon AWS Cloud Credits for Research. Teams Runs ES/EN EN/ES PT/EN EN/PT</p> <h2>References</h2> <p>Marta R Costa-Juss` a, Cristina Espa˜ na-Bonet, Pranava Madhyastha, Carlos Escolano, and Jos´ e AR Fonollosa. 2016.</p>

This paper describes the machine translation systems developed by the Barcelona Supercomputing (BSC) team for the biomedical translation shared task of WMT19. Our system is based on Neural Machine Translation unsing the OpenNMT-py toolkit and Transformer architecture. We participated in four translation directions for the English/Spanish and English/Portuguese language pairs. To create our training data, we concatenated several parallel corpora, both from in-domain and out-of-domain sources, as well as terminological resources from UMLS.

Publication:

BSC Participation in the WMT Translation of Biomedical Abstracts

Related Fields of Study

Citations

References