An n-gram is a contiguous sequence of n items from a given sample of text or speech. N can be any integer number, and the items can be phonemes, syllables, letters, words or base pairs according to the application. For example, in a sentence, a 1-gram (or unigram) would be each individual word, a 2-gram (or bigram) would be each sequence of two words, a 3-gram (or trigram) would be each sequence of three words, and so on.

Syntactic Text Processing is a subfield of Natural Language Processing (NLP) that deals with the analysis of the syntax and grammatical structure of natural language texts.

Machine Translation is a subfield of Multilinguality and Text Generation that deals with developing algorithms and models for automatically translating text or speech from one language to another. It involves analyzing the input text or speech, generating a representation of its meaning in the source language, and then producing a new text or speech in the target language that conveys the same meaning.

Text Generation is a subfield of Natural Language Processing (NLP) which attempts to teach machines to automatically generate texts that are indistinguishable from texts written by humans.

Text Segmentation is a subfield of Syntactic Text Processing that deals with automatically dividing a text into smaller segments or units that can be analyzed or processed separately. It involves developing algorithms and models that can identify and classify the structural or semantic boundaries between different sections or paragraphs in a text, and group them into meaningful and coherent segments.

Multilinguality is a subfield of Natural Language Processing (NLP) that refers to the ability of machines to understand, process, and generate natural language text in multiple languages. Multilinguality is concerned with addressing the challenges that arise due to the linguistic and structural variations that exist across different languages. This includes variations in word order, syntax, grammar, or vocabulary.

Natural Language Processing (NLP) is an interdisciplinary subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language, in particular how to program computers to process and analyze large amounts of natural language data. The goal is a computer capable of 'understanding' the contents of documents, including the contextual nuances of the language within them. The technology can then accurately extract information and insights contained in the documents as well as categorize and organize the documents themselves.

<h1>CHR F ++ : words helping character n-grams</h1> <h2>Abstract</h2> <p>Character n -gram F-score ( CHR F) is shown to correlate very well with human relative rankings of different machine translation outputs, especially for morphologically rich target languages. However, its relation with direct human assessments is not yet clear. In this work, Pearson’s correlation coefficients for direct assessments are investigated for two currently available target languages, English and Russian. First, different β parameters (in range from 1 to 3) are re-investigated with direct assessment, and it is confirmed that β = 2 is the optimal option. Then separate character and word n -grams are investigated, and the main finding is that, apart from character n -grams, word 1-grams and 2-grams also correlate rather well with direct assessments. Further experiments show that adding word unigrams and bigrams to the standard CHR F score improves the correlations with direct assessments, though it is still not clear which option is better, unigrams only ( CHR F+) or unigrams and bigrams ( CHR F++). This should be investigated in future work on more target languages.</p> <h2>1 Introduction</h2> <p>Recent investigations (Popovi´ c, 2015; Stanojevi´ c et al., 2015; Popovi´ c, 2016; Bojar et al., 2016) have shown that the character n -gram F-score ( CHR F) represents a very promising evaluation metric for machine translation, especially for morphologically rich target languages – it is fast, it does not require any additional tools or information, it is language independent and tokenisation independent, and it correlates very well with human relative rankings (RR) (Callison-Burch et al., 2008). In order to produce these rankings, human annotators have to decide which sentence translation is better/worse than another without giving any note about the absolute quality of any of the evaluated translations. This type of human judgment has been the offical evaluation metric and gold standard for all automatic metrics at WMT shared tasks from 2008 until 2016. Another type of human judgment, direct human assessment (DA) (Bojar et al., 2016), has become additional official evaluation metric for WMT -16, and the only one for WMT -17. These assessments consist of absolute quality scores for each translated sentence. Contrary to RR, the relation between CHR F and DA has still not been investigated systematically. Preliminary experiments in previous work (Popovi´ c, 2016) shown that, concerning DA, the main advantage of character-based Fscore CHR F in comparison to word-based F-score WORD F is better correlation for good translations for which WORD F often assigns too low scores. In this work, we systematically investigate relations between DA and both character and word n -grams, as well as their combinations. The scores are calculated for all available translation outputs from the WMT -15 and WMT -16 shared tasks (Bojar et al., 2016) which contain two target languages, English (translated from Czech, German, Finnish, Romanian, Russian and Turkish) and Russian (translated from English), and then compared with DAs on segment level using Pearsons’s correlation coefficient.</p> <h2>2 n -gram based F-scores</h2> <p>The general formula for an n -gram based F-score is: 612 where ngrP and ngrR stand for n -gram precision and recall arithmetically averaged over all n grams from n = 1 to N: • ngrP n -gram precision: percentage of n -grams in the hypothesis which have a counterpart in the reference; • ngrR n -gram recall: percentage of n -grams in the reference which are also present in the hypothesis. and β is a parameter which assigns β times more weight to recall than to precision. WORD F is then calculated on word n -grams and CHR F is calculated on character n -grams. As for maximum n -gram length N, previous work reported that there is no need to go beyond N=4 for WORD F (Popovi ´ c, 2011) and N=6 for CHR F (Popovi´ c, 2015). CHR F++ score is obtained when the word n grams are added to the character n -grams and averaged together. The best maximum n -gram lengths for such combinations are again N=6 for character n -grams and N=2 or N=1 for word n grams, which will be discussed in Section 4.3.</p> <h2>3 Motivation for adding word n -grams to CHR F</h2> <p>A preliminary experiment on a small set of texts reported in previous work (Popovi´ c, 2016) with different target languages and different types of DA 1 shown that for poorly rated sentences, the standard deviations of CHR F and WORD F scores are similar – both metrics assign relatively similar (low) scores. On the other hand, for the sentences with higher human rates, the deviations for CHR F are (much) lower. In addition, the higher the human rating is, the greater is the difference between the WORD F and CHR F deviations. These results indicate that CHR F is better than WORD F mainly for segments/systems of higher translation quality – the CHR F scores for good translations are more concentrated in the higher range, whereas the WORD F scores are often too low. In order to further investigate these premises, scatter plots in Figure 1 are produced for CHR F and WORD F with DA for the Russian → English Figure 1 confirms the findings from previous work, since a number of WORD F values is indeed pessimistic – high DA but low WORD F, whereas CHR F values are more concentrated, i.e. correlate better with DA values. However, these plots raised another question – are CHR F scores maybe too optimistic (i.e. segments with high CHR F score and low DA score)? Certainly not to such extent as WORD F scores are pessimistic, but still, could some combination of character and word n -grams improve the correlations of CHR F?</p> <h2>4 Pearson correlations with direct assessments</h2> <p>In order to explore combining CHR F with word n grams, the following experiments are carried out in terms of calculating Pearson’s correlation coefficient between DA and different n -gram F-scores: 1. As a first step, β parameter is re-investigated for DA, both for CHR F and WORD F in order to check if β = 2 is a good option for DA, too; 2. Individual character and word n -grams are investigated in order to see if some are better than others and to which extent; 3. Finally, various combinations of character and word n -grams were explored and the results are reported for the most promising ones.</p> <h2>4.1 β parameter revisited</h2> <p>Previous work (Popovi´ c, 2016) reported that the best β parameter both for CHR F and for WORD F is 2 in terms of Kendall’s τ segment level correlation with human relative rankings (RR). However, this parameter has not been tested for direct human assessments (DA) – therefore we tested several β in terms of Pearson correlations with DA. It is confirmed that putting more weight on precision is not good, and the results for β = 1,2,3 are reported in Table 1. Both for CHR F and WORD F, the correlations for β = 2,3 are comparable, and better than for β =1. Since there is almost no difference between 2 and 3, and putting too much weight to recall could jeopardise some other applications such as system tuning or system combination (for example, (S´ anchez-Cartagena and Toral, 2016) decided to use CHR F1 because CHR F3 lead to generation of too long sentences), we decided to choose β = 2 which will be used for all further experiments. 613 0 20 40 60 80 100 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 w o r d F 0 20 40 60 80 100 w o r d F human 0 20 40 60 80 100 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 c h r F human "ru-en.cf2.for-plot" 0 20 40 60 80 100 0 1 1.5 2 WORD F points lies in the lower right quadrant, i.e. a number of segments with high DA values has a low WORD F value. On the other hand, CHR F points are more concentrated, especially for morphologically rich Russian. However, are some of them too optimistic? (i.e. segments with high CHR F scores and low DA scores)</p> <h2>4.2 Individual character and word n -grams</h2> <p>Individual n -grams were also investigated in previous work, however (i) only character n -grams and (ii) only compared with RR, not with DA. In this work, we carried out systematic investigation on both character and word n -grams’ correlations with DA, and the results are reported in Table 2. It should be noted that, to the best of our knowledge, word n -grams with order less than 4 have not been investigated yet in the given context of correlations with RR or DA. Implicitly, the METEOR metric (Banerjee and Lavie, 2005) is based on word unigrams with additional information and generally correlates better with human rankings than the BLEU metric (Papineni et al., 2002) based on uni-, bi-, 3and 4-gram precision. The results show that, similarly to the correlations with RR, the best character n -grams are of the middle lengths i.e. 3 and 4. The main finding is, though, that the best word n -grams are the short ones, namely unigrams and bigrams. Following these results for individual n -grams, several different experiments have been carried out, involving different character n -gram weights, combining character and word n -grams with different weights, etc., however no consistent improvements have been noticed in comparison to the standard uniform n -gram weights, not even by removing or setting low weight for character unigrams. The only noticeable improvement was observed when word 4-grams and 3-grams were removed.</p> <h2>4.3</h2> <p>The emergence of CHR F++ Findings reported in the previous section raised the following questions: (i) are word 3-grams and 4-grams the ”culprits” for overly pessimistic behaviour of WORD F described in Section 3? (ii) Could the ”good guys”, i.e. word unigrams and 614 2016/2015 cs-en de-en fi-en ro-en ru-en tr-en en-ru mean CHR F1 .644/.542 .452/.600 .454/.565 .570 .522/.601 .551 .642/.606 .562 CHR F2 .658/ .552 .469/ .605 .457/.573 .581 .534/.613 .556 .661/.624 .574 CHR F3 .660/.552 .472 /.604 .455/.572 .582 .535/.614 .555 .661 /.622 .574 WORD F1 .587/.503 .453/.540 .428/.525 .504 .498/.549 .531 .572/.527 .519 WORD F2 .598/.512 .462/.543 .437/.535 .518 .504/.559 .536 .580/.533 .526 WORD F3 .600/.514 .464/.543 .439/.537 .522 .504/.561 .536 .582/.534 .528 2016/2015 cs-en de-en fi-en ro-en ru-en tr-en en-ru mean chr1-gram .544/.448 .355/.407 .313/.417 .443 .358/.527 .337 .531/.489 .431 chr2-gram .644/.537 .441/.556 .420/.547 .554 .504/.599 .513 .652/.631 .550 .662 /.539 .472 /.604 .459/ .582 .579 .533/ .613 .559 .683/.661 .579 .657/ .542 .472/.614 .460 /.581 .582 .538 /.602 .562 .682/.655 .579 .467/.611 .576 .532/.588 .676/.640 .571 .463/.599 .447/.539 .568 .521/.578 .553 .662/.623 .560 .481/.529 .434/.566 .504 .510 .537 .611/.528 .473/.546 .441/.513 .529 .513/.551 .539 .575/.549 .531 .546/.461 .426/.513 .387/.470 .498 .469/.519 .475 .536/.472 .481 .385/.458 .337/.369 .427 .404/.468 .380 .478/.397 .414 615 bigrams diminish potentially too optimistical be-</p> <h2>haviour of CHR F?</h2> <p>In order to get the answers, the Pearson correlations are calculated for CHR F combined with four WORD Fs with different maximum n -gram lengths, i.e. N=1,2,3,4 and the results are presented in Table 3. In addition, correlations are presented also for CHR F and two variants of WORD F (usual N=4 and the best N=2). First, it can be seen that removing word 3-grams and 4-grams improves the correlation for WORD F which becomes closer to CHR F (and even better for one of the two German → English texts). Furthermore, it can be seen that adding word unigrams and bigrams to CHR F improves the correlations of CHR F in the best way. Therefore this is the variant which is chosen to be the CHR F++. Next best option ( CHR F+) is to add only word unigrams i.e. words, and this one is the best one for translation into Russian. Possible reasons are morphological richness of Russian as well as rather free word order, however the test set in this experiment is too small to draw any conclusions. Both CHR F++ and CHR F+ should be further tested on more texts and on more morphologically rich languages. Scatter plots presented in Figure 2 visualise the improvement of correlations by CHR F++: WORD F with N=4 (a) is, as already shown, too pessimistic. Lowering the maximum n -gram length to 2 (b) moves a number of pessimistic points upwards, thus improving the correlation. When added to slightly overly optimistic CHR F (c), the points for both metrics are moved more towards the middle (d).</p> <h2>5 Conclusions</h2> <p>The results presented in this work show that adding short word n -grams, i.e. unigrams and bigrams to the character n -gram F-score CHR F improves the correlation with direct human assessments (DA). Since the amount of available texts with DA is still small, it is still not possible to conclude which variant is better: adding only unigrams ( CHR F+) or unigrams and bigrams ( CHR F++). This is especially hard to conclude for translation into morphologically rich languages, since only Russian was available until now. In order to explore both CHR F+ and CHR F++ more systematically, both are submitted to the WMT -17 metrics task for translations from English. For translation into English, only CHR F++ is submitted since it outperformed the other variant for English. For Chinese, only the raw CHR F has been submitted since the concept “Chinese words” is generally not clear. Further work should include more data and more distinct target languages. The tool for calculating CHR F++ (as well as CHR F+ and CHR F since it is possible to change maximum n -gram lengths) is publicly available at https://github.com/m-popovic/chrF . It is a Python script which requires (multiple) reference translation(s) and a translation hypothesis (output) in the raw text format. It is language independent and does need tokenisation or any similar preprocessing of the text. The default β is set to 2, but it is possible to change. It provides both segment level scores as well as document level scores in two variants: microand macro-averaged.</p> <h2>Acknowledgments</h2> <p>This work has been supported by the TraMOOC project funded from the European Unions Horizon 2020 research and innovation programme under grant agreement No 644333.</p> <h2>References</h2> <p>616 2016/2015 cs-en de-en fi-en ro-en ru-en tr-en en-ru mean WORD F (4-gram) .598/.512 .462/.543 .437/.535 .518 .504/.559 .536 .580/.533 .526 WORD 2-F .642/.537 .490/.557 .455/.563 .535 .526/.592 .553 .603/.575 .552 CHR F (6-gram) .658/.552 .469/.605 .457/.573 .581 .534/613 .556 .661/.624 .574 C 6 W 4F .656/.555 .483/.598 .470/.580 .572 .538/.608 .573 .665/.630 .577 C 6 W 3F .663/.559 .486/.603 . 471 /.584 .578 .542/.615 .574 .672/.641 .582 C 6 W 2F ( CHR F++) . 668/.561 .487/.606 .470/ .585 .580 .544 /.619 .570 .679/.650 .585 C 6 W 1F ( CHR F+) .665/.558 .480/ .606 .464/ .585 .579 .540/ .620 .562 . 685/.654 .583 617 618</p>

Publication:

chrF++: words helping character n-grams

Related Fields of Study

Citations

References