Journal Article


Amirhassan Monadjemi
Andy Way
Antonio Toral
Arefeh Kazemi


Computer Science

statistical machine translation intrinsic evaluation classification reordering model performance models machine translating

Investigating the relationship between classification quality and SMT performance in discriminative reordering models (2017)

Abstract Reordering is one of the most important factors affecting the quality of the output in statistical machine translation (SMT). A considerable number of approaches that proposed addressing the reordering problem are discriminative reordering models (DRM). The core component of the DRMs is a classifier which tries to predict the correct word order of the sentence. Unfortunately, the relationship between classification quality and ultimate SMT performance has not been investigated to date. Understanding this relationship will allow researchers to select the classifier that results in the best possible MT quality. It might be assumed that there is a monotonic relationship between classification quality and SMT performance, i.e., any improvement in classification performance will be monotonically reflected in overall SMT quality. In this paper, we experimentally show that this assumption does not always hold, i.e., an improvement in classification performance might actually degrade the quality of an SMT system, from the point of view of MT automatic evaluation metrics. However, we show that if the improvement in the classification performance is high enough, we can expect the SMT quality to improve as well. In addition to this, we show that there is a negative relationship between classification accuracy and SMT performance in imbalanced parallel corpora. For these types of corpora, we provide evidence that, for the evaluation of the classifier, macro-averaged metrics such as macro-averaged F-measure are better suited than accuracy, the metric commonly used to date.
Collections Ireland -> Dublin City University -> Publication Type = Article
Ireland -> Dublin City University -> Status = Published
Ireland -> Dublin City University -> Subject = Computer Science: Machine translating

Full list of authors on original publication

Amirhassan Monadjemi, Andy Way, Antonio Toral, Arefeh Kazemi

Experts in our system

Andy Way
Dublin City University
Total Publications: 229
Antonio Toral
Dublin City University
Total Publications: 18