2023

ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metrics

Chantal Amrhein, Nikita Moghe, Liane Guillou

Summary

ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metrics. Contrastive challenge sets based on MQM Ontology for Translation Accuracy. 36,476 examples, 146 language pairs, 68 phenomena. Evaluates WMT 2022 and 2023 metrics. No clear winner among metrics.

Themes

llm

Keywords

machine translation, evaluation metrics, ACES, MQM, challenge sets

Poster

Click image to open full size