翻译质量评价,即对翻译文本进行质量评估,在给定源端输入、目标端参考答案、或两者均有提供的情况下,算法用于评估所生成文本的质量。本单一模型可同时支持提供源端输入(src-only)、目标端参考译文(ref-only)、或者两者均有(src-ref-combined)三种评价场景。
模型由一个预训练语言模型(Pretrained Language Model)和一个前馈神经网络(Feedforward Network)组成。模型首先在伪语料上进行继续预训练,而后在WMT’17-18 Metrics Shared Task数据集上进行微调。
此模型为large版本。若想追求更快推理速度,可使用base版本。
包括提供源端输入(src-only)、目标端参考译文(ref-only)、或者两者均有(src-ref-combined)共三种文本质量评价场景。
from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks
from modelscope.models.nlp.unite.configuration import InputFormat
input = {
'hyp': [
'This is a sentence.',
'This is another sentence.',
],
'src': [
'这是个句子。',
'这是另一个句子。',
],
'ref': [
'It is a sentence.',
'It is another sentence.',
]
}
pipeline_ins = pipeline(task=Tasks.translation_evaluation, model='damo/nlp_unite_up_translation_evaluation_English_large')
print(pipeline_ins(input)['score'])
# [0.73634272813797, 0.7258641123771667]
pipeline_ins.change_input_format(input_format=InputFormat.SRC)
print(pipeline_ins(input)['score'])
# [-0.06169187277555466, 0.21349729597568512]
pipeline_ins.change_input_format(input_format=InputFormat.REF)
print(pipeline_ins(input)['score'])
# [0.8135091662406921, 0.8278040289878845]
WMT’17-18 Metrics Shared Task
见论文
在WMT’19 Metrics Shared Task数据集上进行测试,计算与人工打分的Kendall’s Tau系数。启动src-ref-combined评价功能,结果如下:
Method | Model Backbone | #Params. | En-Tgt | En-Src | En-Exc | Avg |
---|---|---|---|---|---|---|
BLEURT | BERT(En) | 109M | 33.1 | - | - | - |
COMET | XLM-R-large | 565M | 34.5 | 56.6 | 42.2 | 45.6 |
XLM-R+Concat | XLM-R-large | 565M | 33.5 | 56.7 | 44.1 | 45.6 |
UniTE-MUP-base | XLM-R-base | 283M | 35.4 | 55.0 | 43.6 | 45.5 |
UniTE-MUP-large | XLM-R-large | 565M | 35.6 | 57.2 | 46.1 | 47.0 |
注:
@inproceedings{wan-etal-2022-unite,
title = "{U}ni{TE}: Unified Translation Evaluation",
author = "Wan, Yu and
Liu, Dayiheng and
Yang, Baosong and
Zhang, Haibo and
Chen, Boxing and
Wong, Derek and
Chao, Lidia",
booktitle = "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = may,
year = "2022",
address = "Dublin, Ireland",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.acl-long.558",
doi = "10.18653/v1/2022.acl-long.558",
pages = "8117--8127",,
}