Nubia: A SoTA evaluation metric for text generation

栏目: IT技术 · 发布时间: 4年前

内容简介:NUBIA is a SoTA evaluation metric for text generation. It stands for NeUral Based Interchangeability Assessor. In addition to returning an interchangeability score, NUBIA also returns scores for semantic relation, contradiction, irrelevancy, logical agreem

What is NUBIA?

NUBIA is a SoTA evaluation metric for text generation. It stands for NeUral Based Interchangeability Assessor. In addition to returning an interchangeability score, NUBIA also returns scores for semantic relation, contradiction, irrelevancy, logical agreement, and grammaticality.

View Paper | Repo | Notebook |Nubia Backstory | FAQ

Nubia: A SoTA evaluation metric for text generation

How does it work?

Nubia: A SoTA evaluation metric for text generation

Nubia is composed of three modules. The first is neural feature extraction. The three main neural features that power the metric are semantic similarity, logical inference, and sentence legibility. These are extracted by exposing layers from powerful (pretrained) language models: RoBERTa STS for semantic similarity, RoBERTa MNLI for logical inference, and GPT-2 for sentence legibility.

The second module is the aggregator. This module is trained to approximate a function mapping input neural features to a quality score that reflects how interchangeable the sentences are. The objective is to come as close as possible to human evaluation.

The final module is calibration. This is necessary because the aggregator is not bound between 0 and 1, nor does a regressed score comparing a reference sentence with itself always ouput 1. So to calibrate, the output is normalized against the score of the reference sentence compared with itself, and bound between 0 and 1.

How does it perform?

We have been able to achieve SoTA performance in assessing the quality of machine translation and image captioning. Results below are the Absolute Pearson correlations with segment-level human judgments on WMT17 to-English translations.

Nubia: A SoTA evaluation metric for text generation

Results below are the Kendall’s Tau correlations with human judgment on the Flickr 8K dataset.

Nubia: A SoTA evaluation metric for text generation

Nubia is especially powerful against BLEU and ROUGE. The following are the score and label graphs of Nubia, ROUGE-L and BLEU for the entire WMT-2017 segment level set.

Nubia: A SoTA evaluation metric for text generation

Can you show me an example?

The following two sentences mean the same thing:

s1, s2 = "He agreed to a proposal of mine.","He gave his agreement to my proposal."

but the BLEU, ROUGE and BERTScore are low (0.43, 0.28, 0.63) whereas the Nubia score is 0.95

Rouge_1 is: 0.4285714235714286
Rouge_2 is: 0.0
BlEU is: 0.2802636449867835
BERTScore is: (0.5967282652854919, 0.6308560967445374, 0.6143679618835449)

Nubia Score: 0.9504227034094436

Semantic relation: 4.672990322113037/5.0
Percent chance of contradiction: 0.26220036670565605%
Percent chance of irrelevancy or new information: 0.5306123290210962%
Percent chance of logical agreement: 99.20719265937805%

Another example shows how the metrics can keep up despite differences in sentence structure:

Nubia: A SoTA evaluation metric for text generation

When "Impossible!" was changed to "Totally feasible!" , the contradiction score went from 0.073% to 99.5% , even though sentence s1 is structured very differently.

For more examples, check out the repo and colab notebook . To better understand the methodology and motivation check out the paper andNubia’s backstory.


以上所述就是小编给大家介绍的《Nubia: A SoTA evaluation metric for text generation》,希望对大家有所帮助,如果大家有任何疑问请给我留言,小编会及时回复大家的。在此也非常感谢大家对 码农网 的支持!

查看所有标签

猜你喜欢:

本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们

编译器设计

编译器设计

Keith Cooper、Linda Torczon / 郭旭 / 人民邮电出版社 / 2012-12 / 99.00元

深入剖析现代编译器运用的算法和技术 强调代码优化和代码生成 体现编译原理教学的最新理念 本书旨在介绍编译器构造法中的艺术和科学。书中深入分析现代编译器后端所用的算法和技术,重点讨论代码优化和代码生成,详细介绍了用几个编程语言编写的示例等。 Keith D. Cooper 莱斯大学计算机科学系计算工程专业Doerr特聘教授,曾任该系系主任。Cooper博士的研究课题涵盖过程间......一起来看看 《编译器设计》 这本书的介绍吧!

URL 编码/解码
URL 编码/解码

URL 编码/解码

正则表达式在线测试
正则表达式在线测试

正则表达式在线测试

HEX CMYK 转换工具
HEX CMYK 转换工具

HEX CMYK 互转工具