FACE: Evaluating Natural Language Generation with Fourier Analysis of Cross-Entropy

Zuhao Yang, Yingfang Yuan, Yang Xu, Shuo Zhan, Huajun Bai +1 more
5/1/2023

No confident code implementation found for this paper.

Abstract

Measuring the distance between machine-produced and human language is a critical open problem. Inspired by empirical findings from psycholinguistics on the periodicity of entropy in language, we propose FACE, a set of metrics based on Fourier Analysis of the estimated Cross-Entropy of language, for measuring the similarity between model-generated and human-written languages. Based on an open-ended generation task and the experimental data from previous studies, we find that FACE can effectively identify the human-model gap, scales with model size, reflects the outcomes of different sampling methods for decoding, correlates well with other evaluation metrics and with human judgment scores.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Cite this paper

@article{yang2023face,
  title  = {FACE: Evaluating Natural Language Generation with Fourier Analysis of Cross-Entropy},
  author = {Zuhao Yang and Yingfang Yuan and Yang Xu and Shuo Zhan and Huajun Bai and Kefan Chen},
  year   = {2023},
  eprint = {2305.10307},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/pdf/2305.10307},
  journal = {NEURIPS 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.10307.svg)](https://codeofpaper.com/paper/2305.10307)

Discussion