Abstract

Precision and Recall are two prominent metrics of generative performance, which were proposed to separately measure the fidelity and diversity of generative models. Given their central role in comparing and improving generative models, understanding their limitations are crucially important. To that end, in this work, we identify a critical flaw in the common approximation of these metrics using k-nearest-neighbors, namely, that the very interpretations of fidelity and diversity that are assigned to Precision and Recall can fail in high dimensions, resulting in very misleading conclusions. Specifically, we empirically and theoretically show that as the number of dimensions grows, two model distributions with supports at equal point-wise distance from the support of the real distribution, can have vastly different Precision and Recall regardless of their respective distributions, hence an emergent asymmetry in high dimensions. Based on our theoretical insights, we then provide simple yet effective modifications to these metrics to construct symmetric metrics regardless of the number of dimensions. Finally, we provide experiments on real-world datasets to illustrate that the identified flaw is not merely a pathological case, and that our proposed metrics are effective in alleviating its impact.

Metadata

publication

International Conference on Machine Learning, 16326-16343, 2023

year

2023

publication date

2023

authors

Mahyar Khayatkhoei, Wael AbdAlmageed

link

https://proceedings.mlr.press/v202/khayatkhoei23a.html

resource_link

https://proceedings.mlr.press/v202/khayatkhoei23a/khayatkhoei23a.pdf

conference

International Conference on Machine Learning

pages

16326-16343

publisher

PMLR