Enhanced Generative Model Evaluation with Clipped Density and Coverage

Technical Deep Dive: Enhanced Generative Model Evaluation with Clipped Density and Coverage

Overview

This work introduces two novel evaluation metrics for generative models, Clipped Density and Clipped Coverage, that address limitations of existing metrics. The key contributions are:

• Robust to outliers: Clipped Density and Clipped Coverage prevent out-of-distribution samples from disproportionately skewing the scores.
• Interpretable absolute scores: The metrics exhibit linear degradation as the proportion of bad samples increases, allowing straightforward interpretation of the results.
• Extensive evaluation: The metrics are tested on a variety of datasets and generative models, outperforming existing approaches in terms of robustness, sensitivity, and interpretability.

Problem & Context

• Deploying generative models in high-stakes applications requires reliable evaluation of synthetic data quality.
• Current evaluation metrics like Fréchet Inception Distance (FID) and FD-DINOv2 provide a single, compound score that does not distinguish between fidelity (realism) and coverage (diversity).
• Existing fidelity and coverage metrics are flawed, lacking robustness to outliers and interpretable absolute scores.

Methodology

The paper introduces two new metrics:

Clipped Density

• Fidelity metric that limits the contribution of individual synthetic samples to prevent over-occurring samples from masking deficiencies.
• Also clips the radii of nearest-neighbor balls used to measure density, mitigating the impact of real outliers.
• Normalized by the score computed on the real data to provide interpretable absolute values between 0 and 1.

Clipped Coverage

• Coverage metric that counts how many real samples have at least one synthetic sample in their nearest-neighbor ball, capping individual contributions at 1.
• Theoretically derived the expected behavior and applied a calibration function to ensure linear degradation as the proportion of bad samples increases.

Data & Experimental Setup

• Tested on synthetic and real-world datasets: CIFAR-10, ImageNet, LSUN Bedroom, FFHQ.
• Compared against a wide range of existing fidelity and coverage metrics.
• Evaluated robustness, sensitivity, and interpretability through various test scenarios:
  • Introducing bad synthetic/real samples
  • Simultaneous mode dropping
  • Translating synthetic Gaussian distributions

Results

• Clipped Density and Clipped Coverage consistently outperformed existing metrics across all test cases.
• Only these two new metrics exhibited the desired properties of robustness, linear score degradation, and interpretable absolute values.
• On real datasets, Clipped Density and Clipped Coverage correlated strongly with human evaluations of generation quality, except for the FFHQ dataset.
• Clipped Density values exceeded 1 in some cases, indicating the generative model targeted a filtered subset of the real distribution rather than the full distribution.

Interpretation

• The careful design of Clipped Density and Clipped Coverage, including capping individual sample contributions and calibrating for linear degradation, enables trustworthy generative model evaluation.
• Absolute interpretability of the scores is crucial for assessing whether a model meets quality thresholds, especially in high-stakes domains.
• Scores above 1 for Clipped Density suggest the model is not targeting the full real distribution, which may be a desirable or undesirable property depending on the application.

Limitations & Uncertainties

• The benchmark of synthetic test cases, while comprehensive, may not cover all possible failure modes.
• Theoretical analysis of the metrics in the infinite sample limit was not provided.
• Other aspects of generative model quality, such as memorization, were not evaluated.
• The choice of embedding model (DINOv2) for image datasets may impact the results, and comparing embedding models remains an open question.

What Comes Next

• Explore extensions to handle memorization, mode dropping, and other quality aspects beyond fidelity and coverage.
• Investigate the theoretical properties of the metrics in the infinite sample limit.
• Apply the metrics to evaluate generative models in diverse domains beyond images, such as text, audio, and video.
• Integrate the metrics into standard benchmarking suites and toolkits to facilitate their widespread adoption.

Sources:

• Enhanced Generative Model Evaluation with Clipped Density and Coverage