Dont Label Twice: Quantity Beats Quality when Comparing Binary Classifiers on a Budget | Social Foundations of Computation – Max Planck Institute for Intelligent Systems

Institute Homepage

Institute Homepage Sign In

Research Overview

Social Prediction

Performative Prediction: Past and Future

Difficult Lessons on Social Prediction from Wisconsin Public Schools

Allocation Requires Prediction Only if Inequality Is Low

Digital Platforms, Power and Work

Performative Power

An Engine Not a Camera: Measuring Performative Power of Online Search

Causal Inference from Competing Treatments

Contesting Algorithmic Systems

Algorithmic Collective Action in Machine Learning

Decline Now: A Combinatorial Model for Algorithmic Collective Action

Algorithmic Collective Action in Recommender Systems: Promoting Songs by Reordering Playlists

Algorithmic Fairness

Fairness and Machine Learning: Limitations and Opportunities

Unprocessing Seven Years of Algorithmic Fairness

Science of Machine Learning Benchmarks

ImageNot: A Contrast with ImageNet Preserves Model Rankings

Don't Label Twice: Quantity Beats Quality when Comparing Binary Classifiers on a Budget

A Theory of Dynamic Benchmarks

Evaluating Large Language Models

Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarks

Evaluating Language Models as Risk Scores

Fine-tuning Large Language Models

Training on the Test Task Confounds Evaluation and Emergence

Lawma: The Power of Specialization for Legal Tasks

Social Foundations of Computation Members Publications

Don’t Label Twice: Quantity Beats Quality when Comparing Binary Classifiers on a Budget

Screenshot 2025 02 10 at 08.35.55

It’s widely believed that clean labels are important for benchmarking purposes. Dataset creators therefore often take a majority vote over multiple labels per data point. Contrary to conventional wisdom, we prove that when labels are costly and may contain errors, picking one label per data point works best for comparing the accuracy of two models.

Members

Thumb ticker sm 2022 04 17 6936 crop

Social Foundations of Computation

Florian Dorner

Doctoral Researcher

Thumb ticker sm 20241104 hardt moritz 12 cleaned kleiner

Social Foundations of Computation

Moritz Hardt

Director

Publications

Social Foundations of Computation Conference Paper Don’t Label Twice: Quantity Beats Quality when Comparing Binary Classifiers on a Budget Dorner, F. E., Hardt, M. In Proceedings of the 41st International Conference on Machine Learning (ICML 2024), PMLR, The Forty-First International Conference on Machine Learning (ICML), July 2024 (Published) ArXiv URL BibTeX