Shapley value, a concept from cooperative game theory, has become a sharp tool in both model interpretability and data valuation. Due to its #P-hardness in general tasks, there are extensive approximating techniques proposed for cost allocation in game theory, model interpretability, or data valuation. However, there lacks a systemic and standardized benchmarking system for these approximating techniques.
In this paper, we introduce OpenSV, an easy-to-use and unified benchmark framework that empowers researchers and practitioners to apply and compare various Shapley value approximation algorithms. OpenSV provides an integrated environment that includes:
- (i) a diverse collection of tasks from game theory, model interpretability, and data valuation,
- (ii) implementations of eleven different state-of-the-art Shapley value algorithms, and
- (iii) a flexible API that can import any tasks like model training in scikit-learn.
We perform benchmarking analysis using OpenSV, quantifying and comparing the efficacy of state-of-the-art Shapley value approximation approaches. We find that no single algorithm performs uniformly best across all tasks, and an appropriate algorithm should be employed for a specific task. OpenSV is publicly available at https://github.com/ZJU-DIVER/OpenSV with comprehensive documentation. Furthermore, we provide a leaderboard with a visualized trade-off between effectiveness and efficiency where researchers can evaluate existing data valuation algorithms and choose among them for demands easily.
