The Dataset-Similarity-Based Approach to Select Datasets for Evaluation in Similarity Retrieval

Most papers on similarity retrieval present experiments exe- cuted on an assortion of complex datasets. However, no work focuses on analyzing the selection of datasets to evaluate the techniques proposed in the related literature. Ideally, the datasets chosen for experimental analysis should cover a variety of properties to ensure a proper evalu- ation; however, this is not always the case. This paper introduces the dataset-similarity-based approach, a new conceptual view of datasets that explores how they vary according to their characteristics. The app- roach is based on extracting a set of features from the datasets to rep- resent them in a similarity space and analyze their distribution in this space. We present an instantiation of our approach using datasets gath- ered by surveying the dataset usage in papers published in relevant conferences on similarity retrieval and sample analyses. Our analyses show that datasets often used together in experiments are more similar than they seem to be at first glance, reducing the variability. The pro- posed representation of datasets in a similarity space allows future works to improve the choice of datasets for running experiments in similarity retrieval.