Anmerkungen:
Diese Datenquelle enthält auch Bestandsnachweise, die nicht zu einem Volltext führen.
Beschreibung:
The reproducibility of earlier findings is fundamental to the empirical sciences. Even though this circumstance is widely acknowledged, several systematic large-scale reproducibility reviews showed that many earlier findings, e.g., in computer science, psychology, or the biomedical sciences, are not reproducible. Information Retrieval (IR) is rooted in experimentation, and empirical findings mainly drive the research progress. Therefore, the IR community established several initiatives to understand the reproducibility of earlier works better and provided solutions for better research practices to enforce reproducible research. For instance, dedicated reproducibility tracks at IR conferences report on the reproducibility of previous work, while other works introduce solutions to prepare an experimental setup for reuse. This thesis contributes perspectives on how reproducibility can be evaluated at different levels of validity. The first part of the thesis deals with internal validity covering the scope of system-oriented experimentation. We note that there is no standard approach in IR when evaluating the quality of reimplementations as part of a reproducibility attempt. To this end, this work proposes a more principled approach to reproducibility analysis for system-oriented IR experiments. Building upon an extended version of the PRIMAD taxonomy, we outline how a derived metadata schema can be combined with reproducibility measures to determine the degree and quality of reproduction in a principled way. The second part of the thesis focuses on external validity by considering user variability in an IR experiment. The user's influence in an IR experiment is a key component that allows us to conclude how well the system-oriented findings can be reproduced in a different experimental context. As an alternative to experiments with real users, simulations provide a more cost-efficient, reproducible, and controllable solution to account for the variation of user behavior. Our simulation experiments specifically ...