> Details

Sarafoglou, Alexandra; Hoogeveen, Suzanne; van den Bergh, Don; Aczel, Balazs; Albers, Casper J.; Althoff, Tim; Botvinik-Nezer, Rotem; Busch, Niko A.; Cataldo, Andrea M.; Devezer, Berna; van Dongen, Noah N. N.; Dreber, Anna; Fried, Eiko I.; Hoekstra, Rink; Hoffman, Sabine; Holzmeister, Felix; Huber, Jürgen; Huntington-Klein, Nick; Ioannidis, John; Johannesson, Magnus; Kirchler, Michael; Loken, Eric; Mangin, Jan-Francois; Matzke, Dora; [...]

Subjective evidence evaluation survey for many-analysts studies

Reference
management

Bookmarks

Remove from
bookmarks

Share this on Whatsapp

Close

Bookmarks



You can manage bookmarks using lists, please log in to your user account for this.
  • Media type: E-Article
  • Title: Subjective evidence evaluation survey for many-analysts studies
  • Contributor: Sarafoglou, Alexandra; Hoogeveen, Suzanne; van den Bergh, Don; Aczel, Balazs; Albers, Casper J.; Althoff, Tim; Botvinik-Nezer, Rotem; Busch, Niko A.; Cataldo, Andrea M.; Devezer, Berna; van Dongen, Noah N. N.; Dreber, Anna; Fried, Eiko I.; Hoekstra, Rink; Hoffman, Sabine; Holzmeister, Felix; Huber, Jürgen; Huntington-Klein, Nick; Ioannidis, John; Johannesson, Magnus; Kirchler, Michael; Loken, Eric; Mangin, Jan-Francois; Matzke, Dora; [...]
  • Published: The Royal Society, 2024
  • Published in: Royal Society Open Science, 11 (2024) 7
  • Language: English
  • DOI: 10.1098/rsos.240125
  • ISSN: 2054-5703
  • Origination:
  • Footnote:
  • Description: Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.
  • Access State: Open Access