Reputation or peer review? The role of outliers

Authored by Jordi Sabater-Mir, Francisco Grimaldo, Mario Paolucci

Date Published: 2018

DOI: 10.1007/s11192-018-2826-3

Sponsors: European Union

Platforms: NetLogo

Model Documentation: Other Narrative

Model Code URLs: https://github.com/LABSS/PeerReviewGPS

Abstract

We present an agent-based model of paper publication and consumption that allows to study the effect of two different evaluation mechanisms, peer review and reputation, on the quality of the manuscripts accessed by a scientific community. The model was empirically calibrated on two data sets, mono- and multi-disciplinary. Our results point out that disciplinary settings differ in the rapidity with which they deal with extreme events-papers that have an extremely high quality, that we call outliers. In the mono-disciplinary case, reputation is better than traditional peer review to optimize the quality of papers read by researchers. In the multi-disciplinary case, if the quality landscape is relatively flat, a reputation system also performs better. In the presence of outliers, peer review is more effective. Our simulation suggests that a reputation system could perform better than peer review as a scientific information filter for quality except when research is multi-disciplinary and in a field where outliers exist.

Tags

Agent-based model agent-based simulation Evolution Reputation peer review Science Publication Multi-disciplinary science Outliers Information filter