A SIMULATION OF DISAGREEMENT FOR CONTROL OF RATIONAL CHEATING IN PEER REVIEW

Authored by Francisco Grimaldo, Mario Paolucci

Date Published: 2013-10

DOI: 10.1142/s0219525913500045

Sponsors: European Union

Platforms: Jason

Model Documentation: Other Narrative Flow charts Pseudocode

Model Code URLs: Model code not found

Abstract

Understanding the peer review process could help research and shed light on the mechanisms that underlie crowdsourcing. In this paper, we present an agent-based model of peer review built on three entities - the paper, the scientist and the conference. The system is implemented on a BDI platform (Jason) that allows to define a rich model of scoring, evaluating and selecting papers for conferences. Then, we propose a programme committee update mechanism based on disagreement control that is able to remove reviewers applying a strategy aimed to prevent papers better than their own to be accepted (”rational cheating”). We analyze a homogeneous scenario, where all conferences aim to the same level of quality, and a heterogeneous scenario, in which conferences request different qualities, showing how this affects the update mechanism proposed. We also present a first step toward an empirical validation of our model that compares the amount of disagreements found in real conferences with that obtained in our simulations.
Tags
agent-based simulation peer review Artificial social systems trust reliability and reputation