Complex systems approach to scientific publication and peer-review system: development of an agent-based model calibrated with empirical journal data

Authored by Michail Kovanis, Raphael Porcher, Philippe Ravaud, Ludovic Trinquart

Date Published: 2016

DOI: 10.1007/s11192-015-1800-6

Sponsors: Sorbonne Paris Cité

Platforms: MATLAB

Model Documentation: Other Narrative Flow charts Mathematical description

Model Code URLs: http://www.clinicalepidemio.fr/peerreview_abm/

Abstract

Scientific peer-review and publication systems incur a huge burden in terms of costs and time. Innovative alternatives have been proposed to improve the systems, but assessing their impact in experimental studies is not feasible at a systemic level. We developed an agent-based model by adopting a unified view of peer review and publication systems and calibrating it with empirical journal data in the biomedical and life sciences. We modeled researchers, research manuscripts and scientific journals as agents. Researchers were characterized by their scientific level and resources, manuscripts by their scientific value, and journals by their reputation and acceptance or rejection thresholds. These state variables were used in submodels for various processes such as production of articles, submissions to target journals, in-house and external peer review, and resubmissions. We collected data for a sample of biomedical and life sciences journals regarding acceptance rates, resubmission patterns and total number of published articles. We adjusted submodel parameters so that the agent-based model outputs fit these empirical data. We simulated 105 journals, 25,000 researchers and 410,000 manuscripts over 10 years. A mean of 33,600 articles were published per year; 19 \% of submitted manuscripts remained unpublished. The mean acceptance rate was 21 \% after external peer review and rejection rate 32 \% after in-house review; 15 \% publications resulted from the first submission, 47 \% the second submission and 20 \% the third submission. All decisions in the model were mainly driven by the scientific value, whereas journal targeting and persistence in resubmission defined whether a manuscript would be published or abandoned after one or many rejections. This agent-based model may help in better understanding the determinants of the scientific publication and peer-review systems. It may also help in assessing and identifying the most promising alternative systems of peer review.

Tags

Epidemiology health Improve Quality Randomized-trial