Making journals better

Here's my naive and simple idea as to how to reform peer review so as to reduce the "publish or perish" culture without stifling ideas, increase reproducibility of studies and generally make the world a nicer place without spending any money. I guess this one isn't particularly timely - it's been sat in the draft folder for quite a while. I've mentioned this before; I really just wanted to have a slightly more fully-developed and permanent record.

With fierce competition for jobs, it's inevitable that employers resort to using very simple methods to assess candidates : largely publication rate, hence the "publish or perish" guideline. Hence the obvious tactic : publish lots of mediocre papers. And a publication is a publication. There's no way to judge by glancing at a C.V. whether that research was really top-grade or just plain mediocre. But what if there was ?

What I'm proposing is that we try and label the papers as a guideline by which employers can quickly assess performance based on something more than sheer number of publications. I say "label" rather than "grade" because this can be a complex non-linear system. It might, for instance, be useful to label papers according to content. Most regular journals already have a main journal plus a "letters" section which publishes much shorter, timely articles. Why not extend this further ? Instead of just MNRAS Letters, also have MNRAS Observational Catalogues, MNRAS Numerical Simulations, MNRAS Serendipitous Discoveries, MNRAS Data Mining, MNRAS Clickbait, MNRAS Essays, MNRAS Breakthroughs, MNRAS Replication Studies, MNRAS Things I Just Thought Up Off The Top Of My Head While I Was On The Toilet, MNRAS Things Some Bloke In A Pub Said Last Tuesday, etc.

Papers could be labelled not only by content but also review rigour - not to be confused with research quality, because that's not the same thing. Indeed this might be necessary under the new system. If more complex papers are to be seen as more valuable, they'll need more careful review. All levels of peer review are going to need some basic guidelines, which will require some thinking about what we want journal-based peer review to actually mean. Currently, reviewers are given a free hand to request whatever changes they like.

For instance, the lowest level of review (for an "essay" paper, maybe) might be a single referee doing a check to make sure there are no internal inconsistencies, known problems with the methodology, factual errors etc. For the highest level, there might be three or more independent reviewers checking everything with a fine-toothed comb, and they'd be expected to check everything.

Replication studies are a time-consuming procedure, with a high potential just to confirm the previous findings and not learn anything new.One way to offset this would be to award replication studies an extra level of prestige : insist that these studies be subject to the highest level of review possible. Getting such a paper accepted would be a real challenge and recognised as such. So there would be a motivational balance of glory on the one hand, difficulty and low likelihood of new discoveries on the other. A successful replication study could also have a transformative effect on the original paper, changing it from a merely interesting result to one that deserves strong attention. That in turn encourages everyone to publish research which can be replicated in the first place.

https://astrorhysy.blogspot.com/2017/03/this-is-not-crisis-youre-looking-for.html https://astrorhysy.blogspot.com/2017/03/this-is-not-crisis-youre-looking-for.html