I've just had a second review which is pretty much comparable, if not even worse. It's extremely annoying when the referee asks for something as simple and obvious as a number, you provide that number and highlight it in bold, and the referee refuses to acknowledge this. Being a referee shouldn't entitle you to behave like an obnoxious arsehole.
So how do we prevent this ? I've been thinking for a while that journals should provide a fairly standard set of instructions for reviewers. They wouldn't have to be identical in detail, but they ought to be broadly similar. So if I had my own journal, here's how I would run things. I won't try and give the whole set of conditions, just the ones which are different from how it is in astronomy currently. And I'm only focusing on the review process, not every aspect of submitted papers (e.g. writing style) because that'd take too long.
All parties shall behave with respect, recognising that :
- The authors have invested substantial time and effort into generating a manuscript which they sincerely hold to be correct
- The reviewer has dedicated significant amounts of their own time without compensation in order to provide a valuable scientific service.
The key phrase for a reviewer shall be instruct and justify. They must at all times state what it is they want the authors to do, so that they can proceed clearly and their changes judged as to whether they are appropriate, and explain the need for the changes, so that the authors can respond accordingly.
Instructions to reviewers
The underlying reason for peer review is to ensure that the results presented are scientifically accurate and provide assistance to the authors where necessary. Reviews may be skeptical but always helpful and respectful. Constructive criticism is the order of the day. Criticism for the sake of it shall be disregarded : courtesy costs nothing.
You main purpose is to critique the scientific content of the paper. The methods of investigation used should be sound and the conclusions well-supported by the evidence. If and only if you believe that either of these are seriously flawed, you should request major revisions to the paper. If the paper relies on some fundamental error and likely to require a complete rewrite, you may recommend the paper is rejected.
Both the editor and the authors shall have the right to respond to any request for a rejection before it is finalised. It is good practise to define which of your suggestions are crucial and which are optional, especially on the first or subsequent revisions; authors should be receive fair warning if the referee views their change requests as mandatory.
Every effort should be made to scrutinise the conclusions and methods as much as possible on the first draft. If the authors address the criticisms sufficiently at the first revision, then requesting additional scientific changes is strongly discouraged - except when new issues have arisen due to the changes in the manuscript that were not present previously.
You should distinguish between what the authors are claiming as fact and what they are claiming as interpretation. Statements claimed as fact deserve the most rigorous examination. If the authors clearly state something as a matter of opinion, then this should be viewed with more liberal tolerance unless they are in blatant conflict with the facts. Authors are allowed to present interpretations that others disagree with - the verdict here should primarily rest with the community, not the referee. You are encouraged to suggest that subjective statements be clearly labelled as such, but not to remove them entirely unless there is a strong reason to do so (e.g. excessive unsubstantiated speculation). If this is necessary, reviewers must justify their reasons carefully. Your role is to help the authors write a better paper, not to write it for them.
In terms of style, you should mainly examine the structure of the paper. The paper should follow a basic narrative layout where each section follows logically from the previous. The paper should be concise but clarity is preferred above all : it should be written with the aim that it can and should be read from start to finish, providing sufficient detail that another researcher could (in principle) replicate the results. To this end, all appendices need to be reviewed, but do not count towards the length of the paper - appendices are a useful way to provide technical details that might detract from the narrative flow for the main audience. The decision to remove content entirely or instead move it to an appendix rests with the author and editor.
If the paper is unclear due to a poor layout, but could be improved with re-arrangements (e.g. moving or omitting certain sections or numerous paragraphs), then this will count as a request for moderate revisions. This applies also if the paper appears scientifically correct but lacking in detail.
Reviewers may request the document be shortened but only if they provide explicit directions and an overall purpose, e.g. to focus on the new results, or to avoid repetition. There is no point in complaining that the text is too long unless you state at the very least which sections are too long ! Text should be omitted if it does not provide any useful scientific or explanatory content, if it is excessively speculative, or if it can be replaced with a citation to existing works. Reviewers must justify for each section why they think it should be shortened.
Spotting typographical or language errors is not strictly necessary (this is the role of the editor and typesetter) but helpful if done respectfully. Do not, for instance, question the author's English skills, especially if they may not be a native speaker, but instead simply provide a full list of suggested changes. Vague complaints about the number of typographical errors are insulting to the authors and completely unhelpful. If the paper requires only changes of wording or other similar small modifications, such as labels in figures, then this counts as a request for minor revisions unless they are very extensive, in which case they shall be deemed to be moderate revisions.
In all cases, any requested changes must be as specific as possible and justified. Explaining the reasons for the changes is strongly encouraged, however, providing a commentary with no clear instructions is useless and potentially confusing to the authors. If you dispute any of the author's claims, you should state if they should simply be removed or replaced with something else - and if so, state also the replacement and whether and how this also requires altering the conclusions.
In general, it is not necessary that the referee is convinced the conclusions are correct, so long as the authors (a) clearly differentiate between opinion and fact; (b) provide sufficient instruction that the community can reproduce their results and judge for themselves; and (c) the referee can provide no clear reason the authors are incorrect. It is not sufficient for a referee to simply declare that they are unconvinced, and if they reject the arguments of the authors, they must explain why and do so as directly as possible. If you cannot explain what's wrong with the arguments, the presumption of the editor will be that the referee is biased; if they provide otherwise beneficial critiques which the authors address, then the editor may choose to accept the paper regardless of the referee's recommendation.
The editor has the absolute right to determine whether the reviewer has infringed these guidelines. If so, they may require changes to the report before disseminating it to the authors. If the reviewer fails to modify the report appropriately, they may be rejected and the editor will seek a new referee. Reviewers who repeatedly fail to meet the journal's standards may be removed from further consultation and in extreme cases may lose their right to anonymity.
The role of editors
The scientific editor's main role is to act as the reviewer's referee, not as a second referee for the paper itself. Editors will decide if the initial referee report meets the journal's standards and provide instructions to the reviewer when necessary. Similarly, they will decide if responses to the author's revised manuscripts and accompanying comments are suitable and fair.
They will also adjudicate the exchange between authors and referee. While authors should make every effort to accommodate the requested changes, there are often good reasons why this cannot be done. Authors have the right to respond to all change requests and are not automatically bound to comply with all such requests. However, they should justify their reasons for this, either directly in the manuscript or in their response letter. The editor's role in this is to examine the reviewer's response and ensure that, in the event the reviewer disputes the author's responses, that the reviewer remains fair and abides by the standards outlined earlier : they should address the author's points clearly and directly; they must explain any continued disputes with the authors where appropriate.
In the event of a dispute, both the authors and referee shall have grounds for appeal to the editor. In this case the editor will either attempt to adjudicate the dispute themselves or consult an additional referee, who will be given limited access to the paper sufficient for the adjudication. Each party must then agree to the editor's decision, or, if still unsatisfied : the referee may refuse to continue; the authors may request a new referee for the entire manuscript. If a new reviewer is chosen, the full content of the review process will be made available to them. A change of reviewer is permitted a maximum of twice per manuscript (e.g. three reviewers in total), beyond which the paper shall be rejected.
In all cases of dispute, the final decision rests with the editor and the editor's decision is final.
That wasn't so hard, was it ?
No, it wasn't. Yet currently there are no stated criteria for major/moderate/minor revisions, which makes the status more or less random. Reviewers appear to be able to act on a whim, criticising whatever they like for any reason - sometimes debasing reports to the status of typesetting. We've already got typesetters for that, so there's no need to waste researcher's time on this. On other occasions referees try and twist a paper to say what they want it to say rather than what the author's want. I think this gives a misleading impression of what the author's actually believe, and that's something we should stop. Sometimes it's hard to know if a piece of tortured logic is the fault of the authors or the referee, and I for one would like to know who to blame. And still other times the referee will simply reject an author's argument without any explanation, making it impossible for the authors to properly respond.
In short, some referee's presume they're another helpful co-author trying to improve the paper, while others think of themselves more as God, able to arbitrarily decide which papers are worthy and which aren't. This isn't how it should be.
You might think that stating that everyone should be respectful of each other shouldn't be necessary at this level of professional research. Sadly it is, though insults are rarely as direct as calling each other Mr Stinky Poopy Pants or whatever. What's much more common is selective ignorance. The best referee's report I've ever seen was about 10 pages long and immaculately clear, constructive and detailed. In contrast one of the worst was a mere paragraph which essentially ignored everything we wrote at the first revision - and that's an insulting, unprofessional waste of everyone's time. Hence also the emphasis on providing justifications and explanations.
There's also not really any sort of adjudication process. Sometimes editors do step in and say, "you don't need to do this", but I would like them to be more pro-active about it. Having formal procedures defined as to the purpose of review - what referees are allowed to criticise and where they should step back - as well as making the editor play more of an active role (and less of a mere postman) would help a lot. That way, I hope, we could make the review process what it often is but ought to always be : useful and helpful, critical and skeptical - part of a collective effort of inquiry, not an attempt to disparage others for the sake of it or to make unfounded arguments from authority.
So how do we prevent this ? I've been thinking for a while that journals should provide a fairly standard set of instructions for reviewers. They wouldn't have to be identical in detail, but they ought to be broadly similar. So if I had my own journal, here's how I would run things. I won't try and give the whole set of conditions, just the ones which are different from how it is in astronomy currently. And I'm only focusing on the review process, not every aspect of submitted papers (e.g. writing style) because that'd take too long.
All parties shall behave with respect, recognising that :
- The authors have invested substantial time and effort into generating a manuscript which they sincerely hold to be correct
- The reviewer has dedicated significant amounts of their own time without compensation in order to provide a valuable scientific service.
The key phrase for a reviewer shall be instruct and justify. They must at all times state what it is they want the authors to do, so that they can proceed clearly and their changes judged as to whether they are appropriate, and explain the need for the changes, so that the authors can respond accordingly.
Instructions to reviewers
The underlying reason for peer review is to ensure that the results presented are scientifically accurate and provide assistance to the authors where necessary. Reviews may be skeptical but always helpful and respectful. Constructive criticism is the order of the day. Criticism for the sake of it shall be disregarded : courtesy costs nothing.
You main purpose is to critique the scientific content of the paper. The methods of investigation used should be sound and the conclusions well-supported by the evidence. If and only if you believe that either of these are seriously flawed, you should request major revisions to the paper. If the paper relies on some fundamental error and likely to require a complete rewrite, you may recommend the paper is rejected.
Both the editor and the authors shall have the right to respond to any request for a rejection before it is finalised. It is good practise to define which of your suggestions are crucial and which are optional, especially on the first or subsequent revisions; authors should be receive fair warning if the referee views their change requests as mandatory.
Every effort should be made to scrutinise the conclusions and methods as much as possible on the first draft. If the authors address the criticisms sufficiently at the first revision, then requesting additional scientific changes is strongly discouraged - except when new issues have arisen due to the changes in the manuscript that were not present previously.
You should distinguish between what the authors are claiming as fact and what they are claiming as interpretation. Statements claimed as fact deserve the most rigorous examination. If the authors clearly state something as a matter of opinion, then this should be viewed with more liberal tolerance unless they are in blatant conflict with the facts. Authors are allowed to present interpretations that others disagree with - the verdict here should primarily rest with the community, not the referee. You are encouraged to suggest that subjective statements be clearly labelled as such, but not to remove them entirely unless there is a strong reason to do so (e.g. excessive unsubstantiated speculation). If this is necessary, reviewers must justify their reasons carefully. Your role is to help the authors write a better paper, not to write it for them.
In terms of style, you should mainly examine the structure of the paper. The paper should follow a basic narrative layout where each section follows logically from the previous. The paper should be concise but clarity is preferred above all : it should be written with the aim that it can and should be read from start to finish, providing sufficient detail that another researcher could (in principle) replicate the results. To this end, all appendices need to be reviewed, but do not count towards the length of the paper - appendices are a useful way to provide technical details that might detract from the narrative flow for the main audience. The decision to remove content entirely or instead move it to an appendix rests with the author and editor.
If the paper is unclear due to a poor layout, but could be improved with re-arrangements (e.g. moving or omitting certain sections or numerous paragraphs), then this will count as a request for moderate revisions. This applies also if the paper appears scientifically correct but lacking in detail.
Reviewers may request the document be shortened but only if they provide explicit directions and an overall purpose, e.g. to focus on the new results, or to avoid repetition. There is no point in complaining that the text is too long unless you state at the very least which sections are too long ! Text should be omitted if it does not provide any useful scientific or explanatory content, if it is excessively speculative, or if it can be replaced with a citation to existing works. Reviewers must justify for each section why they think it should be shortened.
Spotting typographical or language errors is not strictly necessary (this is the role of the editor and typesetter) but helpful if done respectfully. Do not, for instance, question the author's English skills, especially if they may not be a native speaker, but instead simply provide a full list of suggested changes. Vague complaints about the number of typographical errors are insulting to the authors and completely unhelpful. If the paper requires only changes of wording or other similar small modifications, such as labels in figures, then this counts as a request for minor revisions unless they are very extensive, in which case they shall be deemed to be moderate revisions.
In all cases, any requested changes must be as specific as possible and justified. Explaining the reasons for the changes is strongly encouraged, however, providing a commentary with no clear instructions is useless and potentially confusing to the authors. If you dispute any of the author's claims, you should state if they should simply be removed or replaced with something else - and if so, state also the replacement and whether and how this also requires altering the conclusions.
In general, it is not necessary that the referee is convinced the conclusions are correct, so long as the authors (a) clearly differentiate between opinion and fact; (b) provide sufficient instruction that the community can reproduce their results and judge for themselves; and (c) the referee can provide no clear reason the authors are incorrect. It is not sufficient for a referee to simply declare that they are unconvinced, and if they reject the arguments of the authors, they must explain why and do so as directly as possible. If you cannot explain what's wrong with the arguments, the presumption of the editor will be that the referee is biased; if they provide otherwise beneficial critiques which the authors address, then the editor may choose to accept the paper regardless of the referee's recommendation.
The editor has the absolute right to determine whether the reviewer has infringed these guidelines. If so, they may require changes to the report before disseminating it to the authors. If the reviewer fails to modify the report appropriately, they may be rejected and the editor will seek a new referee. Reviewers who repeatedly fail to meet the journal's standards may be removed from further consultation and in extreme cases may lose their right to anonymity.
The role of editors
The scientific editor's main role is to act as the reviewer's referee, not as a second referee for the paper itself. Editors will decide if the initial referee report meets the journal's standards and provide instructions to the reviewer when necessary. Similarly, they will decide if responses to the author's revised manuscripts and accompanying comments are suitable and fair.
They will also adjudicate the exchange between authors and referee. While authors should make every effort to accommodate the requested changes, there are often good reasons why this cannot be done. Authors have the right to respond to all change requests and are not automatically bound to comply with all such requests. However, they should justify their reasons for this, either directly in the manuscript or in their response letter. The editor's role in this is to examine the reviewer's response and ensure that, in the event the reviewer disputes the author's responses, that the reviewer remains fair and abides by the standards outlined earlier : they should address the author's points clearly and directly; they must explain any continued disputes with the authors where appropriate.
In the event of a dispute, both the authors and referee shall have grounds for appeal to the editor. In this case the editor will either attempt to adjudicate the dispute themselves or consult an additional referee, who will be given limited access to the paper sufficient for the adjudication. Each party must then agree to the editor's decision, or, if still unsatisfied : the referee may refuse to continue; the authors may request a new referee for the entire manuscript. If a new reviewer is chosen, the full content of the review process will be made available to them. A change of reviewer is permitted a maximum of twice per manuscript (e.g. three reviewers in total), beyond which the paper shall be rejected.
In all cases of dispute, the final decision rests with the editor and the editor's decision is final.
That wasn't so hard, was it ?
No, it wasn't. Yet currently there are no stated criteria for major/moderate/minor revisions, which makes the status more or less random. Reviewers appear to be able to act on a whim, criticising whatever they like for any reason - sometimes debasing reports to the status of typesetting. We've already got typesetters for that, so there's no need to waste researcher's time on this. On other occasions referees try and twist a paper to say what they want it to say rather than what the author's want. I think this gives a misleading impression of what the author's actually believe, and that's something we should stop. Sometimes it's hard to know if a piece of tortured logic is the fault of the authors or the referee, and I for one would like to know who to blame. And still other times the referee will simply reject an author's argument without any explanation, making it impossible for the authors to properly respond.
In short, some referee's presume they're another helpful co-author trying to improve the paper, while others think of themselves more as God, able to arbitrarily decide which papers are worthy and which aren't. This isn't how it should be.
You might think that stating that everyone should be respectful of each other shouldn't be necessary at this level of professional research. Sadly it is, though insults are rarely as direct as calling each other Mr Stinky Poopy Pants or whatever. What's much more common is selective ignorance. The best referee's report I've ever seen was about 10 pages long and immaculately clear, constructive and detailed. In contrast one of the worst was a mere paragraph which essentially ignored everything we wrote at the first revision - and that's an insulting, unprofessional waste of everyone's time. Hence also the emphasis on providing justifications and explanations.
There's also not really any sort of adjudication process. Sometimes editors do step in and say, "you don't need to do this", but I would like them to be more pro-active about it. Having formal procedures defined as to the purpose of review - what referees are allowed to criticise and where they should step back - as well as making the editor play more of an active role (and less of a mere postman) would help a lot. That way, I hope, we could make the review process what it often is but ought to always be : useful and helpful, critical and skeptical - part of a collective effort of inquiry, not an attempt to disparage others for the sake of it or to make unfounded arguments from authority.
No comments:
Post a Comment