Not really all that much about AI in this one, actually.
I think everyone agrees that "publish or perish" is bad, but I don't think the approach suggested here makes much sense. However, I do like the following point very much :
AI is entering a publication system already swollen by proliferation, marked by signs of declining disruptiveness, and under growing pressure at the level of review and evaluation. Under those conditions, scientific papers can acquire negative epistemic value: not because they are wrong, but because the understanding they add no longer compensates for the time and attention they draw away from editors, referees, readers and colleagues trying to place them within what is already known.
Once papers and citations become central to hiring, promotion and funding, while publication itself also becomes a commercial object, proliferation acquires a force of its own. Science fills with large bubbles of urgent-but-not-important writing: work that is timely, legible to evaluators, easy to package and profitable to circulate.
When scarce time and judgment are drained by papers whose contribution no longer justifies what they demand from everyone else, they contribute negatively to the collective production of knowledge, slowing and hampering the rise of the mountain.
Well, I agree ! The author is also careful to say that having lots of incremental papers is not in itself a bad thing. But we're reaching the point where trying to maintain the vast wealth of relevant knowledge needed for a small amount of progress is outlandish. We require typically 15-20 pages or more to describe in meticulous detail what was done, why it was done etc. etc. etc. all for the sake of a small advancement that nobody will care about except a handful or direct competitors who will tear it apart limb from limb... we're burning the candle at both ends, making an enormous amount of work for ourselves both when reading and publishing.
I exaggerate, but only slightly. We also have to spend an inordinate amount of time adhering to strict and and utterly pointless journal standards which do exactly nothing to advance the state of the field and do an awful lot towards making the final paper less readable.
So I agree with the diagnosis. I'm much more skeptical of the suggested treatment, not because I have anything against AI in science (quite the opposite !), but because I don't think this is right approach and won't do anything much to address the problem.
The first concerns the visibility of non-paper contributions, such as code on GitHub, data on Zenodo, curated benchmarks, public notebooks, reproducibility packages, and living syntheses. They have existed for years and are valued by practising scientists. But they have remained second-class citizens: largely invisible to hiring and evaluation committees, and usually legitimized only via a paper that describes them rather than recognized in their own right. AI can change that communication layer.
Can it though ? Only very weakly, I think. It can make a code or other product more intelligible to evaluators, but this won't mean anything if they don't have some box to tick on their reports.
The second concerns time. AI can absorb much of the routine labour that now consumes researchers’ effort: literature mapping, code scaffolding, documentation, reproducibility checks, exploratory work on alternative paths, first-pass synthesis across neighbouring literatures... It is that it can remove some of the weight that now pushes many worthwhile directions out of reach.
I agree with that one. AI – and I mean AI from the last few months or so with very low hallucination rates – is very, very good at turning routine but unintelligible research into something accessible and useable by experts who aren't specialists in that particular field. It can also be trusted at least with grunt-work that produces easily-testable results. Don't want to spend time writing a GUI for your enormously complex script ? I completely get that, it's boring. But you can easily have an AI slap something on which, if not perfect, is still massively better at not having one at all, and can generally be refined to something of a decent standard fairly easily. Thus you end up producing stuff which is not only powerful, but actually useable and accessible to a wider audience instead of the hardcore loonies who insist that everything should be done via the command line for some reason.
The third concerns evaluation. AI can strengthen the front end of review by helping editors, evaluation panels, and funders with triage, novelty checks, literature comparison, technical consistency checks and the detection of obvious pitfalls... This is one of the places where AI can directly counter the danger of negative epistemic value. When scarce time is spent processing papers that add too little in return, knowledge suffers. Review would still take time where needed. What could change is the amount of low-level labour surrounding it, so that more of the community’s limited attention is reserved for contributions that genuinely deserve it.
Here I think this is basically true, but not so much for review itself as for distilling knowledge into what researcher's actually need. I still want a human expert reviewing the paper and checking the whole thing carefully for errors, but once published, AI is very powerful for checking on whether a paper actually contains something I actually need to use. It's just not reasonable to expect authors to fully read hundreds of 20+ page papers in full when producing their own; the vast majority of citations are selected only because of one or two key results in each paper, not because everyone is reading absolutely everything.
I've said it before and I'll say it again. What we need are two main changes, one at the level of the journals, and the second at the level of evaluation. The two are inextricably linked. Instead of publishing just in a regular journal or Science/Nature (i.e. ordinary versus prestigious), we need far more journals and divisions within journals. We need to actively demark papers that required a shittonne of work from those which were rattled off in an afternoon. There is real value in disseminating pure ideas with absolutely no testing, but such a paper shouldn't be held to the same standard as the results of running a huge simulation or cataloguing an enormous set of observations. And we need, therefore, to insist that these different levels of papers – which need different standards of review rigour, clearly and publicly stated (most journals do nothing of the sort, never stating what the reviewer is actually supposed to do and when they should shut up) – are actually accounted for in evaluations. Maybe your department already has lots of hard-working incrementalists and needs someone more creative. Maybe it's the opposite. All have value in the right context.
(Incidentally, my institute does account for non paper-producing duties in our internal evaluations, but I've yet to see this much used in external applications like jobs and grants)
A closely related point is that we probably need to think more about how we want papers to be structured. The prestigious journals tend toward a much more readable format : here are the key results together with the primary reasoning and potential pitfalls in this 6-page report, and here, in this 20+ page appendix, are the deep technical details of how we did it. This makes it massively easier to read the key results if you don't need the gory details, and has no real downsides if the technical stuff is what you're after.
So : papers which are easier to read; papers reviewed to different standards and with different labelled metrics; and evaluations which account for the different value that different types of product bring to the table. That would help a great deal, I think : not so much by publishing less as publishing differently and recasting what it is we actually have to read. AI might help here, but only as a second-order effect. It's not the main route, in my view, to beating "publish or perish" culture to its deserved death with a big stick.
No comments:
Post a Comment