Goldilocks And The Three Ghosts

On the 23rd December 2019 I received a particularly nice early Christmas present : confirmation that my paper had, at long last, been accepted. We submitted it on 18th October 2018 and it went through three(!) reviewers before it was finally accepted, so this was much, much more of a saga than I was expecting. Whether it's more like Goldilocks And The Three Reviewers, or of a visitation of the three ghosts of Christmas, I'm not sure. It's probably best told as a mash-up of the two.

Challenge accepted. Here goes.


Once upon a time, there was a nice little blonde astronomer named Goldilocks who noticed some interesting things in his data that he hadn't noticed before. He decided to write a paper about it.

"Ho hum !", he said to himself. "This should be an easy little write-up, and then I can do something else."

But then he found that there was a weird pattern in his data that didn't make much sense. He decided to run a public poll to make sure he hadn't gone mad. He got about a hundred responses on the so-called "ghost town" that was Google Plus, and everyone agreed that the findings were correct. So he thought about it very carefully and realised that it was probably just an interesting but not terribly unlikely coincidence. He wrote up his paper and sent it to his co-authors for comments. As no-one had much to add, he submitted it to a journal and slept soundly, thinking it should not be a terribly controversial discovery.

That night he awoke with a start. A chill air filled the room and there was a most terrible wailing. Suddenly a ghastly phantom rose from the floorboards and cried,
"Rhyyyyy.... I mean, Gooolldddiiiiiloooooocks ! You shall be visited by three ghostly reviewers to inspect your paaaapppeeerrrr !"
"But why ?", cried Goldilocks. "It's nothing special. I mean, it's nice enough, but it's just some galaxies with stripped tails of gas that are pretty much exactly what we expected to find. Come on, galaxies in the Virgo cluster, losing gas exactly as predicted ? Fun, but hardly a revelation. The only real oddity is the way the streams are pointing, but we've explained that well enough. Why should it need three reviewers ?"
"I doooon't knoooooooow !" wailed the phantom. "They just woooooon't belieeeeeeeve yoooooou ! Expect the first ghost quite soooooon !".

A few weeks later Goldilocks again awoke to the clanking of chains and a mysterious wailing. Creeping downstairs, he found a middle-aged man lounging on his sofa and throwing popcorn at his TV in a carefree fashion.

"Yo," said the man, "Someone's been sleeping in MY bed... ! I'm the Daddy Ghost of Utter Pointlessness. Here's yer report." And with that he vanished.

Goldilocks picked up the report and read it carefully. It was quite long, but it didn't seem too bad at first. It was pretty darn clear that the Ghost really just wanted extra citations to their own papers and didn't understand some very basic concepts from radio astronomy. That was a bit worrying, but easy enough to address. It also came with the dreaded task of "shorten the text", which was, as usual, quite meaningless as it came with no further instructions. "Too many notes", tutted Goldilocks to himself. "But everyone says that, so it's not much to worry about".

More reassuringly, the Ghost didn't ask for anything drastic or express any major scientific skepticism, and the requests to make things "more convincing" seemed quite reasonable : clarifying the improved sensitivity from the new analysis, more labels on figures, that kind of thing. It made sense that one might be a bit skeptical about detecting this many new gas streams, even knowing that the cluster was exactly the environment where one should expect such features, since they were somewhat on the faint side.

Goldilocks did as the seemingly sensible (if rather ignorant and uninformed), Ghost suggested and duly returned the paper. Before long, the Ghost came back with a new report and vanished once more, feeling even less inclined to discourse than the last time.

Goldilocks eagerly read the report and immediately fell into despondency. This wasn't so much pointless as it was downright rude. He'd carefully addressed all the points from the first report and explained things at length in the accompanying letter. Yet the Ghost's report was barely a single paragraph and, worst of all, insisted that Goldilocks hadn't done what was asked. Even on those points that were really simple, like asking for a number which was now very clearly highlighted in bold. And the Ghost had asked for more explanation on the improved sensitivity, while Goldilocks had explained several times that sensitivity wasn't the issue, it was about visualisation. Of course he'd explained the procedure in more detail as well, just to make sure, but the Ghost either just didn't get it or was being deliberately obtuse.

"What on Earth am I to do," said Goldilocks to the co-authors, "when someone asks me what the number is, I tell them 'it's six, six is the number, and the number shall be six' and they insist that I haven't told them what the number is ?"

Goldilocks was both cross and confused. The Ghost's response was hopelessly inconsistent. Whereas before the Ghost seemed a bit concerned if the admittedly quite faint tails were real, now they were wondering if they could have been produced by something other than ram pressure stripping. That was something they easily could have pointed out at the first stage, and adding it now really felt like being strung along. And the Ghost made a bizarre claim that one source, already firmly established in the literature through several other independent observations, was only "probably" real. This was a bit like saying it bricks would only "probably" hurt if you dropped them on your toe. It was pointless.

Goldilocks couldn't see the point of answering a referee report knowing that they might just ignore everything and shift the goalposts again. Especially since they insisted the paper was now longer when it was objectively shorter.

"FFS", said Goldilocks to himself.

After consulting the co-authors, Goldilocks decided to ask the editor what to do. He was a bit disappointed that the editor hadn't already intervened, because the problems with the Ghost's response weren't subtle. They were, in fact, glaringly obvious, and he'd seen editors intervene by themselves in the past with things less blatant than this. Goldilocks complained that addressing this new response wouldn't work, since the referee was so inconsistent and asked for things which were already done and stated very clearly indeed in the main text. Trying to address things raised by someone who would simply ignore you no matter how clearly you stated things was indeed Utterly Pointless.

The editor thought for a while and declared, "Hum ! So, this porridge is a bit hot, is it ? We'll see if we can find some that's a bit cooler". And with that Goldilocks waited for a brand new Ghost.

Some time later Goldilocks again awoke to hear a low moaning. This time there was a slightly older matronly figure sitting in a more dignified position and wearing a monocle. "WoooOOooo !", said she. "Behold, I am the Mummy Ghost of Undue Skepticism. Read my report, mortal, if you dare !". She shook her fist in a dramatic fashion and disappeared.

Goldilocks read the report with some trepidation, but was soon confident he knew what to do. The report wasn't without problems. This Ghost was asking for a figure to be both improved and removed, which was very confusing. However, they were very explicit about their main concerns, which made them a lot easier to address. First, they were worried that some of the streams might not be due to ram pressure stripping. Goldilocks was fine with that, he'd never thought that the situation would be otherwise. Making this clearer was no problem. Second, the Ghost wasn't sure all the streams were even real. That wasn't too big of a deal either, as it was quite straightforward to give their statistical significance and predict how many false streams should be expected in a data set this large (the answer, it turned out, was a healthy zero).

The other referees' comments being minor, Goldilocks soon found a way to measure the statistical significance objectively and clarify that the streams might have multiple formation mechanisms. He didn't really understand why anyone would be hung up on these points though, as it was hardly a breakthrough discovery and plenty of other much stupider papers were floating around in the literature. Surely, he thought, the results are at least solid enough that the rest of the community deserved a look at them. "And anyway," he said to himself, "it's well-known that if you provide enough details to recreate your results, which I bugger well have, it isn't necessary that the referee actually has to agree with your conclusions. They shouldn't reject it unless they can actually find a flaw in the analysis, or better yet they should correct it."

So Goldilocks sent off the report feeling cautiously optimistic that this time he'd succeed. The referee certainly seemed more familiar with radio astronomy, which seemed like a good sign.

Alas ! Some considerable time later, the Ghost re-appeared. "Woe !" she cried with a banshee wail, "I remain unduly skeptical ! This porridge is too cold. Thou hast not addressed my concerns, and I reject thine paper ! May it be cast into the pits of hell !"

Goldilocks was astonished and dismayed. He read the report with contempt. The Ghost had blathered about a few points that made little or no sense, but worst of all he hadn't responded to the correction on the main point - at least, not sensibly. Instead of addressing the whole new section dedicated to assessing statistical significance, which was objective and quantitative, she'd simply said she "understood" it, but thought that "the evidence should be in the images".

This didn't sit well with Goldilocks at all. "Fair enough," he thought to himself, "an objective analysis can absolutely be wrong if the wrong procedure is used or whatnot. But surely in this case someone needs to tell me what the blazes actually is wrong with it, rather than just saying they understand it. If they really understand it, they bloomin' well ought to be able to explain why it's wrong." And he was also more than a little annoyed that they wanted "evidence in the images". All this amounted to the Ghost wanting subjective proof in place of an objective one, without saying what was wrong with the method. Goldilocks was Not Happy.

(He thought about complaining to the editor but decided it would do little good. He also noted that the Ghost claimed to remain skeptical of the "majority" of the streams, but when you added up the number of individual streams they said they were happy with, found that they came to 60% of the total.)

What to do ? Goldilocks was not as despondent as you might think. He'd been working on the analysis for well over a year already, and every time someone had come up with a reason to doubt their existence, the tests had only strengthened the case for the streams. True, two ghastly shades hadn't been convinced, but both appeared to be quite bizarre. The other co-authors were all happy with the result, all of whom were more senior and more experienced than him.

"Right," said Goldilocks. "None of the objections raised by the referees make any sense. Therefore, strange as it is, the only reasonable conclusion is that I'm right and they're wrong. I'm not going to dump more than a year of work on the scrapheap because some spectral nit doesn't understand it. I'm going to submit it to a whole new journal."

Goldilocks did, however, accept that images can often be more persuasive than numbers. So he did a whole new analysis in which he injected fake sources into real data, not only measuring the very few false positives that appeared but also making the same contour plots of them as were presented for the real streams. It was pretty effin' clear that you just didn't get false positives that looked anything like the real streams, exactly as the earlier analysis had shown. And so Goldilocks submitted the paper and once again waited.

And waited.

And waited some more.

Well, actually not really, because this time the Ghost was very prompt.

"Ahhhwoooooo !" cried the spectre. "Behold, I am the Baby Ghost of Precise Instruction ! Read my report with the utmost care and all will be well !"

"Oh spirit," said Goldilocks, "I fear you more than any ghost I have yet witnessed. Can it really be true that you are indeed the Ghost of Precise Instruction ?"

But the miniscule phantom only pointed a spectral finger at the report and disappeared.

Now you must understand that Goldilocks was in a pretty strange mental state but this point. He'd been haunted by three strange spectres all questioning his spectra, and was both quite cross and trepidatious. He didn't doubt himself, but he was highly suspicious that the Ghost would actually do their dang job properly. He read the report quite nervously, and decided to avoid reaching any conclusions, knowing that you can't judge someone until you see how they respond a second time.

Still, it looked promising. There were no clear indications that this doubt thought the porridge was too hot or too cold, only that the height of the chair was a bit off and the window needed oiling. That is, the Ghost didn't seem concerned about whether the streams were real, only that the paper was too long and didn't have a good comparison sample.

"Well, fair enough really," thought Goldilocks. "A comparison sample is a great idea, but unfortunately just not practical. Hopefully the ghost will understand this if I explain in sufficient detail."

The Ghost had, however, provided very Precise Instruction indeed when it came to shortening the manuscript. Goldilocks didn't particularly want to do this, but instructions this clear were difficult to get wrong. Best of all, by the very simple direction to "concentrate on the new results", this made it trivial to extrapolate as to which other parts could be cut. Those few words transformed a task ordinarily fraught with problems into the work of a couple of a days. Soon the paper was five pages shorter and, Goldilocks had to admit, considerably more focused.

"I'm still in two minds about it," said he, "but overall this is probably better. I liked the more detailed original version, but more people are likely to actually read this shorter document."

So Goldilocks made the remaining changes and carefully explained why they couldn't provide a comparison sample, substituting this for a literature search of similar features instead. He poked and prodded his co-authors until they finally gave the go-ahead, and then he submitted the revised paper. And very soon the Ghost returned and said "this porridge is just right !", all was well, and there was dancing in the streets.

"Hooray !" said Goldilocks. "But we've learned some valuable lessons here. First, the rules of refereeing ought to be clearly spelled out and not just left to the referee to make them up however they see fit. You can't just go around saying, 'I don't agree' without providing any justification. Second, editors ought to actively check if both sides follow the rules, and not just act as postmen. Pretty much a year of valuable research time has been wasted dealing with this crap and that needn't have happened. I'm going home."

And with that he stomped off and had a lovely Christmas. The end.