I know, I know, I get far less than my proverbial five-a-day so far as reading papers goes. Let me try and make some small amends.
Today, a brief overview of a couple of visualisation papers I read while I was finishing off my own on FRELLED, plus a third which is somewhat tangentially related.
The first is a really comprehensive review of the state of astronomical visualisation tools in 2021. Okay, they say it isn't comprehensive, which is strictly speaking true, but that would be an outright impossible task. In terms of things at a product-level state, with useable interfaces, few bugs and plenty of documentation, this is probably as close as anyone can realistically get.
Why is a review needed ? Mainly because with the "digital tsunami" of data flooding our way, we need to know which tools already exist before we go about reinventing the wheel. As they say, there are data-rich but technique-poor astronomers and data-poor but technique-rich visualisation experts, so giving these groups a common frame of reference is a big help. And as they say, "science not communicated is science not done". The same is true for science ignored as well, of which I'm extremely guilty... you can see from the appallingly-low frequency of posts here how little time I manage to find for reading papers.
So yeah, having everything all together in one place makes things very much easier. They suggest a dedicated keyword in papers "astrovis" to make everything easier to find. As far as I know this hasn't been adopted anywhere, but it's a good idea all the same.
Most of the paper is given to summarising the capabilities of assorted pieces of software, some of which I still need to check out properly (and yes, they include mine, so big brownie points to them for that !). But they've also thought very carefully about how to organise all this into a coherent whole. For them there are five basic categories for their selected tools : data wrangling (turning data into something suitable for general visualisation), exploration, feature identification, object reconstruction, and outreach. They also cover the lower-level capabilities (e.g. graph plotting, uncertainty visualisation, 2D/3D, interactivity) without getting bogged-down in unproductively pigeon-holing everything.
Perhaps the best bit of pigeon-unholing is something they quote from another paper : the concept of explornation, an ugly but useful word meaning the combination of exploration and explanation. This, I think, has value. It's possible to do both independently, to go out looking at stuff and never getting any understanding of it at all, or conversely to try and interpret raw numerical data without ever actually looking at it. But how much more powerful is the combination ! Seeing can indeed be believing. The need for good visualisation tools is not only about making pretty pictures (although that is a perfectly worthwhile end in itself) but also in helping us understand and interpret data in different ways, every bit as much as developing new techniques for raw quantification.
I also like the way they arrange things here because we too often tend to ignore tools developed for different purposes other than our own field of interest. And they're extraordinarily non-judgemental, both about individual tools and different techniques. From personal experience it's often difficult to remain so aloof, to avoid saying, "and we should all do it this way because it's just better". Occasionally this is true, but usually what's good for one person or research topic just isn't useful at all for others.
On the "person" front I also have to mention that people really do have radically different preferences for what they want out of their software. Some, inexplicably, genuinely want everything to do be done via text and code and nothing else, with only the end result being shown graphically. Far more, I suspect, don't like this. We want to do everything interactively, only using code when we need to do something unusual that has to be carefully customised. And for a long time astronomy tools have been dominated too much by the interface-free variety. The more that's done to invert the situation, the better, so far as I'm concerned.
The second paper presents a very unusual overlap between the world of astronomy and... professional athletes. I must admit this one languished in my reading list for quite a while because I didn't really understand what it was about from a quick glance at the abstract or text, mostly because of my own preconceptions : I was expecting it to be about evaluating the relative performance of different people at source-finding. Actually this is (almost) only tangential to the main thrust of the paper, though it's my own fault for misreading what they wrote.
Anyway, professional sports people train themselves and others by reviewing their behaviour using dedicated software tools. One of the relatively simple features that one of these (imaginatively named "SPORTSCODE") has is the ability to annotate videos. This means that those in training can go back over past events and see relevant features, e.g. an expert can point out exactly what and where something of interest happened – and thereby, one hopes, improve their own performance.
What the authors investigate is whether astronomers can use this same technique, even using the same code, to accomplish the same thing. If an expert marks on the position of a faint source in a data cube, can a non-expert go back and gain insight into how they made that identification ? Or indeed if they mark something they think is spurious, will that help train new observers ? The need for this, they say, is that ever-larger data volumes threaten to make training more difficult, so having some clear strategy for how to proceed would be nice. They also note that medical data, where the stakes are much, much higher, relies on visual extraction, while astronomical algorithms have traditionally been... not great. "Running different source finders on the same data set rarely generates the same set of candidates... at present, humans have pattern recognition and feature identification skills that exceed those of any automated approach."
Indeed. This is a sentiment I fully endorse, and I would advocate using as much visual extraction as possible. Nevertheless, my own tests have found that more modern software can approach visual performance in some limited cases, but a full write-up on that is awaiting the referee's verdict.
While this paper asks all the right questions, it presents only limited answers. I agree that it's an interesting question as to whether source finding is a largely inherent or learned (teachable) skill, but most of the paper is about the modifications they made to SPORTSCODE and its setup to make this useful. The actual result is a bit obvious : yes indeed, annotating features is useful for training, and subjectively this feels like a helpful thing to do. I mean... well yeah, but why would you expect it to be otherwise ?
I was hoping for some actual quantification of how users perform before and after training – to my knowledge nobody has ever done this for astronomy. We muddle through training users as best we can, but we don't quantify which technique works best. That I would have found a lot more interesting. As it is, it's an interesting proof of concept, and it asks all the right questions, but the potential follow-up is obvious and likely much more interesting and productive. I also have to point out that FRELLED comes with all the tools they use for their training methods, without having to hack any professional athletes (or their code) to get them to impart their pedagogical secrets.
The final paper ties back into the question of whether humans can really outperform algorithms. I suppose I should note that these algorithms are indeed truly algorithms in the traditional, linear, procedural sense, and nothing at all to do with LLMs and the like (which are simply no good at source finding). What they try to do here is use the popular SoFiA extractor in combination with a convolutional neural network. SoFiA is a traditional algorithm, which for bright sources can give extremely reliable and complete catalogues, but it doesn't do so well for fainter sources. So to go deeper, the usual approach is to use a human to vet its initial catalogues to reject all the likely-spurious identifications.
The authors don't try to replace SoFiA with a neural network. Instead they use the network to replace this human vetting stage. Don't ask me how neural networks work but apparently they do. I have to say that while I think this is a clever and worthwhile idea, the paper itself leaves me with several key questions. Their definition of signal to noise appears contradictory, making it hard to know exactly how well they've done : it isn't clear to me if they're really used the integrated S/N (as they claim) or the peak S/N (as per their definition). The two numbers mean very different things. It doesn't help that the text is replete with superlatives, which did annoy me quite a bit.
The end result is clear enough though, at least at a qualitative level : this method definitely helps, but not as much as visual inspection. It's interesting to me that they say this can fundamentally only approach but not surpass humans. I would expect that a neural network could be trained on data containing (artificial) sources so faint a human wouldn't spot them, but knowing they were there, the program could be told when it found them and thereby learn their key features. If this isn't the case, then it's possible we've already hit a fundamental limit, that when humans start to dig into the noise, they're doing about as well as it's ever possible to do by any method. When you get to the faintest features we can find, there simply aren't any clear traits that distinguish signal from noise. Actually improving, in any significant way, on human vision, might be a matter of a radically different approach... but it might even be an altogether hopeless challenge.
And that's nice, isn't it ? Cometh the robot uprising, we shall make ourselves useful by doing astronomical source-finding under the gentle tutelage of elite footballers.
Or not, because that algorithms can be thousands of times faster can more than offset their lower reliability levels, but that's another story.
Phew ! Three papers down, several hundred more to go.