This builds on my post from two years ago, ChatGPT Is Not A Source Extractor. My, how things change.
Last time I wrote :
They're still plagued with frequent hallucinations, since they're not grounded in anything so that they have no internal distinction between a verifiable, observable truth and the CPU-equivalent of a random brain-fart firing of the neurons.
Which was reflected in the results of that test, in which I fed ChatGPT-4o PV-slices from HI data cubes and asked it to find sources, having essentially no connection with reality.
In these latest tests I proceed in a similar way but using ChatGPT-5.6. Since the new model has more sophisticated features, the tests are considerably simplified : I feed it an image and ask it to mark the positions of sources it thinks are confidentially real HI detections and those it thinks are at least plausible. This is much easier than the old technique of providing it with an alphanumeric coordinate grid, allowing the tests to be much more systematic.
I ran three tests, all with full chat links given below :
1) A casual, "Can you find the sources in this image ?" prompt with unstructured discussion.
2) Systematically feeding in five images, deliberately selected to have different numbers of sources of different visibility levels, with a detailed prompt describing the input data and how to proceed.
3) Following the results of (2), an attempt at recursive self-improvement : ChatGPT itself suggested what extra information it should be given in the prompt to improve its result.
Since I don't have Memory enabled, all tests can be regarded as fully independent.
The first test I found very intriguing because it immediately found 7 out of 8 obvious sources with no false positives :
Because this is a single declination slice, the source might lie farther from the slice centre than the other detections. Beam attenuation could make an intrinsically bright galaxy appear faint. This is particularly important if the plotted slice samples the cube at a single declination rather than displaying a spatially integrated signal.The most useful next step would be to examine adjacent declination slices. A genuine source should exhibit spatial coherence, whereas an isolated noise fluctuation generally will not. That would also help determine whether this faint feature is simply being sampled away from its peak.
Which is entirely correct in this case : this slice is at the edge of the source, hence it appears much fainter than it might otherwise. The other suggestions are, however, perfectly valid ones.
Admittedly, it did have slight "huh ?" further idea :
There is also the possibility that its HI emission is spread over a larger velocity width, reducing its peak brightness without necessarily reducing its integrated flux.
From the image, this is clearly not the case, with the line width being about the same as the other sources here. Technically it's possible that we're seeing just the brightest part of a velocity-extended source, but my inclination would be that ChatGPT likely didn't check its statement against the actual image. "Higher velocity width" is a sensible stock answer for why peak S/N is reduced, but it's not at all obviously the case if the width isn't actually very high.
This led to the systematic tests. The first of these used a more detailed description of what it was looking at in terms of the data and the expected nature of detected sources. This resulted in a sort "not bad for a first go" attempt; if it was a student, I'd be confident I could train them to do a decent job. It didn't make any false positives, but it missed a few faint sources and mislabelled some really quite bright features as of lower confidence. Sometimes it would also have a glitch wherein its requested explanation of what it found wasn't reported in the thread, though this feels like an interface problem more than an issue with the model itself.
That result suggested that ChatGPT might be more capable than in this first more carefully-controlled result. It seemed to have decided to prioritise reliability, whereas for source extraction, what I'd normally want on a first pass is completeness : reliability control comes later. It's important to begin with a generous catalogue, because it's far easier to filter out crappy sources at this stage than it is to have to redo the extraction and find them again if they're missed on the first run (though this does depend on what your goal is, of course). It also seemed to be placing too much weight on the spatial extent of the sources, which for largely unresolved data like this is nearly irrelevant.
Having fed that all back to ChatGPT, it suggested additions to the prompt to explain that completeness should be prioritised and a description of the pixel and beam size added to properly emphasise the effects of resolution. Hence, the second systematic test, using the same images as before.
The results of this ? Quite honestly, I'd have to say human or near-human quality, which is high praise from someone who visually inspected 8,500 data cubes by eye just to see how good humans really are (the answer, it turns out, is very). Now it did much better at distinguishing confident from merely plausible detections. It made a few false positives, but from single slices of the data this is almost unavoidable for a human : when you have the full cube, it's much easier to see if a feature is just a transient alignment of noise or persistent across multiple slices. About half of its 15 less secure detections were real features which are much more easily visible in other parts of the data.
I found only one false negative (a real source that ChatGPT missed), though a caveat here is that I haven't done a rigorous check for these... but given that this is a data set I know extremely well, I would be surprised if there were more. Against this, ChatGPT actually found one real source with a confirmed optical counterpart that had never been catalogued in this data set ! A few more are also at the level where they can't be immediately dismissed. So yeah, taken at face value, ChatGPT is doing human-level visual source extraction.
... not that this means it can replace anyone at this though. For starters, I like looking at my data sets, so I simply wouldn't use it for this anyway. Actually getting to grips with the data in this way is by far the best way to learn about it, to see at a glance how the sources relate to each other and what weird features the data itself contains. For another, it took 2–7 minutes to analyse each image; for a human, the assessment time is probably of the order of seconds and only the recording time bottlenecks this (and even then probably only to something like 30 seconds or so). Given a cube of 100 slices each about five times larger than these ones, we're talking about something like a full 50 hours work to catalogue the entire cube; a properly trained and equipped human could do it in less than 10.
Still, in two years, we've gone from "total gobshite" to "entirely competent but a bit slow". My own take is that this should temper the hype on both ends. No, it's not the hallucinatory madman it was two years ago and it can't be treated as such any more. But also no, with the resources ordinary users have access to, it's not poised to take over anything. Sure, it may be vastly more capable if you through more compute at it, but that's not remotely economically feasible on a larger scale. It's a transformative tool, to be sure, but I don't lie awake at night wondering if AI is going to start finding all the sources for me.
Case in point : I could redo these tests with ChatGPT-6 "Astra", but Astra's token consumption means that this would waste resources I need for doing things that are actually productively useful. I'd still rather hire a human postdoc, thanks... not that I have the money for that either, mind you.
No comments:
Post a Comment