RadioGalaxyNET: Dataset and Novel Computer Vision Algorithms for the Detection of Extended Radio Galaxies and Infrared Hosts

Creating radio galaxy catalogues from next-generation deep surveys requires automated identification of associated components of extended sources and their corresponding infrared hosts. In this paper, we introduce RadioGalaxyNET, a multimodal dataset, and a suite of novel computer vision algorithms designed to automate the detection and localization of multi-component extended radio galaxies and their corresponding infrared hosts. The dataset comprises 4,155 instances of galaxies in 2,800 images with both radio and infrared channels. Each instance provides information about the extended radio galaxy class, its corresponding bounding box encompassing all components, the pixel-level segmentation mask, and the keypoint position of its corresponding infrared host galaxy. RadioGalaxyNET is the first dataset to include images from the highly sensitive Australian Square Kilometre Array Pathfinder (ASKAP) radio telescope, corresponding infrared images, and instance-level annotations for galaxy detection. We benchmark several object detection algorithms on the dataset and propose a novel multimodal approach to simultaneously detect radio galaxies and the positions of infrared hosts.


Introduction
Recent advancements in radio astronomy have enabled us to scan large areas of the sky in a short timescale while generating incredibly sensitive continuum images of the Universe.This has created new possibilities for detecting millions of galaxies at radio wavelengths.For example, the ongoing Evolutionary Map of the Universe (EMU; Norris et al., 2021) survey, conducted using the Australian Square Kilometre Array Pathfinder (ASKAP; Hotan et al., 2021) telescope, is projected to discover more than 40 million compact and extended galaxies in the next five years (Norris et al., 2021;Hotan et al., 2021).Similarly, the Low-Frequency Array (LOFAR; van Haarlem et al., 2013) survey of the entire northern sky is also expected to detect more than 10 million galaxies.With the advent of the Square Kilometre Array (SKA 2 ) radio telescope, which is expected to become operational in the coming years, the number of galaxy detections is expected to increase further, potentially reaching hundreds of millions.Such an enormous dataset will significantly impact our understanding of the physics of galaxy evolution.It will allow us to constrain the theoretical models of the Universe (e.g. the Big Bang model) at unprecedented levels.To capture the full potential of these radio surveys comes the need to redesign the galaxy detection techniques.

Raw Radio
Processed Radio ).The processed radio images highlight the categories of extended radio galaxies, and the bounding boxes denote their total radio extent encompassing all of its components.The infrared images show host galaxies inside the circles.
Radio galaxies are characterized by giant radio emission regions that extend well beyond their structure at visible and infrared wavelengths.While most radio galaxies typically appear as simple, compact circular sources, increasing the sensitivity of radio telescopes result in the detection of more radio galaxies with complex extended structures.These structures typically consist of multiple components with distinct peak radio emissions.Figure 1 displays examples of these extended radio galaxies in the first (raw noisy data) and second (processed data) columns, along with their compact infrared host galaxies in the third column.To construct scientifically useful catalogues of radio galaxies, it is crucial to group the associated components of extended radio galaxies accurately.Currently, visual inspections are used to cross-identify associated radio source components and their infrared host galaxies.This limitation highlights the critical need for developing automated methods, such as machine learning algorithms, to accurately and efficiently cross-identify and group associated components.However, to train and test such algorithms, a large and diverse dataset of labelled radio galaxy images is necessary.Unfortunately, such a dataset is not currently available to train models for the next generation of radio surveys, which poses a significant challenge to developing automated methods for detecting and grouping components of radio galaxies.This paper introduces a novel dataset aimed at addressing the problem of radio galaxy component association.The dataset has been • We introduce the first publicly available dataset curated by professional astronomers that includes state-of-the-art images from a highly sensitive radio telescope and instance-level annotations for extended radio galaxies.
• As a novel addition, our dataset also includes corresponding images of the infrared sky, along with the positional information of the host galaxies.
• We benchmark the object detection algorithms on our dataset to demonstrate the challenge of detecting and associating components of radio galaxies.Additionally, we propose a novel method to detect the positions of infrared host galaxies simultaneously.
2 The Dataset

Radio and Infrared Images
Our dataset contains radio images derived from observations with the ASKAP telescope.We use the Evolutionary Map of Universe pilot survey (EMU-PS; Norris et al., 2021) that covers a sky area of 270 deg 2 , achieving an RMS sensitivity of 25 − 35 µJy/beam at a frequency range of 800 to 1088 MHz, centred at 944 MHz (wavelength of 0.37 to 0.28m, centred at 0.32m).The extended radio galaxies were visually identified by the experts in the 270 deg 2 EMU-PS image.At the same sky locations of radio images, we obtain AllWISE (Cutri et al., 2021) infrared images from the Wide-field Infrared Survey Explorer's (WISE; Wright et al., 2010) W1 band that correspond to 3.4 µm wavelength.We create 3-channel RGB images by combining the processed radio and infrared images.To achieve this, we fill the B and G channels with 8-16 bit and 0-8 bit radio information, respectively.In contrast, the 8-16 bit infrared information is inserted into the R channel.

Annotations
Our dataset comprises four types of annotations: the classification labels for extended radio galaxies, bounding boxes encompassing all components of each radio galaxy, segmentation masks for radio galaxies, and the positions of infrared host galaxies.The comprehensive methodology for source identification will be presented in detail by Yew et al. in prep. (2024).Here, we provide a brief overview of the process.We visually inspected infrared images to determine the infrared host galaxy associated with each radio source.Following the criteria of Fanaroff and Riley (1974), we classified the galaxies as FR-I and FR-II.The unreliable classifications, which can either be FR-I or FR-II in reality, are labelled as FR-x sources.In some cases, barely resolved sources have only one peak outside the central component, we classify them as R (for "resolved") sources.The radio annotations for each galaxy are stored as 'categories', 'bbox', and 'segmentation'.The positions of the infrared hosts are stored as 'keypoints'.The statistics for the train, validation, and test data splits, including the number of objects in one frame, categories of extended radio galaxies, and the occupied area of labeled objects, are depicted in Figure 2. Additionally, Table 1 provides a comparison with the existing MiraBest (Miraghaei and Best, 2017) and Citizen Science RGZ (Wu et al., 2019)    The process of obtaining annotations for our dataset took nearly 1.5 years, involving multiple discussions over each source, marking the first such dataset in radio astronomy that utilizes such extensive scientific resources.

Experiments
We propose a novel multimodal modelling approach to simultaneously detect radio galaxies and their corresponding infrared hosts by incorporating keypoint detection in existing object detection algorithms.Note that the multimodal methods are tailored to specific tasks.Here we have radio images where galaxies appear larger due to extended emission, while in infrared images, the same galaxies look like point objects (as depicted in columns 2 and 3 of Figure 1).To the best of our knowledge, there are no specific models that deal with objects that look completely different in two image modalities.As a result, we introduce our own approach to multimodal modelling.
We implemented keypoint detection for Gal-DETR (based on DETR; Carion et al., 2020), Gal-Deformable DETR (based on Deformable DETR Zhu et al., 2021), and Gal-DINO (based on DINO; Zhang et al., 2022).Specifically, we implemented keypoint detection to the model, augmentations, and Hungarian matcher and added additional random rotation augmentations during training.We reduced the learning rate to 5 × 10 −5 and the number of queries to 10. Similar changes were made for Gal-Deformable DETR model, where keypoint detection was also implemented in the deformable attention mechanism.For Gal-DINO model, we made the same changes as for Gal-DETR and additionally implemented keypoint detection in the de-noising anchor box mechanism.All networks are trained and evaluated on an Nvidia Tesla P100.Table 2 presents the results of Gal-DETR, Gal-Deformable DETR, and Gal-DINO for bounding box detection of extended radio galaxies and keypoint detection for the positions of infrared host galaxies, evaluated using the COCO evaluation metric.Figure 3

Conclusions
We present a multimodal dataset comprising 2,800 images capturing both radio and infrared sky data, with annotations curated by professional astronomers.The dataset features 4,155 instances of annotations, including class information of extended radio galaxies, bounding boxes encompassing all associated components of each radio galaxy, segmentation masks for radio galaxies, and positions of host galaxies in infrared images.We benchmark various object detection strategies on the dataset and propose a novel method for simultaneously detecting the extended radio galaxies and the positions of infrared host galaxies.The availability of our dataset will facilitate the development of machinelearning methods to detect radio galaxies and infrared hosts in the next generation of radio sky surveys, enabling the creation of efficient multimodal algorithms with a focus on small objects and partial annotations.

Figure 1 :
Figure1: Raw radio (left), processed radio (middle) and processed infrared (right) images with the frame size of 450 × 450 pixels (0.25 • × 0.25 • ).The processed radio images highlight the categories of extended radio galaxies, and the bounding boxes denote their total radio extent encompassing all of its components.The infrared images show host galaxies inside the circles.

Figure 2 :
Figure2: The dataset split distributions.Shown are the distributions of extended radio galaxies in one frame (left), their categories (middle) and the occupied area per galaxy.
displays RGB images and ground truth annotations (first column), ground truth and predicted keypoints as circles and triangles over infrared images (second column) and Gal-DINO Radio Detection (SIOD-DM FR-II: 0.56 FR-II: 0

Figure 3 :
Figure 3: Object detection results: Shown are the processed radio-radio-infrared images and ground truth annotations (first column), ground truth and Gal-DINO keypoint detections as circles and triangles over infrared images (second column), Gal-DINO (third column) class and bounding box predictions over radio images.

Table 1 :
(Lin et al., 2014)for radio galaxy classification.The annotations C, B, S, and K are categories, bounding boxes, segmentation and keypoint labels, respectively.structured in the COCO dataset format(Lin et al., 2014), allowing for straightforward comparison studies of various object detection strategies for the machine learning community.It features 2,800 3-channel images, each containing two radio sky channels, one corresponding infrared sky channel, and 4,155 annotations.To summarize, our work contributes to the following aspects: datasets.

Table 2 :
Bounding box and keypoint detection results on the test set.Model Params Epochs AP AP 50 AP 75 AP S AP M AP L