3.1. Control and monitoring

ASKAP’s control system is built on the Experimental Physics Instrument Control System (EPICS) library.Footnote ^d Each major hardware subsystem has an associated EPICS input/output controller (IOC), which is a software application that issues commands and gathers monitoring information via network transactions. Several servers located at the MRO run these IOCs and there is a dedicated monitoring and control network in addition to the office network and astronomical data network. The design of the control system is described in Guzman & Humphreys (Reference Guzman and Humphreys2010).

The digital signal processing hardware was designed with a common command and control interface that translates instructions from the network into firmware registers on field-programmable gate arrays (FPGAs) throughout the system. A common software library was written to support the hardware layer and this library is used by the EPICS IOCs. Various other commodity electronics devices (such as drive controllers and power supplies) are also used, requiring different network protocols at the IOC level.

3.2. Power infrastructure

Due to its remote location, the MRO is not connected to a mains power grid. The site is supplied by a hybrid solar–diesel power station purpose-built with sufficient RFI shielding to protect the observatory from stray radio emissions. The power station is located approximately 6 km from the core of the telescope to further aid in RFI mitigation. The solar component of the power station can supply 1.6 MW (enough to run the entire site during the day) and is linked to a battery bank with 2.5 MWh capacity. Several redundant diesel generators are the main power source at night.

Power is transmitted via underground high-voltage cable to the central control building. Distribution to the antennas is on six independently switched underground tracks that roughly follow the surface roads and power several antennas each. A transformer at each antenna pad steps the high-voltage input down to three-phase 415 V for the equipment in the antenna pedestal.

3.3. Cooling infrastructure

ASKAP’s digital signal processing hardware in the central control building consumes 280 kW of power and requires a commensurate amount of cooling. This is achieved by circulating chilled water to each electronics cabinet in the central building, where internal heat exchange units extract heat from the air. The cooling system vents this waste heat into a geothermal exchange system. When the air temperature is low, a water-to-air heat exchanger can be put in series, extending the life of the geothermal system. The entire cooling system was designed to minimise radio frequency emissions, as described in Abeywickrema et al. (Reference Abeywickrema, Allen, Ardern, Schinckel, Leitch, Wilson and Beresford2013).

3.4. Network infrastructure

The network between Perth and the MRO consists of approximately 900 km of optical fibre using carrier-grade transmission equipment. The network was designed and is managed jointly by CSIRO and AARNet (the Australian Academic and Research Network). Using dense wavelength division multiplexing, the system can support up to 80 channels on a single fibre pair. Over this distance, optical amplification is required every 80–100 km. Only two of these channels are used for production services, each lit by a 100 Gbit s^-1 transponder split out into ${10 \times {10}\,\mathrm{Gbit\,s^{-1}}}$ services. The first runs direct to the Pawsey Supercomputing Centre in Perth with $4 \times {10}\,\mathrm{Gbit\,s^{-1}}$ used for ASKAP data and one 10 Gbit s$^{-1}$ service to carry MWA data. The second 100 Gbit s^-1 link provides services between Perth, Geraldton, and the MRO such as CSIRO internal network connectivity, IP telephony, and monitor and control traffic.

There are additional transponders to support SKA development activities at Curtin University and for long-haul network performance testing undertaken by AARNet and CSIRO. Currently, there is 500 Gbit s^-1 of lit capacity between Perth and the MRO using 4 of the 80 available channels. There is also a parallel network consisting of two 1 Gbit s^-1 links directly patched to fibre pairs between Geraldton and the MRO. This provides network services at the various locations en route and acts as a backup network should the transmission system fail, however, it does not protect against a fibre cable cut.

The network at the MRO itself is a dual-core design. One core handles all the telescope data and connects the digitisers and beamformers. The second core handles the monitoring and control of the telescope as well as the general purpose network at the MRO. The dual-core allows for separation of the different systems providing better control and resilience. The data network is large, the core router has over 2 400 network ports.

The second core makes extensive use of virtual local area networks (VLANs) to partition traffic. This avoids telescope monitor and control commands being mixed with unrelated network traffic. VLANs provide different network connections at relevant locations across the MRO.

The conditions at the MRO present some challenges to network equipment; the antenna pedestals are not cooled, so industrial switches are used, providing greater reliability and enabling connectivity for the varied range of equipment that has been installed at the observatory. The flexible network design allows for services to be provided to several clients located at the observatory, helping to broaden the research taking place at the site.

4.1. Mechanical design and axes of motion

The antenna structure consists of a steel pedestal and support frame, topped with solid panels made of non-metallic honeycomb sandwiched between aluminium sheets. The feed is located at the prime focus.

Due to the small size of the reflectors and the comparatively large size of the PAF, special consideration was given to the strength and rigidity of the prime focus support structure. The robust feed legs, specified to support a 200 kg PAF receiver, block 4% of the total collecting area with the symmetric antenna design used. However, the symmetric design produces more uniform offset beams, in both total intensity and polarisation.

The use of a PAF with offset beams also makes it necessary to track parallactic angle on the sky during an observation. This is typically done with an equatorial mount, which creates extra mechanical complexity. For ASKAP, a unique hybrid approach was developed. CSIRO designed a simple azimuth–elevation mount that included a third axis of rotation for the reflector itself.Footnote ^e

While it would be possible to track parallactic angle by continuously updating the electronic beamformer weights or rotating just the feed, the third axis exchanges a small amount of mechanical complexity for greatly reduced computational complexity at the time of calibration and imaging, since it maintains the angular relationship of the feed elements with respect to the support structure.

Unlike the azimuth axis, the antenna polarisation axis (also known as the roll axis) does not have overlapping range limits and can only be driven slightly less than $\pm 180^{\circ}$ from its neutral position. This means it is not possible to continuously track northern sources without unwrapping at transit or beginning the track at an offset roll angle.

The antennas were specified to operate up to 10 GHz, providing the flexibility to consider higher frequency receiver upgrades in future. A single ASKAP antenna was successfully used for several years with a single-pixel feed for VLBI observations at 8.4 GHz (Kadler et al. Reference Kadler2016). Preparation for a large-scale receiver system upgrade to higher frequencies should include direct measurements of atmospheric water vapour to assess the suitability of the site itself. Photogrammetry was used to measure both the location of the PAF at the focus and the reflector accuracy after installation. The rms deviation from the expected parabolic shape is typically less than 1 mm. This is consistent with measurements of the first antenna during test assembly at the factory (Feng, Li, & Li Reference Feng, Li and Li2010).

4.2. Wind limits and storm monitoring

The ASKAP antennas were specified to operate under conditions similar to other radio telescopes. The antennas will stow if the peak wind speed exceeds 45km h^-1 during the last 30 min. We find that very little observing time is lost as a result.

Strong winds at the MRO are usually associated with oncoming storm fronts. These can cause the wind speed to rise extremely rapidly, sometimes faster than the time it takes the antennas to stow. On a few occasions, this has led to drive errors during the stow procedure as wind loading exceeded the safe operating range of the motors. To prevent circumstances such as this, we have implemented a storm stow system that uses satellite meteorological data and lightning detectors to sense approaching storm fronts and stow the antennas in advance of their arrival (Indermuehle et al. Reference Indermuehle, Harvey-Smith, Marquarding and Reynolds2018a).

Not all nearby storms lead to high winds at the telescope, but associated lightning activity usually produces extensive radio-frequency interference. False positive stow triggers are, therefore, not a major concern as astronomy data would be impacted by a storm in any case.

4.3. Array configuration

Of the 36 ASKAP antennas, 27 were placed to provide a Gaussian distribution of spatial scales with a point spread function of 30” (Gupta et al. Reference Gupta, Johnston, Feain and Cornwell2008). Three additional antennas were added to the core of the array to increase surface brightness sensitivity and another six antennas were added on longer baselines (up to 6 km) in a Reuleaux triangle (see Figure 4) to provide improved resolution (approximately 10$^{\prime\prime}$) for compact sources. Antennas are plotted relative to ASKAP antenna 1, which is located at $26.6970007225^{\circ}\textrm{S}$, $116.6314242861^{\circ}\textrm{E}$ and an elevation of 361 m.

Figure 4.

Location of each ASKAP antenna plotted relative to Antenna 1. A circle of radius 1 km is drawn for scale. The outer six antennas form a Reuleaux triangle and provide approximately 10$^{\prime\prime}$ resolution. The dense cluster of antennas in the core provides excellent surface brightness sensitivity.

ASKAP has excellent instantaneous (u,v) coverage (Gupta et al. Reference Gupta, Johnston, Feain and Cornwell2008), making snapshot surveys and equatorial imaging possible.

5.1. Design background

The first production prototype PAFs for ASKAP used copper coaxial cables for signal transport, with digitisation inside the antenna pedestal (Hotan et al. Reference Hotan2014). Six of these units were constructed for assessment in the field. Aperture array measurements (Schinckel et al. 2011; Chippendale et al. Reference Chippendale, Hayman and Hay2014), tests at the focus of a 12 m antenna located at the Parkes observatory (DeBoer et al. Reference DeBoer2009; Chippendale et al. Reference Chippendale, O’Sullivan, Reynolds, Gough, Hayman and Hay2010; Sarkissian et al. Reference Sarkissian, Reynolds, Hobbs and Harvey-Smith2017) and the BETA array at the MRO (Hotan et al. Reference Hotan2014) revealed several issues with the Mk I PAF. These included effective system temperatures in excess of 150 k (more than twice the design requirement) in the upper half of the frequency band (McConnell et al. Reference McConnell2016) and the need to fully remove the PAF from the focus of the antenna to perform internal maintenance.

In order to better assess the performance of prototype PAF designs, Chippendale et al. (Reference Chippendale, Hayman and Hay2014, Reference Chippendale2015, Reference Chippendale2016a) developed an aperture array method of determining the noise temperature of a PAF beam using measurements of a microwave absorber, the radio sky, and broadband noise transmitted from a reference antenna. This was a valuable step in the testing process as it helped validate electromagnetic modelling of the PAF that in turn enabled improvements in sensitivity (Shaw, Hay, & Ranga Reference Shaw, Hay and Ranga2012). Aperture array testing was also practical as it allowed rapid testing of PAF prototypes on the ground instead of at the focus of an antenna.

During these early tests, research into RF over fibre showed that the PAF signal could be transported to the central site for processing rather than digitising the signals at the antenna. A major redesign was initiated to incorporate experience from the Mk I tests and take advantage of new technological developments. The ASKAP Design Enhancements (ADE) project developed a Mk II system with lower overall cost, improved maintainability, and effective system temperature less than 80 k across most of the band. LNAs could be changed without removing the PAF from the dish and sensitive electronics were relocated from the antenna pedestal to the climate-controlled central building. All 36 ASKAP antennas are now fitted with Mk II PAFs and digital systems.

Figure 5.

Photograph of a Mk II PAF installed on one of the ASKAP antennas at the MRO. The chequerboard surface is visible, along with the composite case and air vents for the cooling system. Power and optical cables attach via several bulkheads on the side of the case.

5.2. Planar chequerboard connected array

The ASKAP PAFs fill a region of the antenna’s focal plane with small receptors that consist of flat, square conductive patches (see Figure 5) printed on a circuit board substrate to form a planar connected-array antenna (Hay et al. 2007; Hay & O’Sullivan Reference Hay and O’Sullivan2008). The distance between each patch is 90 mm. A row of patches can be considered as a row of bow-tie antennas that are connected edge to edge, forming a linearly connected array. The feed point at the centre of each ‘bow-tie’ is differential and both orthogonal linear polarisations are available due to the two-dimensional nature of the grid. Each element has a radiation pattern that would over-illuminate the dish surface, but the beamforming process combines these elements to create an efficient illumination pattern for a given direction on the sky.

The precise geometry of the chequerboard surface was determined through electromagnetic simulation (Hay, O’Sullivan, & Mittra Reference Hay, O’Sullivan and Mittra2011). Matching the simulation results to experimental data from the Mk I system was a key step to ensure that design modifications could be assessed prior to manufacturing. Co-design of the chequerboard array and its LNAs was critical for achieving high sensitivity. An initial impedance target for the LNA and matching network design was derived by optimising for maximum beamformed sensitivity over the field of view (Hay Reference Hay2010). Slight modifications were made to the shape of the surface elements on the Mk II design to improve the impedance match with the LNAs in the upper observing band (1 400–1 800 MHz), and therefore reduce the system temperature.

The surface panels are bonded to a ground plane via several centimetres of non-conductive honeycomb backing, which provides structural rigidity as well as insulation. Twin-wire feed lines are used to connect the corners of each ‘bow-tie’ element to LNAs housed underneath the ground plane. A conformal coating is applied to the outward-facing surface for protection from dust and weather.

5.3. Composite enclosure

The density of electronics inside the PAF and the sometimes harsh environmental conditions at the focus of the antenna makes the design of the enclosure extremely challenging. As well as keeping the internal systems dry, free of dust, and thermally isolated from the external environment, the enclosure must provide more than 30 dB of extra RF shielding (Beresford & Bunton Reference Beresford and Bunton2013). Experience with the BETA array showed that poorly shielded enclosures led to significant leakage of noise that could correlate, including between PAF elements and nearby antennas.

The Mk II enclosure is a moulded composite structure that is bonded to the metal ground plane and shielded with carbon fibre. Four access hatches at the top are fitted with RF gaskets to reduce radio frequency leakage. The access hatches allow replacement of faulty electronics modules, even when a PAF is installed at the focus of an antenna.

5.4. Dominoes and monitoring

Each pair of LNAs is housed in a self-contained RF shielded enclosure known as a domino. The domino design was developed for the Mk II PAF to improve shielding of the components with the highest gain, and also to improve modularity and maintainability. The dominos are bolted directly to the ground plane to provide conductive heat transfer and can be removed individually for maintenance.

Each domino has three layers that contain differential LNAs, filters, and electro-optical converters stacked in order of distance from the ground plane. A photograph of the interior of each layer is shown in Figures 6, 7, and 8, and a transparent view of the assembled stack is shown in Figure 9. The signal path through the domino layers is shown in the upper left of Figure 10 in the boxes marked LNA, FILTER, and TRANSMITTER.

Figure 6.

Photograph of the low-noise amplifier layer in the domino, referred to elsewhere as LNA.

Figure 7.

Photograph of the filter layer in the domino, referred to elsewhere as FILTER.

The top of each domino provides connections for two fibre-optic cables that carry the analogue RF output, and for a copper ribbon cable that supplies DC power and supports monitoring and control within the PAF. Due to the proximity of the LNAs, front-end monitoring has a very high risk of generating RFI. ASKAP’s PAF monitoring system was designed to be low impact, with no continuous clock signal and RFI filters on all connections to wires within the enclosure. The monitoring system is based on the Serial Peripheral Interface (SPI) bus and is shown in the lower right part of Figure 10. Transmission of data packets may generate RFI, but most monitoring is disabled during an observation, with short updates between scheduling blocks to maintain a record of the system health. Control of each PAF is managed by a front-end controller (FEC) unit that converts instructions from an EPICS IOC into optical signals for the PAF and returns monitoring data to the software layer.

5.5. Ground plane and cooling

The Mk I PAFs employed water cooling inside their enclosures, using a heat exchange unit in the PAF and fans to circulate cool air throughout the electronics. Although this provided good cooling capacity, it required housing a water chiller at each antenna pedestal (with associated RFI issues) and pumping water up to the focus of the antenna. In addition, the possibility of condensation around the chilled water inlet or water leakage inside the PAF enclosure was a significant concern.

Figure 8.

Photograph of the optical transmitter layer in the domino, referred to elsewhere as TRANSMITTER.

Figure 9.

Diagram showing the assembled domino with transparent walls. The LNA card is closest to the bottom of the stack, followed by the filter card in the middle and the optical transmitter at the top.

The Mk II PAFs use heat pipes embedded in the ground plane to transport heat to the edge of the enclosure. Further heat pipes connect to thermoelectric coolers (TECs, as shown in Figure 10 and seen around the edge of the PAF in Figure 5) that transport the heat to the outside of the case where a ducted fan blows ambient air over heat sink fins. This system has proven effective at keeping the internal temperature as much as 15$^{\circ}$C below ambient. It can be regulated via the voltage applied to each TEC. A servo loop keeps the internal temperature at a steady 25$^{\circ}$C set point (which can be changed in software if necessary).

The PAFs were designed to operate in ambient temperatures up to 45$^{\circ}$C. This threshold is rarely exceeded except during the most extreme summer conditions. The cooling system is capable of maintaining safe internal operating temperatures in all conditions experienced to date. However, it is not always possible to keep the internal temperature fixed. The optimal internal temperature of 25$^{\circ}$C is exceeded when ambient temperatures rise above 35$^{\circ}$C, which is quite common during summer. The PAF is fitted with an automatic safety shutdown that is triggered when the internal air temperature exceeds 50$^{\circ}$C.

The exact cooling capacity is slightly different for each unit. When the cooling capacity is exceeded, the internal temperature of the PAF begins to track the external ambient temperature, with a constant offset of approximately 10$^{\circ}$C. This can cause amplitude calibration errors during astronomical data processing as the gain of the analogue electronics depends on their operating temperature. The impact of this on image quality has yet to be determined and it may be possible to correct using time-dependent self-calibration, a temperature-based lookup table, or the on-dish calibration (ODC) system described in the next section.

Figure 10.

The analogue signal path for a single PAF element from the feed to the ADC in the digital receiver. After the chequerboard feed elements, the signal encounters a low-noise amplifier (LNA), followed by switchable bandpass filters (BPF) and power amplifiers (PA). The transmitter module provides an option to introduce phase switching ($\phi_{\textrm{SW}}$), after which the signal is used to modulate a laser diode for transmission over single mode optical fibre (SMF) from the antennas to the control building. After analogue-to-digital conversion (ADC), the digital signal processing (DSP) stage begins. PAF monitoring and control (M$/$C) is distributed to all elements via a system based on the serial peripheral interface (SPI) standard. Cooling inside the PAF is provided by a series of 8 thermoelectric cooler (TEC) modules. Direct current (DC) and extremely low voltage alternating current (ELVAC) powers the various subsystems inside the PAF. The on-dish calibration system consists of a log-periodic dipole antenna (LPDA) mounted at the dish vertex and connected to an optical receive module (ROAR) fed by a noise source (CICADA) located in the front end control (FEC) module. This can be driven synchronously using binary atomic time (BAT) events loaded via a gigabit ethernet (1Ge) interface.

5.6. On-dish calibration

For a dish with a single feed, amplitude calibration is often obtained by placing a switched noise source in the feed that adds a known amount of power to both polarisations when activated. To determine the system amplitude gain, the power induced by the noise source is synchronously demodulated and measured with respect to the background.

Individually injecting a calibration signal into each PAF element is non-trivial due to the number of active elements. Instead, a common noise signal is radiated from the dish vertex to the PAF, illuminating all elements as uniformly as possible. The calibration signal is typically set to inject around 15 k equivalent noise temperature into the PAF, but this can be adjusted within the range 100 mK–200 K as required. We expect to reduce the calibration signal level as we gain experience in its use. The calibration noise is currently on continuously during all observations, but it is largely cancelled by beamforming with maximum signal-to-noise ratio beamformer weights resulting in less than 1% increase to beamformed system temperature (Chippendale, Button, & Lourenço Reference Chippendale, Button and Lourenço2018).

Using a radiated calibration signal has the disadvantage that it can leak into neighbouring antennas, but the signal power is such that the correlated contribution in a nearby antenna is less than that due to leakage of amplifier noise. Such common noise contributions could be further mitigated via Walsh modulation of the phase switches incorporated into each domino module, however this has not been found necessary.

Processing one or two calibration signals is a small engineering overhead compared to the 188 signals generated by each PAF. This allows the injected calibration signal to be digitised and correlated with the signals from all PAF elements, yielding an amplitude and phase calibration for each port. Corrections are then made to the beamformer weights to maintain stable beam patterns and sensitivity as described in Section 9.6.

The on-dish calibration system consists of several modules, as shown in Figure 11. There is one Cicada module per antenna located in the central building. During normal operation, it generates an independent noise signal that is directly modulated onto an optical carrier and sent to the antenna on single-mode fibre. At the antenna, the optical signal is detected and the resulting electrical signal is sent to a small log-periodic dipole array antenna (LDPA, $G={4}\,\textrm{dBi}$) located at the vertex of the 12 m dish, which points at the PAF and is oriented so that its linear polarisation is at 45 to that of the PAF elements. The illumination of the PAF is close to uniform but there is a path length difference of up to 21 mm from the central elements to the edge. To a first approximation the signal injected into all PAF elements is the same.

Figure 11.

Diagram showing key components and signal paths for ASKAP’s on-dish calibration (ODC) system.

The Cicada also generates an optical copy of the noise signal. Half the power is sent directly to the corresponding antenna’s digital receiver and half is sent to the antenna on the same multicore fibre cable as the PAF signals. This copy is reflected by a Faraday mirror in the antenna pedestal and returns to the Cicada, then continues into the digital receiver. In the digital receiver, both optical signals are processed in the same way as the PAF signals, becoming inputs to the array covariance matrix and a higher sensitivity calibration correlator that forms all correlation products between a nominated reference signal and the PAF signals.

The direct signal from the Cicada gives the best estimates of gain and phase variation. The signal returned from the Faraday mirror allows the optical path to the LPDA to be calibrated via a round-trip phase measurement. This allows the phase variation of the full analogue optical path to be characterised. The system also provides a way to inject a common calibration signal into the PAFs on all dishes, though this is not routinely used.

Although the main purpose of the ODC system is to measure complex gain and provide updates to the beamformer weights, it can also be used to improve system diagnostics in areas of RF continuity, RF distortion assessment and filter-bank frequency integrity. The main contribution of the ODC at present is to stop beams from being completely destroyed by phase slopes corresponding to unpredictable delay changes that occur with each full reset of the analogue-to-digital converters (ADCs). With further research it is hoped to use the ODC to temporally stabilise PAF beam (voltage) patterns to better than 1% at the half power points (Hayman et al. 2010).

6.1. Design background

In the BETA version of the ASKAP PAF (Schinckel et al. 2011; Hotan et al. Reference Hotan2014), all analogue signal processing and digitisation occurred in the antenna pedestal. At the time of the BETA design, photonics for radio frequency over fibre (RFoF) signal transport appeared expensive and sufficiently unproven to deploy on the scale of a 6 768 element array for radio astronomy.

The BETA system comprised two shielded cabinets packed full of equipment and occupying most of the space in each antenna pedestal, along with a chiller plant located on the concrete antenna foundation pad to keep both the cabinets and PAF cooled. The chillers required continuous maintenance and the pedestal electronics were not easily accessible due to the confined space. Containment of radiated emissions was less than satisfactory due to the single layer of shielding and the large number of filtered conductive connections to the outside world. Even so, BETA served well to verify basic PAF performance and signal processing algorithms and was a pivotal step to the more advanced Mk II design.

For the Mk II version of ASKAP described here, the price of high linearity uncooled distributed feedback (DFB) laser diodes had fallen. Investigations of performance of low-cost RFoF analogue signal transport using these DFB components were undertaken with fibre spans up to 6km. It was found that, with suitable precautions, a directly modulated DFB laser solution could meet astronomy requirements (Beresford, Cheng, & Roberts Reference Beresford, Cheng and Roberts2017a). The cost of the optical transmitter was reduced to one hundred dollars rather than several hundred dollars for cooled laser devices or thousands of dollars for high quality externally modulated lasers traditionally used in broadband applications.

This allowed all analogue electronics, except the PAF itself, to be moved to the central building, which eliminated self RFI problems, lowered cost, and improved accessibility and maintainability of the analogue signal processing hardware.

6.2. LNA

Each domino houses the low-noise amplifiers (LNAs) for a pair of PAF elements (Figure 6) and there are 94 dominoes in each PAF. ASKAP’s LNA design forms an active balun by connecting the twin-wire feed line of one connected-array element on the PAF surface directly into two single-ended LNAs with a passive output-side balun (Shaw et al. Reference Shaw, Hay and Ranga2012). A passive input-side balun would have introduced too much loss. The differential input complicates the measurement of LNA noise temperature, so an innovative method was developed by Shaw et al. (Reference Shaw, Hay and Ranga2012). The measured LNA minimum noise temperature $T_{\text{min}}$ of the current design is 30 k in the middle of the frequency range (Shaw & Hay 2015).

6.3. Filter

After the LNA, the signal encounters selectable anti-aliasing filters (Beresford et al. Reference Beresford2017b) as shown in Figure 7. These filters define four observing bands summarised in Table 3 and Figure 14. The fourth band was included for assessment of the low-frequency performance of the PAF and has not been used for astronomy.

Table 3.

ASKAP receiver bands

6.4. Transmitter

After filtering, the signal is converted from electrical to optical for transmission back to the central building. The optical RFoF transmitter (Beresford et al. Reference Beresford, Ferris, Cheng, Hampson, Bunton, Chippendale and Kanapathippillai2017c,b; Hampson et al. Reference Hampson2012) is shown in Figure 8. The RF signal is directly added to the laser diode DC bias current and modulates the optical power. An average power control loop ensures constant optical level, compensating for component ageing. The low loss in the single-mode fibre allows the signal to be transported several kilometres. Each antenna connects to the central processing building via underground fibre-optic cables containing 216 cores per antenna, 188 of which carry RFoF PAF signals. Three additional fibres are used for the RFoF ODC system described in Section 5.6.

Figure 12.

Distortion performance, measured as the spurious-free dynamic range (SFDR), of the RF over fibre link.

6.5. Optical link design considerations

All observing bands are suboctave as they are in the second or third Nyquist band of the analogue-to-digital converter (ADC). This mitigates the second-order harmonic distortion of the RFoF link. Radio astronomy signal paths are designed to respond linearly to their input and this was an important consideration for the optical link as well. For systems operating close to linearity, small departures are often characterised using a measure known as the input or output intercept point (IIP or OIP) for various polynomial orders. The ASKAP RFoF link has an OIP1 of -10 dBm, an OIP2 of 8.3 dBm, and an OIP3 of 6 dBm (Beresford et al. Reference Beresford, Cheng and Roberts2017a) as shown in Figure 12. This marginally degrades as the fibre span increases (Beresford et al. Reference Beresford, Ferris, Cheng, Hampson, Bunton, Chippendale and Kanapathippillai2017c). The nominal RFI-free RFoF link output signal level is $-47\,$dBm (0 dBc), relative intensity noise (RIN) and shot noise define the output noise floor at $-72\,$dBm ($-25\,$dBc).

For narrow-band RFI equal in power to the system noise at the RFoF output (0 dBc), the second-order distortion is $-53\,$dBc and the third is $-102\,$dBc. The strongest narrow-band RFI is due to aviation Automatic Dependent Surveillance-Broadcast (ADS-B) transmitters, located on aircraft flying over the observatory at high altitude. This can be as high as 29 dBc for short periods of time (minutes) if the aircraft passes overhead.

Placement of half the anti-aliasing filter after the RFoF link eliminates the severe in-band second-order response from the RFoF link as well as out of band relative intensity noise and shot noise (Beresford et al. Reference Beresford, Ferris, Cheng, Hampson, Bunton, Chippendale and Kanapathippillai2017c). This allows the system to handle an extra 17 dB of RFI (third-order intermodulation at $-53\,$dBc, see Figure 12). The other half of the anti-aliasing filter preceding the RFoF link removes out of band RFI that could corrupt the link. Adjustment of the attenuator in the domino transmitter (see Figure 10) can reduce the signal level on the RFoF link to increase the compression headroom, but this comes at the cost of increased additive noise. With nominal settings, the RFoF link adds 1.5 k to the receiver noise temperature. Decreasing the power on the RFoF link by 3 dB gives 3.7 dB extra headroom for RFI but adds 1.9 k to the receiver noise temperature for the most distant antennas.

6.6. Optical receiver

The conversion from optical back to electrical signals occurs in the digital receiver (see Section 7.1). Further amplification increases the electrical signals to the level needed to drive the analogue-to-digital converter (ADC) in the digital receiver ($-35\,$dBm). The ADC has a specified spurious-free dynamic range (SFDR) of approximately 61 dB. The RF signal chain has an SFDR of approximately 48 dB. Marginal degradation of SFDR occurs for longer fibre spans due to the optical loss of 0.35dB km^-1. The longest fibre span in the array is 6km.

7.1. Digital receiver

ASKAP’s digital receiver (Brown et al. Reference Brown2014) is implemented with 12 Dragonfly modules (see Figure 13) for each antenna, each processing 16 optical signals. The 192 signals include 188 from each PAF, 2 calibration signals, and 2 spares for future applications such as RFI mitigation.

Figure 13.

The Dragonfly digital receiver module, designed by CSIRO specifically for ASKAP. There are 12 per antenna.

The ADC devices used for ASKAP are National Semiconductor ADC12D1600 parts that have 12-bit resolution with an effective number of bits (ENOB) of 9 and a full-power analogue bandwidth of 2.8 GHz. The sample clock is generated in the Dragonfly synthesiser module as described in Section 7.6.

The ASKAP digital receiver has a bandpass sampling architecture. It directly samples the PAF signals at RF (Figure 10) without the need for analogue frequency conversion. The full 700–1 800 MHz frequency range of the PAF is covered by three selectable and overlapping sampling bands (see Table 3 in Section 6).

Each sampling band is designed to match a Nyquist zone of the ADC. Two sampling frequencies, 1 280 and 1 536 MHz, are used to sample at RF with an instantaneous bandwidth of $640\,\textrm{MHz}$ or $768\,\textrm{MHz}$. Figure 14 illustrates how the sampling bands correspond to sampling frequency, how they overlap, and how much usable RF band is left after filtering. The lower and upper observing bands are located in the second and third Nyquist zones, respectively, of the ADC running at 1 280 MHz, while band 2 is located in the second Nyquist zone of the ADC running at 1 536 MHz, which overlaps the adjacent bands at the other sampling frequency. The sampling frequency and corresponding FPGA firmware are configured at the start of an observation.

An overview of the signal path following the ADC, for a single PAF element, is shown in Figure 15. Also shown, in unshaded boxes, are a number of other ancillary signal statistics and monitoring modules. In operation, the ADC histogram is particularly useful as it shows whether the signal is Gaussian and helps to set the signal level into the ADC.

The sampled PAF element voltages feed directly into a polyphase filter bank (PFB) with an oversampling ratio of 32/27 (Tuthill et al. Reference Tuthill, Hampson, Bunton, Brown, Neuhold, Bateman, de Souza and Joseph2012). The PFB forms channels with a sample rate of 1.185 and 1 MHz spacing. As the channels have a noise bandwidth of $1\,\textrm{MHz}$ and a channel spacing of $1\,\textrm{MHz}$ they will be referred to as $1\,\textrm{MHz}$ channels. The oversampling provides substantial gains in sub-band fidelity and downstream processing capabilities with only a modest increase in system complexity (Bunton Reference Bunton2003b; Tuthill et al. Reference Tuthill, Hampson, Bunton, Brown, Neuhold, Bateman, de Souza and Joseph2012). The channel-dependent frequency rotation within each output channel that is introduced by oversampling is compensated for within the PFB implementation so that the output 1 MHz channels are all correctly DC-centred before being forwarded on to the beamformer subsystem (Tuthill et al. Reference Tuthill, Hampson, Bunton, Harris, Brown, Ferris and Bateman2015).

Figure 14.

ASKAP digital system sampling bands. For convenience, we refer to these as bands 1, 2, and 3 in left-to-right order elsewhere in this document. The labels indicate whether the frequency channel order is inverted or not, with respect to the natural ascending order.

Figure 15.

The signal path for a single PAF element through the ASKAP digital receiver ADC and PFB.

The PFB produces either 768 or 640 $1\,\textrm{MHz}$ channels across the selected observing band (Figure 14), depending on the sampling frequency. In both cases, the channel sample rate is the same and the channel centre frequency is an integer multiple of $1\,\textrm{MHz}$. The output of the PFB for each channel is the complex-valued analytic envelope of the full-band signal component within that sub-band, i.e. the single-sided spectrum in that sub-band, translated to DC (Rice Reference Rice1982). The equivalent operations to generate the output of the PFB for each channel are a multiplication of the input by $e^{2 \pi ift}$, where f is the channel centre frequency, followed by low-pass filtering and decimation to the output sample rate.

Of the total number of output channels from the PFB, only 384 channels are selected for transport to the beamformer subsystem for further processing. These are transported as 8 data streams each with 48 channels. However, cost constraints and computing limitations mean that only seven of these data streams are beamformed and only six are correlated. This limits the total bandwidth available for imaging (see Section 11) to $288\,\textrm{MHz}$.

The channel selection process is very flexible and permits contiguous or non-contiguous groups of channels to be selected and also multiple copies of the same channels to be selected and transported to different downstream end points. The complex task of managing channel selection is done by the Telescope Operating System (TOS) software. At present, only contiguous bands are implemented to simplify visibility data storage and imaging. In future, we plan to support two or more split bands for purposes such as avoiding satellite interference and observing widely separated spectral lines.

Bit growth is permitted through the PFB to ensure that the DSP system meets dynamic range and noise floor requirements, so the output number representation for the coarse $1\,\textrm{MHz}$ channels is 16-bit real and 16-bit imaginary two’s complement (signed) integers. The resulting aggregate data rate at the output of the ASKAP digital receiver and PFB is approximately 100Tbit s^-1. This is efficiently transported to the beamformer using a custom point-to-point packetised streaming protocol that has very low overheads. The physical transport medium consists of 10 368 multimode optical fibres in 864 12-fibre ribbons. All fibres in a ribbon route to different locations and the fibre reordering is achieved by using custom passive optical cross-connects.

7.2. Beamformer

Output from the digital receiver for a single antenna is processed by seven Redback modules (Hampson et al. Reference Hampson, Brown, Bunton, Neuhold, Chekkala, Bateman and Tuthill2014), as shown in Figure 16. Each receives $48\,\textrm{MHz}$ of bandwidth from all PAF elements for a single antenna and can generate 36 independent dual-polarisation beams. With seven modules, 336 MHz out of $384\,\textrm{MHz}$ available from the digital receiver is processed. Wiring for the eighth module is in place, but cost constraints meant it was not installed. Each Redback has six processing FPGAs, which are fully interconnected electrically. The interconnect is used to distribute data so that each FPGA has $8\,\textrm{MHz}$ of data from all 192 digitised ports (188 PAF elements + 2 calibration signals + 2 spare ports).

The signal path through the beamformer is shown in Figure 17. The beamformer engine produces up to 72 independent beams on the sky, each a linear combination of M digitised port voltages ${\textbf{x}=\left[ {{x}_{1}},{{x}_{2}},...,{{x}_{M}} \right]^{T}}$. The beamforming operation is defined by

(2) \begin{equation}y(n) = \textbf{w}^H\textbf{x}(n),\end{equation}

where $\textbf{w}^H$ is the conjugate transpose of the beamformer weights and $\textbf{x}(n)$ is the vector of port voltages for a single frequency channel at time sample n. The beamformer weights are complex valued and may be varied independently for each $1\,\textrm{MHz}$ channel of each beam. The weights are represented as 14-bit real and 14-bit imaginary two’s complement (signed) integers.

Table 4.

Number of ports per polarisation per beam for various numbers of dual-polarisation beams

Figure 16.

Redback module used for beamformer and correlator. Field-programmable gate arrays (FPGAs) used for digital signal processing are hidden beneath cooling fans. Banks of dynamic random access memory (DRAM) are visible beside each FPGA.

The number of ports M that are weighted into each beam can be as many as all 192 ports from the digital receiver for 1 ASKAP antenna, but it reduces to 60 ports as the number of beams increases to the full complement of 72 single polarisation beams. The reduction of input ports with beams is summarised in Table 4 and is required due to hardware resource limitations in the beamformer. The selection of M ports from 192 can be specified arbitrarily for each $1\,\textrm{MHz}$ channel of each beam.

The system is typically configured to produce 36 dual-polarised beams. The beamformed voltages for each $1\,\textrm{MHz}$ channel pass into a second polyphase filter bank, which performs a final fine frequency channelisation. For normal operation the frequency resolution is $18.5\,\textrm{kHz}$.

A key advantage of beamforming on 1 MHz channels is that the beamformer can be most efficiently implemented mid-stage as 336 separate narrow-band beamformers (Van Veen & Buckley Reference Van Veen and Buckley1988). This allows beamforming with complex weights as opposed to frequency dependent weighting with FIR filters, which would be needed if the beamforming occurred on the ADC voltages. Furthermore, since the fine filter bank (FFB) is operating on beam voltages rather than raw PAF element voltages, there is a reduction in the FFB data throughput by a factor of approximately three (188 PAF elements/72 beams). This reduction also means that although three 12-fibre ribbons are needed to input data to a Redback module only one is needed at the output. That is, two fibres per FPGA each carrying $4\,\textrm{MHz}$ of beamformed data.

A key data product for many beamforming algorithms is the array covariance matrix (ACM), which is defined as

(3) \begin{eqnarray} \textbf{R}=\langle \left( \textbf{x}-{{\mu }_{\textbf{x}}} \right){{\left( \textbf{x}-{{\mu }_{\textbf{x}}} \right)}^{H}} \rangle,\end{eqnarray}

where ${\textbf{x}=\left[ {{x}_{1}},{{x}_{2}},...,{{x}_{K}} \right]^{T}}$ is a vector of K array element voltages assumed to be stationary discrete-time stochastic processes, ${{\upmu }_{\textbf{x}}}$ is the element-wise mean value of $\textbf{x}$, and $\langle\cdot\rangle$ denotes expectation. For the ASKAP signal model the $x_i$ are the frequency-channelised PAF element voltages, which are complex-valued random time series whose statistics are assumed to be stationary with zero mean over the beam calibration interval. Under these conditions, the elements of $\textbf{R}$ in (3) are adequately approximated for each $1\,\textrm{MHz}$ channel by the time average

(4) \begin{eqnarray} {{\hat{{r}}}_{i,j}} = \frac{1}{N}\sum\limits_{n=1}^{N}{{{{x}}_{i}}\left( n \right){x}_{j}^{*}\left( n \right),}\end{eqnarray}

where ${x}_{i}\left( n \right)$ is the $n\text{th}$ voltage sample of the $i\text{th}$ input port of $K=192$ and $x^*$ is the complex conjugate of x. With this definition, the ACM estimate $\hat{\textbf{R}}$ is a ${192\times 192}$ matrix where each element $\hat{r}_{i,j}$ is the correlation between $x_i$ and $x_j$. This is computed for all $1\,\textrm{MHz}$ frequency channels.

The full computation of the ACM has a higher compute load than the final correlator for each antenna. This load is reduced by using every fourth time sample and computing ACMs for ${42 \times 1\,\textrm{MHz}}$ frequency channels concurrently for each PAF. The ACM estimation process must be run eight times to cover the entire $8\times 42\,\textrm{MHz}= 336\,\textrm{MHz}$ of processed bandwidth. Raw ACM data products are transported to conventional compute hardware where they are used to calculate the beamformer weights offline as described in Section 9. The beamformer weights are then uploaded back to the beamformer block, which is shown below the ACM block in Figure 17.

Depicted above the ACM in Figure 17 is the calibration correlator. This calculates the correlation between a reference calibration signal and all PAF element outputs. This is mathematically equivalent to computing one row of the ACM, but in the calibration correlator, it is done for a specific reference input without time or frequency decimation. This achieves the best possible SNR for tracking PAF element gains. The calibration system is described in more detail in Section 5.6.

The beam voltages output from the beamformer block are fed into two separate processing modules:

1. 1. a deep ‘corner-turn’ buffer; and

2. 2. a transient ring buffer.

Both of these modules share a large double data rate (DDR) memory resource for temporarily buffering large amounts of beam voltage data. The transient module implements a ring buffer that can be frozen by an external trigger and is described in Section 7.3.

The primary signal flow for ASKAP, however, is the corner turn buffer that implements a streaming data transpose, supplying large blocks of sample data for the same beam and frequency channel to the FFB, which is operated in block-mode for efficiency. The FFB is a second-stage frequency channeliser implemented as a critically sampled polyphase filter bank on the $1\,\textrm{MHz}$ channels which have a sample rate of $1.185\,\textrm{MHz}$. This module can be configured at the start of an observation to provide a standard observing frequency resolution, or one of five ‘zoom’ resolutions, according to Table 5.

Note that since each of the 42 FPGAs in a single PAF beamformer subsystem processes the $8 \times 1\,\textrm{MHz}$ channels that are routed to it from the digital receiver, different zoom modes can be applied to different or even the same parts of the total processed bandwidth by loading the appropriate zoom mode firmware on each FPGA and routing the corresponding 1 MHz channels to it. Control software to implement this is not available currently. Each FPGA always generates 423 frequency channels independent of zoom mode. Thus, the total bandwidth decreases as frequency resolution improves as shown in Table 5. Processing of a full 1 MHz channel, for zoom modes, requires multiple correlator FPGAs to process the same 1 MHz channel.

Figure 17.

Major components of the signal path through the ASKAP beamformer.

Table 5.

ASKAP frequency zoom modes. In all cases, the number of frequency channels correlated is 15 552 for the current hardware deployment

7.3. Fast transient detection

As shown in Figure 17, a secondary beamformer output is sent to the fast-transient detector. This operates with the 1 MHz channels and has an adjustable time resolution that is typically set to 864 s or 1.7 ms. Power averages are sent over the MRO network to a dedicated computer that searches for dispersed pulses in real time (Bannister et al. Reference Bannister, Zackay, Qiu, James and Shannon2019a). In the search algorithm, these data are currently added incoherently across all antennas.

If a pulse is detected, a trigger is sent to the transient ring buffer in the beamformer, telling it to freeze the contents of the ring buffer (which stores an adjustable amount of raw voltage data from all beams, corresponding to the last 0.9–14.2 s depending on data precision selected) and download to disk. The voltage data are then correlated offline using the DiFX software correlator (Deller et al. Reference Deller2011) and imaged in order to localise the position of the event with sub-arcsecond resolution and provide high sensitivity data on the pulse (Bannister et al. Reference Bannister2019b).

7.4. Delay tracking and fringe stopping

Distant astronomical sources remain effectively stationary in the sky, while the Earth (and therefore the telescope) rotates once per day. This rotation means that the projected baseline vector between any pair of antennas in the array is constantly changing. Interference fringes, therefore, move during an observation, which would decorrelate the signal from the interferometer if not compensated for. The process of adjusting the delay and phase of each antenna to keep interference fringes stationary is known as delay tracking and fringe stopping.

Since ASKAP is a multi-beam instrument, we independently stop fringes in the nominal pointing direction of each beam. This is straightforward when using the antenna polarisation axis to keep beams pointing at a fixed position on the sky. We also offer the option to stop fringes in the boresight direction of the antenna for all beams, which can be useful for experiments such as holography where polarisation axis tracking is not used.

The ASKAP digital hardware implements:

1. 1. a pure time delay to a precision of 0.84 s applied across all beams and 1 MHz coarse frequency channels; and

2. 2. a time-dependent and frequency-dependent phase correction applied to each fine frequency channel.

A pure ‘coarse’ time delay (with precision of 0.84 s) is implemented between the beamformer and the FFB by offsetting the read pointer into the corner turn sample buffer for the FFB frame input. The coarse delay is the same for all beams and 1 MHz coarse frequency channels for a single PAF system, but is typically different between antennas in the array to allow for a coarse compensation of the difference in arrival time (up to 20 s) of a wavefront at different antennas. During an observation, the required coarse delay is computed repeatedly and set in the hardware at times synchronised with signals from the timing and synchronisation subsystem (see Section 7.6). Changes occur at the start of each data frame and do not introduce any discontinuities.

The second ‘fine’ stage of fringe stopping occurs in the fringe rotator (Figure 17), which uses complex multipliers to rotate the phase of each fine frequency channel time series as they come out of the FFB. The rotation phase is calculated from coefficients loaded via the control system software and permits a unique phase correction to be applied at each of the four levels:

1. 1. coarse 1 MHz channels;

2. 2. dual-polarised beams;

3. 3. fine frequency channels; and

4. 4. particular time samples.

The fringe rotator module exploits the fact that the rotation phase, for any beam, is linearly dependent on frequency. Furthermore, the module parameters are updated often enough that the time variation of phase is adequately approximated with a piecewise linear function. Under these conditions, the rotator can calculate the required phase, for a beam, from a unique starting phase for each coarse frequency channel and two increment parameters. These parameters are the change in phase with time increment and change in phase with frequency increment.

For each coarse channel and beam combination, the phase correction applied for an individual time sample $\Delta t$ units after the update epoch and $\Delta f$ fine frequency channels from a reference channel, is given by

(5) \begin{eqnarray} {{{\phi }\left(\Delta t,\Delta f\right)}={{\phi }_{O}}+{{\phi }_{T}}\Delta t+{{\phi }_{F}}\Delta f},\end{eqnarray}

where ${{\phi }_{O}}$ is an initial phase offset value and ${{\phi }_{T}}$ and ${{\phi }_{F}}$ are the time slope and frequency slope values, respectively.

The array-wide synchronous event that updates the coarse time delay has an appropriate processing delay applied before being used to update new fringe rotator parameters ${{\phi }_{O}}$, ${{\phi }_{T}}$ and ${{\phi }_{F}}$ so that the coarse delay and fringe parameters are consistent for each update interval.

The smallest incremental change that can be made to the fringe stopping parameters at the fine control stage (corresponding to 18.5 kHz channel bandwidth) is 0.206 ns in delay, ${0.044}{\deg}$ in phase, and 0.0248${\deg\,{\rm s}^{-1}}$ in phase rate. The fine-level delay slope across the band is implemented as a combination of delay steps and matching phase offsets with the increments given above.

At the output of the FFB, the beamformer subsystem produces ${{2}^{M-1}}\times 64$ fine frequency channels for each 1 MHz channel of each PAF beam, where M is the zoom mode number in Table 5 and $336 \times 1\,\textrm{MHz}$ channels are processed in the beamformer. As a result of the oversampling ratio of ${32}/{27}$ in the coarse filter bank, each 1 MHz of processed bandwidth is contained within the centre ${{2}^{M-1}}\times 64\times {27}/{32={{2}^{M-1}}\times 54}\;$ fine channels. The redundant ${{2}^{M-1}}\times (64-54)=10\times {{2}^{M-1}}$ fine channels at the band edges of each 1 MHz sub-band are discarded before further processing. The number of channels processed is independent of zoom mode.

7.5. Correlator

The correlator subsystem is the final stage in the FPGA-based signal processing chain at the MRO central site. It takes in fine frequency channelised beam voltage time-domain data, streaming from the 36 antenna beamformer subsystems, and generates the raw visibilities (cross-correlations) for the final science data processing stage. The data are carried from the beamformer to the correlator via multimode optical fibre and are transported using a custom, unidirectional streaming protocol for maximum link efficiency.

The outputs from the beamformer Redback modules connect to the correlator subsystem via several 12-fibre ribbon cables. A single correlator subsystem (also known as a block) processes 48 MHz of bandwidth and currently six of a possible eight blocks are installed. Each correlator block consists of 12 Redback modules (programmed for correlator functionality) and passive optical cross-distribution networks. The numbers in this section represent the installed capacity.

The accumulation operation in computing the visibilities results in a large reduction in the data rate such that the correlator output can be sent as Ethernet data over conventional network infrastructure to the remote high-performance computing facility in Perth, which is approximately 600 km from the telescope site.

The aggregate data arriving at the correlator input consists of 15 552 fine channels for 72 beams from each of the 36 antennas. To provide the capacity for polarimetry, these are configured as 36 dual-polarisation beams (with a linear basis). For each fine channel of each dual-polarisation beam, the correlator computes the correlations between all 36 antennas for both polarisations, resulting in $(2\times 36 + 1)\times (2\times 36)/2=2,628$ products. This is full Stokes cross-correlation for each of 630 baselines and full Stokes autocorrelation for each of 36 antennas. The total correlator output in one integration period, for all 36 dual-polarisation beams and all fine channels, is ${2\,628}\times 36\times {15552} = {1471343616}$ complex-valued visibilities in the current 6-block configuration with 15552 fine channels.

The complex-valued visibilities are streamed from the correlator as pairs (real/imaginary) of 32-bit floating-point numbers. The data streaming cycle from the correlator is locked to the integrate-and-dump cycle which, in turn, is controlled through the timing event subsystem (see Section 7.6). For the minimum allowed integration interval of 5 s, the total output data rate from the correlator is ${1471343616}\times 32\times 2/({8}{\text{bit}}/\text{Byte}\times {5} {\rm s}) \approx {2.4}\mathrm{GB\,s^{-1}}$. See Section 8 for more information on system-level constraints.

Data from the correlator are transmitted as user datagram protocol (UDP) packets, each containing up to 657 complex floating-point numbers corresponding to a single spectral channel, beam and polarisation. Partitioning the data in this way makes it easy to scale up the size of the telescope, but means the data must be reordered before further processing. Each UDP packet contains a total of 5 256 bytes of raw data, to which an additional 48 bytes of metadata are added, including the reference time of the accumulation cycle. The metadata expands the total amount of data leaving the correlator hardware by approximately 1%.

7.6. Timing and synchronisation

For a synthesis array radio telescope, the accurate distribution of a stable clock signal to multiple endpoints is a critical part of the instrument design. ASKAP is an array of arrays and this aspect of the system architecture is particularly important and challenging, where signal processing is distributed across some 4 000 separate FPGA devices and 8 000 separate ADCs. The station reference and timing system (Figure 18) must ensure synchronous sampling and processing of the data across the entire array and throughout the whole DSP chain.

ASKAP’s timing system is based on a general architecture used at CSIRO’s observatories (Hoyle & Mirtschin Reference Hoyle and Mirtschin2015). It distributes a 16 MHz reference clock to phase reference the signals from the antennas and a serial-encoded timestamp to enable absolute timing of synchronous events across the array.

The 16 MHz reference clock is provided by a GPS (global positioning system)-disciplined rubidium time standard. The rubidium clock provides the necessary short-term stability while the GPS corrections ensure absolute long-term stability of the reference. Also present on site is a hydrogen maser that can serve as an external reference input to the rubidium sources. This is not used for typical ASKAP operation, but is used for VLBI (very long baseline interferometry) experiments (e.g., Kadler et al. Reference Kadler2016).

The rubidium time standard provides a one pulse per second (1PPS) signal as well as the 16 MHz reference clock to the ASKAP timing reference control computer (ATRCC), which generates a serial-encoded timestamp known as binary atomic time (BAT). The BAT serial data stream is phase-locked to the 16 MHz reference clock and encodes a 64-bit integer representing the count of microseconds since the epoch MJD 0.0, which is 00:00 UT on 1858 November 17.

The BAT serial stream and the 16 MHz reference clock are distributed together in a star topology from the ATRCC by the optical time and reference distributor, Figure 18, to the rest of the ASKAP DSP hardware subsystems (digital receivers, beamformers and correlators). The transmission is by single-mode, length-matched optical fibres using optical splitters. The BAT signal is transported using Manchester encoding to ensure no DC bias for compatibility with optical transmission.

As shown in Figure 18, the system has dual-parallel redundancy, and is remotely configurable via the Optical Time and Reference Distributor. Either of the redundant time standards can be selected to provide the primary 16 MHz and BAT references for distribution to ASKAP DSP endpoints. The final stage in the time and reference distribution occurs at the digital receiver, beamformer or correlator subsystems where the optically distributed 16 MHz and BAT signals are used to generate all the timing signals needed for the subsystem.

In each beamformer and correlator subsystem, the timing signals are received by the timing reference distributor (TRD). The digital receiver subsystems use the low-noise reference distributor (LRD). The architecture of the TRD and LRD are shown in Figure 19. In the LRD and TRD, the 16 MHz clock is multiplied up to 64 MHz and the 64 MHz and BAT signal are distributed by CAT-6 cables to all modules in the subsystem. These signals are used to generate synchronous processing events in all Redback and Dragonfly modules. Each TRD can supply signals to eight Redback modules.

The LRD has an additional low-noise distributor for the 64 MHz clock, which uses length-matched coaxial cable to distribute a low phase-noise copy of the 64 MHz signal to the 12 digital receiver Dragonfly modules. This generates a high quality ADC sample clock with low jitter.

Figure 18.

Station reference: ASKAP’s master clock generation and event timing system. The ASKAP Timing Reference Control Computer (ATRCC) block is an industrial PC with a custom PCIe card that generates a serial encoded timestamp referred to as Binary Atomic Time (BAT) that provides a precise, absolute time reference with a resolution of 1 $\upmu$s.

Figure 19.

End point of the reference/timing distribution system. The low-noise reference distributor (LRD) generates and distributes a 64 MHz reference clock for the digital receiver sample clocks plus ${12 \times (64\,\textrm{MHz} + \text{BAT})}$ signals for synchronous event timing. The timing reference distributor (TRD) is the same hardware as the LRD without the separate 12-way 64 MHz reference distributor. The TRD is used for synchronous timing of events in the beamformer and correlator subsystems.

8.1. Digital system hardware considerations

The digital receiver outputs 384 channels of 1 MHz bandwidth each. Processing all of these would require eight beamformers per antenna, but only the first seven were installed. Wiring for the final beamformer is in place, but the construction budget did not allow the additional boards to be manufactured and they were not needed to meet the nominal bandwidth specification. Similarly, only sic of the full eight sets of correlator redbacks were installed. Since the correlator works in blocks of 48 MHz, meeting the nominal 300 MHz specification would have required installation and configuration of a partial block. While possible, this would have increased the complexity of control system logic and configuration management, for only a few percent improvement in continuum sensitivity. Using all available test hardware and reducing the stock of spare parts, it may be possible to install a seventh correlator block. However, the data ingest and image processing system was only designed to work with up to 300 MHz of bandwidth, so expanding to 336 MHz will be left for a future upgrade once operational experience shows how much spare capacity exists in the post-correlator stages.

8.2. Correlator output modes

The flexibility of ASKAP’s digital processing system allows each beamformer FPGA to operate in a different frequency resolution mode with arbitrary channels from the digital receiver. However, the complex channel routing logic required to implement a generalised configuration manager is prohibitive, and the control system is currently restricted to a single frequency resolution mode with contiguous bandwidth across all hardware. A summary of the available modes and the resulting bandwidth available for imaging is shown in Table 5. Regardless of frequency resolution, the 6-block correlator always outputs 15 552 spectral channels.

To reduce the amount of storage space required for initial capture of data intended for continuum science, the ingest pipeline software can perform online averaging down to $1\,\textrm{MHz}$ frequency resolution. This reduces the storage footprint by a factor of 54 and can be used to balance resource requirements when scheduling observations. This averaging mode is only intended for use with the full bandwidth, not zoom modes.

At the time of writing a 10 s integration interval is used in the correlator accumulation stage, reducing the data volume by a factor of two, for a total of 1.2GB s^-1 leaving the observatory. If pilot surveys show that this accumulation interval does not introduce any significant smearing, it will be used for large-scale surveys as well. We have tested cycle times down to roughly 7 s successfully with the existing system, so there is some room for improvement. Attempts to record with the nominal 5-second cadence lead to data loss that will require more investigation to understand. An upgrade to the ingest computing hardware at Pawsey scheduled for November 2020 should also improve performance.

8.3. Data processing assumptions

At the time of design, ASKAP presented an extreme computational challenge. Short-term storage of the correlator output was deemed impossible (and still presents challenges, though advances in technology have made temporary storage possible) and efforts were made to minimise resource-intensive aspects of downstream processing. One such limitation is that the outer six antennas were not expected to be used in spectral line mode, since the size of the resulting cube would be too large. Advances in visualisation and analysis software make it desirable to consider creating high-resolution sub-cubes, at least for specific regions of interest.

9.1. Maximum sensitivity weights

We compute the maximum sensitivity (maxSNR) beamformer weights following the procedure described by Hotan et al. (Reference Hotan2014) and McConnell et al. (Reference McConnell2016), using the Sun as our reference source. The estimated array response $\hat{\textbf{a}}$ of the PAF to a far-field signal incident from the direction of the desired beam is given by $\hat{\textbf{a}}=\textbf{u}_1$ where $\textbf{u}_1$ is the dominant eigenvector of $(\hat{\textbf{R}}_{n+s} - \hat{\textbf{R}}_{n})$ (Jeffs et al. Reference Jeffs, Warnick, Landon, Waldron, Jones, Fisher and Norrod2008). The maxSNR weights (Lo, Lee, & Lee Reference Lo, Lee and Lee1966; Applebaum Reference Applebaum1976) are then computed as

(6) \begin{eqnarray}\textbf{w} & = & \hat{\textbf{R}}_n^{-1} {\hat{\textbf{a}}}.\end{eqnarray}

The noise-plus-signal covariance estimate $\hat{\textbf{R}}_{n+s}$ is measured via (4) with the antenna pointed so that the calibration source (Sun) is at the same position within the field of view as the desired beam pointing. The noise covariance estimate $\hat{\textbf{R}}_{n}$ is measured with the antenna pointing at an ‘empty’ piece of sky, typically $15^{\circ}$ south of the calibration source.

To ensure there is only one dominant eigenvector, we use only X-polarisation ports to make X-polarisation beams and only Y-polarisation ports to make Y-polarisation beams. One eigenvalue equation is solved for the sub-array of X-polarisation ports and then a second eigenvalue equation is solved for the near-independent sub-array of Y-polarisation ports. This means that the polarisation states and angles of the X and Y beams will be tied to the inherently linear and orthogonal polarisations of the X and Y elements of the chequerboard array. Commissioning measurements have shown that this technique results in beams with low and stable polarisation leakages (Sault Reference Sault2014, Reference Sault2015; Deng et al. Reference Deng2017) and that ASKAP already has excellent polarisation imaging capability (Anderson et al. Reference Anderson2018).

Weights are calculated independently for each $1\,\textrm{MHz}$ channel of each beam, which can lead to a randomised phase response. As described in McConnell et al. (Reference McConnell2016), we adjust the phase of the weights at each $1\,\textrm{MHz}$ channel to ensure a smooth phase response with frequency for each beam.

Digital beamforming with $1\,\textrm{MHz}$ resolution introduces quasi-periodic bandpass errors with a $1\,\textrm{MHz}$ period. To work around this, for spectral-line observations, we sometimes reduce the beamformer resolution by fixing the beamformer weights to be identical over sub-bands of $N \times 1\,\textrm{MHz}$ where N is an odd integer. This increases the magnitude of the bandpass errors, but ensures their smoothness over the larger $N \times 1\,\textrm{MHz}$ sub-bands. This is advantageous when looking for astronomical features at $1\,\textrm{MHz}$ resolution. It also reduces the probability of an astronomical feature falling on a sub-band boundary.

Since McConnell et al. (Reference McConnell2016), we have extended our maxSNR calibration algorithm to:

• calibrate the XY phase of beams to simplify polarisation observations;

• accept the limit on the number of ports that our hardware can weight into each beam;

• avoid bias due to ports with abnormally high or low signal levels;

• correct weights for gain, phase and delay changes in the receiver electronics; and

• avoid bias due to RFI by interpolating weights with frequency.

9.2. Calibrating XY phase

Some of these changes were required to improve the robustness and reliability of the beamforming process for operational deployment. However, one great advantage of digital beamforming is that it allows control over aspects of the primary beam that would normally be fixed by construction. One such example is the relative phase of the two orthogonal polarisations.

We use the ODC system and the technique of Chippendale & Anderson (Reference Chippendale and Anderson2019) to adjust the beamformer weights so that the XY phase of each dual-polarisation beam pair is near zero by design. Calibrating XY phase up-front in the beamforming allows polarisation calibration and imaging to take place in the standard ASKAP software pipeline, so that polarisation studies can be made commensally with all ASKAP observations.

9.3. Beamforming with limited port selection

ASKAP’s beamformer (see Section 7.2) can simultaneously form 36 dual-polarisation beams compared to BETA’s nine. However, as the number of processed beams increases towards the full 36, the number of ports that can be weighted into each beam reduces towards 60 according to Table 4. This is due to hardware resource limitations in the beamformer.

If all ports were functional and had the same gain and phase response, we could just select the weights with the largest amplitude when selecting the finite number of weights M that can be processed by the beamformer. However, the inversion of the noise covariance in (6) can lead to very high numeric weights for ports that have low gain. A first step towards avoiding these problems is to use RF attenuators along the signal-chain of each port to normalise the signal level that is presented to the digital backend from each PAF element.

9.4. Robust port selection

In the case of a malfunctioning port, with near-zero gain even after attenuator adjustment, the covariance matrices are poorly conditioned and can further complicate the solution for maxSNR beamformer weights. To avoid this problem, we use a diagonal loading process to avoid singularities in the inversion of the full covariance matrix. This adds a small amount of white noise to the noise covariance matrix before its inversion and application in (6) to calculate the weights

(7) \begin{eqnarray}\textbf{w} & = & \left(\hat{\textbf{R}}_n+\alpha\textbf{I}\right)^{-1} {\hat{\textbf{a}}}.\end{eqnarray}

The diagonal loading places an upper limit on the weight that will be ascribed to any malfunctioning port with near-zero gain.

We use diagonal loading in a two-step process to robustly exclude malfunctioning ports from the beamformer weights. First, we select M ports with the largest signal contribution to the beam defined by (7). The signal contribution of each port is estimated with the aid of the ODC (see Section 9.5). Second, we recalculate the basic maxSNR weights using (6) on a reduced $M\times M$ covariance matrix of the selected ports.

The first step of the process tries to select the ports that are important to the beam, using diagonal loading to exclude malfunctioning ports. The second step applies the basic maxSNR algorithm, with no diagonal loading and therefore no bias, to the selected ports.

There is a sizeable literature on the appropriate selection of $\alpha$ in the diagonal loading process. We use an empirical technique guaranteed only to make improvements upon the maxSNR solution. We set

(8) \begin{equation}\alpha = \beta \textrm{median}(\textrm{diag}(\hat{\textbf{R}}_n))\end{equation}

and iterate the above solution over diagonal loading values of $\beta = [0, 0.1, 0.2, 0.3, 0.4]$, calculating the Y-factor (signal plus noise to noise ratio) at each step. Only solutions that improve Y-factor upon previous trials are retained. The first trial, with $\beta=0$, is equivalent to the maxSNR solution with no diagonal loading, so the iterative process is guaranteed to achieve or improve upon the sensitivity of the basic maxSNR solution.

Comparison of SEFDs achieved by this two-step process to those achieved by the raw maxSNR algorithm show that it improves sensitivity over the raw maxSNR algorithm in almost all scenarios. The improvement is most noticeable when there are low-gain ports with significant energy falling on them due to the focal plane field of the desired PAF beam.

9.5. Focal plane field estimation

We use the technique of Chippendale et al. (Reference Chippendale2016a) to decouple the PAF element gains from the beamformer weights and estimate the focal plane field $\textbf{s}$ matched by beamformer weights $\textbf{w}$

(9) \begin{equation}\textbf{s} = \mathbf{d^*_{\text{cal}}}\circ\textbf{w}.\end{equation}

Here, $\circ$ is the Hadamard (element-wise) product and $\mathbf{d^*_{\text{cal}}}$ is the complex conjugate of the PAF’s response to a plane wave. The plane-wave response is estimated by correlating each PAF element with a copy of the broadband calibration noise that is radiated into the PAF

(10) \begin{equation}\mathbf{d_{\text{cal}}} = \frac{\langle\textbf{x}_{\text{paf}}x^*_{\text{rtn}}\rangle}{\langle|x_{\text{rtn}}|\rangle}.\end{equation}

Here, $\textbf{x}_{\text{paf}}$ is a vector of the PAF element voltages sampled at the digital receiver and $x_{\text{rtn}}$ is the sampled copy of the calibration noise that has been sent out to the antenna and reflected back to the digital receiver via a Faraday mirror as shown in Figure 11. We commonly refer to $\textbf{d}_{\text{cal}}$ as the ODC response. In fact $\textbf{d}_{\text{cal}}$ are the conjugate field match beamformer weights that would form an aperture array beam matched to the ODC source.

Ideally, the calibration source would present a plane wave to the PAF, illuminating all ports equally and in phase. In this case, $\textbf{s}$ is a good estimate of the focal plane field excited when the telescope observes a point source in the centre of the beam created with weights $\textbf{w}$. The ports with the largest component amplitudes in $\textbf{s}$ are the ports with most significant focal plane field excitation by the desired signal. They are the ports with the largest signal contribution to the beam.

9.6. Correcting PAF element gain drifts

We infrequently observe the Sun to make new beamformer weight solutions; typically after several months have passed or when significant changes have been made to the hardware. Between these solution intervals we use the ODC system to correct existing beamformer weights for changes in the complex gains of the PAF elements. This is effected by the Hadamard product of the original weights by the Hadamard (element-wise) division $\oslash$ of ODC responses at the beamforming (subscript 0) and update (subscript 1) epochs

(11) \begin{equation}\textbf{w}_1 = \mathbf{d^*_{\text{cal,0}}} \oslash \mathbf{d^*_{\text{cal,1}}} \circ \textbf{w}_0.\end{equation}

The phase component of this correction is applied independently for each $1\,\textrm{MHz}$ channel, but the median value over the $336\,\textrm{MHz}$ beamformer bandwidth is used in amplitude. This forces the correction to be frequency independent in amplitude despite the ODC source having some undesired variation in spectral shape with dish antenna motion. We will revise the need for this smoothing in amplitude when the stability of the noise source is improved. The problem appears to be related to mechanical stability of the ODC vertex antenna and its RF connection to the noise source.

Since the ODC system is constantly injecting a low-level broadband reference signal into the feed (Chippendale et al. Reference Chippendale, Button and Lourenço2018), the array covariance measurements used for beamforming also capture a measure of the complex response of each element to the ODC signal. When we wish to update the beamformer weights, we record a smaller data product from part of the beamformer known as the calibration correlator (see Sections 5.6 and 7.2). Updated weights are calculated via (11) and uploaded to the beamformer.

This process requires only a small amount of data and since the beamformer weights are loaded via a double-buffered mechanism in the beamformer firmware, it is possible to update the weights without interrupting an observation in progress. The minimum update timescale is set by the time taken to compute, upload and switch to a new set of weights (a few seconds). However, the weights are currently left fixed during scheduling blocks. We perform an update before each observing session and leave the weights fixed until it is necessary to restart the digital receivers, typically when changing sampling bands.

Currently, the ODC correction in (11) stops beams from being completely destroyed by unpredictable delay changes that occur with each full reset of the ADCs. In the future we hope to use the ODC to stabilise PAF beam (voltage) patterns to better than 1% at the half power points (Hayman et al. 2010).

9.7. Avoiding RFI bias of beam direction

Basic maxSNR weights can be biased at frequencies with strong RFI. If there is RFI stronger than the beamformer calibration source in the measurement of $\hat{\textbf{R}}_{n+s}$, the resulting beam will try to point in the direction of the RFI instead of the calibration source. These weights will have a low gain in the desired beam direction at the frequencies with strong RFI. The astronomical signal in the intended beam direction cannot be recovered when using these weights, even with careful flagging of the RFI-impacted data in the time domain.

We avoid this problem by interpolating valid beamformer weights for ASKAP at interference-affected channels following the technique in Chippendale & Hellbourg (Reference Chippendale and Hellbourg2017). This uses iterative fitting of low-order polynomials to the weight amplitude spectrum of each individual port, taking advantage of the inherent smoothness of maxSNR weight amplitudes with frequency.

9.8. Using maxSNR beams for astronomy

Optimisation for sensitivity tends to create stronger primary beam side lobes than a mechanical feed and allows the beam shape to take on coma distortion off axis. Both of these can introduce undesirable effects in the imaging process.

Holography observations show that the total intensity beam shape is consistent with a circular Gaussian model to within roughly 10%, according to measured eccentricities (McConnell Reference McConnell2016) for a beam spacing of $0.9^{\circ}$. However, the level of coma distortion is a strong function of distance from the field centre, which means that the consequent variation in beam shape needs to be well characterised to form an accurate linear mosaic.

Figure 20.

The two commonly used footprints: square_6x6 (left) and closepack36 (right). In this case, both have beam spacings (pitch) of $1.0^{\circ}$ and a position angle of zero. The scales are in degrees, and celestial north (west) is to the top (right) of both diagrams. The + symbol represents the optical axis of the reflector and the lines extending from this position ending in filled circles represent the pointing shift required to optimally interleave the footprint by using other observations to fill in the least sensitive parts. The square arrangement has larger intrinsic sensitivity variation but requires only one interleaving position, while the closepack arrangement is more uniform initially but requires two interleaving positions for improvement.

Over time we intend to develop other optimisations that may improve the symmetry, polarisation and other properties of the beams. A starting point will be linearly constrained minimum variance (LCMV) beamforming (Frost Reference Frost1972), which can be used to optimise sensitivity with additional constraints on main beam shape. See Elmer, Jeffs, & Warnick (Reference Elmer, Jeffs and Warnick2014) for an example of constrained beamforming with a PAF.

9.9. Measurement of beams using holography

To assist with beam optimisation research, we have developed a raster grid holography method that provides a measure of the beam shapes over a large fraction of the field of view (Hotan 2016). Conducting a full measurement requires roughly six hours of observing time. This can be used to measure the stability and reliability of the beamforming process and assess the impact of changes to the constraints used when computing beamformer weights.

One of the next planned improvements to the imaging software will allow primary beam correction in the mosaicking stage using beam shapes measured via holography, instead of the circular Gaussian model currently applied.

9.11. Tiling footprints for surveys

Since ASKAP will spend most of its time conducting all-sky surveys, it is useful to define a standard tiling scheme to cover the celestial sphere with the footprints described above. In consultation with ASKAP’s survey science teamsFootnote ^f, we have developed a tiling scheme in which most of the sky is covered by bands of footprints stacked end to end at constant declination (Figure 21). This becomes less efficient towards the celestial poles, so at 72 degrees above or below the equator we switch to polar caps consisting of a square grid that is truncated at the overlap with the outermost declination band. This approach allows a fixed beam footprint to be used for the entire survey. Methods such as HEALPix (Górski et al. Reference Górski, Hivon and Banday2005) were considered, but deemed unsuitable. HEALPix was designed to efficiently partition a spherical surface by allowing the shape of each partition to vary. Matching the different partition shapes would greatly increase the operational overheads associated with beamforming.

The number of tiles required to cover the sky depends on the beam spacing within each tile, which can be adjusted to trade between survey speed and uniformity or mitigate widefield polarisation leakage. This optimisation is also a function of the observing frequency.

The first ASKAP all-sky survey, known as the Rapid ASKAP Continuum Survey (RACS), used a square_6x6 footprint with beam spacing of $1.05^{\circ}$ at a centre frequency of 888 MHz. RACS consisted of 904 tiles observed for 15 min each, covering the entire sky south of $+40^{\circ}$ declination. A future publication will describe the forthcoming data release and source catalogue.

9.12. Beamformer weight storage and management

Beamformer weights are stored in a custom HDF5 (The HDF Group 2020) file format. Currently, each scheduling block specifies which weights file to use. We have developed, and are now commissioning, a database service that tracks weights files and can automatically retrieve the best available weights for a requested footprint and frequency setup. It is important to ensure that details of the beamformer weights are included in the metadata associated with an observation, to provide a complete record of the system configuration.

11.1. Pipeline workflow

The processing pipeline currently runs on Galaxy, the Cray XC30 at the Pawsey Supercomputing Centre. It creates a workflow from a series of connected compute jobs, running under the Slurm workload manager, with appropriate dependencies to ensure completion in the correct order.

The processing is mostly performed on individual beams independently, before combining images in a linear mosaic. The top-level workflow for a single beam can be expressed as follows:

1. 1. Bandpass calibration is done using a separate observation of PKS B1934–638, establishing the overall flux scale. Frequency-dependent complex gains are determined for each beam and later applied to the science observation. See Section 11.2.

2. 2. Visibility data are prepared by splitting raw data measurement sets into single beam datasets (where necessary), then applying the bandpass calibration. The splitting may also create datasets for short time windows (typically one hour in length), which may be processed independently.

3. 3. RFI is flagged, before the data are averaged to 1 MHz resolution (this will be used for the continuum imaging). Coarse-resolution data may be flagged again. If time splitting was used, the time windows are recombined.

4. 4. The coarse-resolution measurement set is imaged and a sequence of self-calibration is performed. The self-calibration results in a fully calibrated image and table of time-dependent (but frequency-independent) complex gains. See Section 11.2.

5. 5. Both the coarse and full-resolution measurement sets are calibrated with the self-calibration gains.

6. 6. If necessary, continuum emission is subtracted from the full-resolution visibilities. This emission may be characterised by clean components, fitted components identified through source finding, or through fitting to the visibility spectra. The optimal strategy for a given science case is being determined in conjunction with the Survey Science Teams, using the pilot survey processing.

7. 7. The calibrated visibilities are imaged, producing final continuum images, coarse-resolution spectral cubes or full-resolution spectral cubes as required.

8. 8. If necessary, any residual continuum emission remaining in spectral cubes (as a result of incomplete modelling of the continuum emission) is removed through an image-based continuum subtraction step.

9. 9. Diagnostic and quality control information is assembled into a set of reports that are archived along with the image data.

Once all the beams have been imaged, full-field images and cubes are created through linear mosaicking. This corrects for the primary beam attenuation by applying a two-dimensional Gaussian primary beam model (see Section 9.9 for efforts to develop more realistic models) centred on the position of each beam in the footprint. Once the mosaics have been created, source finding is run to create catalogues. The data products can then be uploaded to the archive (see Section 12).

11.2. Calibration

Most ASKAP observations are calibrated in two steps. First, the flux scale and frequency-dependent complex gains (the ‘bandpass’) are determined from a dedicated observation of PKS B1934–638. This is the primary southern-hemisphere flux calibrator, and has a well-determined spectral energy distribution (Reynolds Reference Reynolds1994). The bandpass observation points each PAF beam in turn at PKS B1934–638 and observes for several minutes per beam. Although we currently assume a simple point source model for the calibrator source, Heywood et al. (Reference Heywood2020) have shown that slight improvements could be made using a model of the full field that includes confusing sources within the primary beam. The calibration parameters derived from this observation are applied to the raw science data prior to any other processing.

Since the bandpass gains are determined from a different observation, some refinement to them is often necessary due to time-variable factors such as atmospheric and ionospheric conditions that can impact the calibration solution at a low level. This is currently done through a self-calibration process, using the continuum-averaged dataset. A continuum image is made, then calibration is performed using a shallow sky model derived from that image. The complex gains are solved for within short ($\sim {5} min$) intervals across the observation. The data are then re-imaged using this updated gain calibration to provide an improved result. This process can be iterated several times if required. We typically solve for the phases of the complex gains and leave the amplitudes fixed, in order to preserve the overall flux scale. Once complete, the gain solution can be applied to the full-spectral-resolution visibilities to facilitate spectral imaging.

11.3. Wide-field imaging

Wide-field imaging is enabled via the W-projection algorithm (Cornwell, Golap, & Bhatnagar Reference Cornwell, Golap, Bhatnagar, Shopbell, Britton and Ebert2005), allowing handling of image-plane effects due to non-coplanar baselines. This gives accurate imaging over the field of view of a single beam. Multiple beams could be combined via AW-projection (Bhatnagar et al. Reference Bhatnagar, Cornwell, Golap and Uson2008; Bhatnagar, Rau, & Golap Reference Bhatnagar, Rau and Golap2013), taking into account the aperture illumination, although this approach is currently not used due to its large memory requirements. As described above, the aperture correction is instead applied when mosaicking the individual beams together.

The near-real-time imaging requirement has driven a number of design decisions within the code. The weighting of visibility data prior to imaging is traditionally done with an initial pass over the data to determine the distribution of visibilities as a function of baseline length and hence the weighting function, and then a subsequent pass for the imaging. To prevent repeated iteration over the large ASKAP data products, the ASKAPsoft code incorporates the preconditioning approach (Rau Reference Rau2010), where the re-weighting is performed after the construction of the dirty image and point spread function (PSF). ASKAPsoft imaging makes use of robust weighting, similar to that described in Briggs (Reference Briggs1995), allowing a tradeoff between the suppression of sidelobes and the overall sensitivity.

Continuum imaging is performed using Taylor-term multi-frequency synthesis (Rau Reference Rau2010; Rau & Cornwell Reference Rau and Cornwell2011). This creates an image of the total intensity at a reference frequency, along with images of the spectral variation across the band, from which can be determined the spectral index (and curvature, if enough terms are used) of sources. For spectral imaging, each channel is imaged independently, using either the averaged (continuum) data to create ‘continuum cubes’, or the full-resolution data to create spectral cubes. Continuum cubes can be made in all Stokes parameters, allowing polarisation imaging.

The imaging software has distributed processing designed in to the code at a low level. This allows the processing to be distributed over many compute nodes. The continuum imaging has the gridding distributed per coarse channel and Taylor term, with the deconvolution currently performed on the head node after combination of all channels. The spectral imaging is performed independently on each channel, and so is extremely parallel, allowing distribution over as many nodes as can be accommodated.

It should be noted that this extremely parallel case assumes a coarse form of Doppler correction done per beam and with only whole-channel offsets. This may be a limiting factor for some science applications.

Images are currently stored in FITS format (Pence et al. Reference Pence, Chiappetti, Page, Shaw and Stobie2010) for broad compatibility and because the science data archive was designed around this format. ASKAP data cubes can be very large (tens of terabytes) and it is possible that future research and development may provide a more efficient data storage format for use in parallel processing environments with remote visualisation services. If a new standard is established, we will investigate upgrading ASKAP’s software platforms to incorporate it.

11.4. Source finding and cataloguing

As part of the pipeline processing, we make catalogues of the sources in the images, using the Selavy source finder (Whiting & Humphreys Reference Whiting and Humphreys2012). Selavy uses the source-finding algorithms of Duchamp (Whiting Reference Whiting2012), a source finder developed for three-dimensional datasets. It can produce catalogues of spectral-line sources from the spectral cubes, largely following the Duchamp methodology, as well as catalogues of continuum sources.

The continuum source finding produces two different catalogues. The first is the catalogue of islands in the image. These are groupings of image pixels that are above the detection threshold. Each island is parameterised by its location and shape, determined solely from the detected pixels. To each island, we fit some number of two-dimensional Gaussians—these are the components, and the second catalogue reports their parameters. Each component will have a location, flux and shape, as well as information on the spectral index. This is obtained by fitting the Gaussian in question to the Taylor-1 image, which gives the product of the Taylor-0 flux and the spectral index.

Source finding is an active topic of research within the broader community and some of the ASKAP survey science teams will produce their own catalogues using different methods and tools. These are referred to as ‘value-added’ data products. For example, the SoFiA spectral-line source-finding software developed by Serra et al. (Reference Serra2015a) is being used on neutral hydrogen (HI) emission-line pilot survey data, and the FLASHfinder line-finding software developed by Allison, Sadler, & Whiting (Reference Whiting2012) is being used to search for HI and OH absorption lines on sightlines to bright radio continuum sources in ASKAP spectral-line cubes.

13.1. Sensitivity

The sensitivity of each ASKAP antenna depends on the physical area A of the reflector aperture and the effective system temperature $T_{\text{sys}}/\eta$ achieved by the receiver as installed on the reflector. Here, $T_{\text{sys}}$ is the system equivalent noise temperature and $\eta$ is the antenna efficiency.

Both $T_{\text{sys}}$ and $\eta$ vary with beamformer weights for a PAF receiver, so we report results for beamformer weights that maximise sensitivity (as used in operation; see Section 9). Figure 22 gives $T_{\text{sys}}/\eta$ across the ASKAP band measured with observations of PKS B1934–638 using the same technique as McConnell et al. (Reference McConnell2016). We use 75 k as a nominal value of $T_{\text{sys}}/\eta$ when summarising ASKAP performance.

The mid-band $T_{\text{sys}}$ of an ASKAP PAF beam, excluding spillover contribution from the ground, is 50 k (Chippendale et al. Reference Chippendale2016a). This measurement was made with the PAF on the ground pointing at the zenith, but using the same MaxSNR beamformer weights as are applied when the PAF is installed on the reflector.

Astronomers characterise sensitivity of each antenna plus receiver system in terms of the system equivalent flux density (SEFD) defined by

(12) \begin{equation}\text{SEFD} = \frac{2kT_{\text{sys}}}{\eta A}.\end{equation}

This is the flux an unpolarised point source must have to yield a signal-to-noise ratio of one at the output of a given beam. The factor of two appears in (12) because a single-polarisation beam can only collect half of the total incident flux from an unpolarised source (Wrobel & Walker Reference Wrobel, Walker, Taylor, Carilli and Perley1999). ASKAP’s nominal $T_{\text{sys}}/\eta$ of 75 k corresponds to an SEFD of approximately 1 800 Jy for a single ASKAP antenna.

Figure 23.

The sensitivity profile over the field of view. The black line traces the observed sensitivity (see text), and the red line shows an analytic approximation of the sensitivity estimated from SEFD observations. These data were obtained from observations using the square_6x6 footprint with a beam pitch of $1.05^{\circ}$ at a centre frequency of 888 MHz.

The sensitivity of the telescope can also be defined by the effective area divided by the system temperature

(13) \begin{equation}S(\Omega) = \frac{A_{\text{e}}}{T_{\text{sys}}}.\end{equation}

This is generally a function of direction $\Omega$ within the field of view and antenna efficiency $\eta$ achieved together with the PAF receiver. The sensitivity of the full ASKAP array evaluates to 54$\,\textrm{m}^{-1}\textrm{K}^{-1}$ for a beam close to the boresight.

13.2. Field of view

ASKAP’s field of view is set primarily by the size of the array of receptors at the focal plane and the size and focal ratio of the reflector. It is also modified by the aperture illumination, which is in turn controlled by the beamformer weights.

A measurement of the field of view with maxSNR weights was made using the ASKAP-12 array and reported in McConnell (Reference McConnell2017b). More recently, that measurement was compared with the observed variation of image noise over the field of view. Figure 23 shows this comparison in one dimension as an east-west profile of image sensitivity across the median of a sample containing 44 36-beam fields. The observations were made with the square_6x6 footprint using a beam spacing of $1.05^{\circ}$ at a centre frequency of $888\,\textrm{MHz}$. Relative sensitivity is calculated as the reciprocal of the image rms, divided by its maximum. Sensitivity is maximum near the centre of the field of view and is reduced by less than 20% at the edge.

These measurements show that ASKAP achieves its design specification of a 30$\,\textrm{deg}^{2}$ field of view at $850\,\textrm{MHz}$. The field of view reduces to 15$\,\textrm{deg}^{2}$ at 1 700 MHz due to the current limit of 36 dual-polarisation beams. The field of view at the high end of the band may be increased to 30$\,\textrm{deg}^{2}$ in the future by adding or reconfiguring digital hardware to process more than 36 beams.

13.3. Survey speed

As seen in Figure 23, the shape of ASKAP’s field of view is very different from that of a dish with a single feed, which is approximately Gaussian. The traditional survey speed figure of merit (1) should not be calculated using the half power field of view. When the sensitivity varies significantly over the field of view, (1) should be viewed as a point estimate that is integrated over the field of view to yield the survey speed (Bunton & Hay Reference Bunton and Hay2010)

(14) \begin{equation}\text{SS} = \int S^2(\Omega)d\Omega.\end{equation}

Equivalently, (1) may be used with the maximum sensitivity $S_{\text{max}}$ and a survey speed weighted field of view

(15) \begin{equation}\Omega_{\text{FoV}} = \frac{1}{S_{\text{max}}^2}\int S^2(\Omega)d\Omega.\end{equation}

All ASKAP survey speed and field of view numbers are calculated by (14) and (15) in this paper.

Using this definition, the survey speed of ASKAP is currently 91400$\,\textrm{m}^{4}\,\textrm{deg}^{2}\textrm{K}^{-2}$ at $800\,\textrm{MHz}$ with a smooth reduction to 44 200$\,\textrm{m}^{4}\,\textrm{deg}^{2}\textrm{K}^{-2}$ at 1 700 MHz. It may be possible to double survey speed at the high end of the band in future by processing more than 36 dual-polarisation beams. However, with current hardware, the extra beams would come at the expense of bandwidth.

As a point of comparison, evaluating (14) for a Gaussian beam yields a survey speed of $0.721\Omega_{G}S_{\text{max}}^2$, where $\Omega_{G}$ is the half power field of view of the Gaussian beam. Using this measure, the survey speed of ASKAP at 1.4 GHz is 245 times faster than the Parkes 64 m dish with a cryogenic single-pixel receiver having $T_{\text{sys}}/\eta$ of 40 k (Hobbs et al. Reference Hobbs2020).

15.1. Early science

The ASKAP early science programme started with 12 antennas equipped with Mk II PAFs in 2016. We planned to run two survey projects on behalf of all the survey science teams, but found that this was not feasible so early in the life of the instrument. Instead we devoted time to smaller test observations for each science team, using the data to test ASKAP’s image processing software and identify problems with the visibility data. These test observations were separated by commissioning periods of a few weeks duration, where antennas were added to the array and features necessary for full operations were implemented.

During this time, various existing software tools were used to cross-check and supplement ASKAP’s custom processing software as it was developed. Eventually we were able to complete a modest continuum survey known as the cosmology survey, which is available on CASDA.

Many publications arose from the early science programme, as a result of identifying projects that the telescope could excel at with only a third of its collecting area. These include a continuum survey and detection of cold gas outflows from the small magellanic cloud (McClure-Griffiths et al. Reference McClure-Griffiths2018; Joseph et al. Reference Joseph2019), studies of neutral hydrogen in nearby galaxy groups (Reynolds et al. Reference Reynolds2019; Lee-Waddell et al. Reference Lee-Waddell2019; Elagali et al. Reference Elagali2019; Kleiner et al. Reference Kleiner2019; For et al. Reference For2019), studies of absorption lines (Glowacki et al. Reference Glowacki2019; Allison et al. Reference Allison2020), continuum observations of the galaxy and mass assembly (GAMA) G23 field (Leahy et al. Reference Leahy2019), searches for transient and variable sources (Bhandari et al. Reference Bhandari2018), and a few targeted studies of individual radio galaxies (e.g. Seymour et al. Reference Seymour2020), including the polarisation characteristics of Centaurus A (Anderson et al. Reference Anderson2018). ASKAP was also the first telescope to localise a non-repeating fast radio burst to its host galaxy (Bannister et al. Reference Bannister2019b).

15.2. Pilot surveys

Integration of all 36 antennas with Mk II PAFs and digital backends was completed in February 2019. Soon after, we commenced a series of test observations designed to verify all the modes required for survey operations with the full array. This involved relatively simple imaging of representative fields. On 2019 July 15, we began a pilot survey programme to demonstrate readiness for extended science operations, and to highlight places of improvement for sustainable full operations. Pilot surveys involved observing small parts (100 h per project) of the larger survey plans developed by the survey science teams. The pilot survey concept arose as a natural extension of the early science programme. Reaching peak operational efficiency is a process that takes time and these pilot surveys provide useful science data while improvements and updates to the telescope’s core systems are ongoing.

There are several goals associated with pilot surveys:

• Provide representative data to each SST.

• Test SST survey strategies.

• Push operational limits to find pressure points.

• Develop operational plans for full surveys.

• Assess and improve processing pipelines.

Observations for phase I of these pilot surveys were completed on 2020 May 14, although processing is expected to take several additional months. Upon completion of pilot surveys phase I observations we entered a 6-month consolidation period, in which the priority for telescope access shifted back to development, testing and maintenance. This provided time to consider lessons learned and deploy various software and firmware updates, while processing the data backlog. The primary goal of pilot surveys phase I was to demonstrate production of science-ready data products. Phase II of pilot surveys will focus on optimising commensality and overall data processing efficiency. We are not currently planning a third phase of pilot surveys, but this may be considered if significant issues arise from phase II.

Full survey operations are expected to begin by the end of 2021 and ramp up in efficiency by 10% each year, reaching a limit of 70% efficiency in 2024.

15.3. Specifying observations

The ASKAP control system implements a flexible observation management system built around the concept of observing procedures and scheduling blocks. Procedures are implemented as Python scripts using a library of functions that provide access to the hardware. The simplest (and most commonly used) procedure simply tracks a source position specified in equatorial coordinates. More complicated procedures have been developed for bandpass calibration and holography observations where on-the-fly calculation of offset tracking positions is required.

A scheduling block is specified as a list of parameters and an associated procedure. The list of parameters currently includes source coordinates and system configuration options. Eventually, we plan to expand this list to include data processing parameters, which will allow automated execution of the image processing pipeline upon completion of a scheduling block. All-sky surveys require roughly 1 000 individual fields, but we have found in practice that scheduling each one individually (rather than specifying multiple fields in a single block) is the most effective approach, since it provides a level of fault tolerance and allows for more predictable timing.

15.4. Observation scheduling

Over the course of the pilot survey period, various scheduling requirements have been incorporated to meet the needs of different survey science cases. Developing a working model for efficient and effective scheduling is an ongoing process that depends on the level of system stability, robustness and automation. For now, scheduling is carried out in one of two modes: (i) transit-centric or (ii) target of opportunity. The first approach applies largely to long tracks, where the requirement is to schedule a long observation while the target is above the horizon and pair it automatically with a bandpass calibration observation. The second approach is more suitable for short tracks such as the 15 min observations used for the Rapid ASKAP Continuum Survey. It selects an appropriate target from a specified list based on a series of constraints (e.g. above the horizon, solar distance, lunar distance, hour angle, etc.). The scheduling algorithms are being refined and evolved as we gain experience with the requirements of survey projects and ensure the system is able to handle these in a way that is as automated and efficient as possible.

Our current intention is to aim for autonomous scheduling, in that the scheduling tool will detect system status and environment and have access to the pool of survey observations as well as survey constraints and historical observational data. It would then use this information to decide on the next observation to schedule and ensure the telescope is in the right state to complete this observation. The scheduling tool would also provide a forecast of the coming observations if the system status remained the same, but it would maintain the flexibility to adapt as necessary based on the changing system and environment. We intend to use the next period of pilot survey observations to prototype this approach and determine its feasibility for full surveys, ultimately maximising automation as much as possible which will help us to maximise efficiency gains. Recent developments by the ASKAP team to enable automated array start-ups based on the input parset have been a key milestone towards this vision for scheduling, and encourage us that our goal of autonomous scheduling may be realised for ASKAP.

15.5. Data quality diagnostics

ASKAP presents an incredible big data challenge, marking a noticeable shift in the ways astronomical data can be managed and interacted with. It is quite difficult to gauge data quality in meaningful ways when data sizes are tens of terabytes, including complex multidimensional axes across time, frequency, antennas, beams and polarisations. While working within the constraints of no longer being able to visualise all data, effort has been invested to ensure that certain aspects of the data can be visualised and used for diagnostic purposes, particularly to ensure that the system is working as expected in a timely way.

For the raw data, a series of plots have been designed and implemented to allow insight into the recorded data regardless of their size. Though currently the amount of time for processing does scale with data size (improving this is the subject of ongoing investigation), we are able to obtain these raw data diagnostic plots automatically in close to real time for short observations, which has been important for determining system status. These plots are extensions of standard astronomical visualisations of the visibilities, but necessarily more complex due to the amount of data needing to be represented. The exact form of ASKAP diagnostics is under development as part of the continuing transition between commissioning and operations. The ultimate goal is to assess data quality in an automated way, making use of anomaly detection algorithms and other machine learning techniques where feasible, and this is the subject of further investigation. Currently, diagnostic plots are visually inspected where necessary and allow human pattern recognition to spot anomalous conditions.

Some examples of diagnostic plots for a continuum dataset (SBID 13830 and 14997) are shown in Figure 24, demonstrating visualisation of (correlation) amplitude versus time, amplitude versus frequency channel, and time-frequency autocorrelation waterfall visualisations that combine the two. In particular, the last two plots are an attempt to visualise the contents of the entire dataset at once, showing a time-frequency waterfall plot of autocorrelation amplitude for each antenna and beam within the array. This is extremely effective at highlighting particular hardware issues, but less so for issues that depend on cross-correlation baselines. Although in this visualisation each subplot is too small to be used for quantitative analysis, broad trends or hardware faults stand out clearly against a background of relatively uniform behaviour. Conversely, the data used to generate these visualisations is extremely rich in diagnostic information, and will likely form a useful input dataset for either classification or machine-learning algorithms.

Figure 24.

Example ASKAP raw data diagnostic plots for Scheduling Block IDs 13830 and 14997, demonstrating visualisations of correlation amplitude versus frequency (top-left), correlation amplitude versus time (top-right), and time-frequency waterfall of autocorrelation for all beams and antennas (bottom-left) for SBID 13830. For comparison, we show the same diagnostic plot for SBID 14997. These diagnostics give us a direct insight into the recorded data prior to processing, and are used to inform overall data quality assessment. In the top two plots, each line represents a baseline colour-coded by baseline length (where red is shortest and purple is longest). In the autocorrelation waterfall plots, orange and red sections indicate beams on particular antennas that may be affected by faulty PAF dominos (in this case, the same bad YY elements are seen on ak12/ak13 for both observations), while the four vertical green stripes across entire beams trace the less-sensitive beams at the corners of the field of view. Dark red stripes indicate a missing antenna, and any antennas with visually outlying amplitudes are likely to have scaling issues. The vertical spikes seen in amplitude versus channel and the baseline-average waterfall plot are RFI.

Extensive quality control is also done post-imaging. Diagnostics are being developed by the science teams and will be incorporated into the standard observatory-driven processing pipelines over time. CASDA includes a validation stage that must occur prior to data release. Currently this requires human inspection of diagnostic information and selection of a three-tier quality grade. As diagnostic algorithms improve, we will seek to automate this process as much as possible.