3.3.1 Iterative gridding

Weighted median gridding is a non-linear process. It has the desirable property of robustness against outliers such as might arise from RFI, the Sun, Moon, or satellites. With the parameters typically used, it also tends to produce a gridded beam that is very close to Gaussian with FWHM that is remarkably insensitive to the cut-off radius, i.e. the radius of the circular ‘catchment area’ surrounding each pixel in the map. For example, in simulations using a circular Gaussian of FWHM 14.40 arcmin and a cut-off radius of 6 arcmin, the gridded FWHM was measured at only 14.45 arcmin, increasing to only 14.90 arcmin for a cut-off of 12 arcmin.

However, as explained in Barnes et al. (Reference Barnes2001), one adverse effect of median gridding is that the flux density scale differs depending on the source size. Without beam normalisation, compact sources appear in the map with reduced peak height, whilst extended sources appear at their correct peak height. Whilst this is also true of mean gridding, in that case the volume is preserved via broadening of the gridded beam, whereas with median gridding it is not because the gridded beam is not significantly broadened. The magnitude of the effect on compact source peak height depends on the gridding parameters, particularly the cut-off radius.

Unless otherwise stated, the simulations referred to below were done using a top-hat kernel, natural (unity) beam weighting, and without beam normalisation. For HIPASS data, with cut-off = 6 arcmin, point sources appear with peak height of about 86% their true value. With a 12 arcmin cut-off radius this quickly degrades to 61%. Whilst compact sources can be corrected by beam normalisation (as used in HIPASS), this inflates, and could introduce other adverse effects for extended sources. On the other hand, if a map consists only of extended sources then there is no problem. However, the all-sky continuum map is composed of extended and compact emission of equal import.

Iteration provides a solution to this problem for maps that contain a mix of compact and extended sources. Initially the zeroth-iteration map (henceforth 0-iter) is produced as normal. In the first iteration, the 0-iter map is subtracted from the raw data, which is then gridded to produce a residual map. Compact and slightly extended sources, which were underestimated in the 0-iter map, will be recovered, at least partially, in the residual map, which is then added to the 0-iter map to produce the first-iteration map (1-iter). Further iterations may be performed in like vein.

As the raw data serves as the standard against which the map in each iteration is compared, gridzilla never modifies it after reading it into memory. There is also too much of it to make a full working copy. Instead, each datum has the map value from the previous iteration subtracted only at the point that it is used, and as most data contributes to several pixels this happens multiple times for each datum. Iterative gridding thus requires a fast and accurate interpolation method for a practical implementation. gridzilla uses barycentric bivariate parabolic interpolation (Rodríguez et al. Reference Rodríguez2012) on the nine nearest pixelsFootnote ⁴ . In its general form

(3) \begin{eqnarray} f(x,y) & = & \sum _{i=1}^{3}\sum _{j=1}^{3} f(x_i,y_j) \left[ \frac{p_i(x) p_j(y)}{p_i(x_i) p_j(y_j)} \right], \nonumber \\ p_i(x) & = & \prod _{k=1,k \ne i}^{3} (x - x_k), \end{eqnarray}

Iteration may also be used productively for mean gridding where it acts to reduce the gridded beam size. Effectively, by driving the residual map towards zero, it makes the gridded map look more like the data in the sense that if the raw data were plotted on the map they would sit close to the surface defined by it.

Iteration in gridzilla is performed after the data has been indexed and read in and so adds only a relatively small overhead in processing time. It can also be sped up by applying a gain factor to the residual map before adding it. This factor compensates for the fact that point sources will appear in the residual map still with underestimated peak height. If the factor used is too high then when the corrected map is subtracted from the raw data it might drive some of it negative. However, negatives appearing in the residual map on the next iteration would then tend to correct this.

However, it is important to chose the gain factor appropriately to minimise the number of iterations, preferably to just one, because the base-level noise increases steadily with each successive iteration. This can be understood by considering a map produced from data that consists solely of noise. Because of the averaging associated with statistical gridding and the fact that the noise in the data is spatially uncorrelated, the root mean square (rms) noise in the 0-iter map will be much less than in the data (about 10% for a 6 arcmin cut-off). Therefore, subtracting the 0-iter map from the data will change it only weakly, whence the residual map will be highly correlated with the 0-iter map, i.e. the noise pattern will be similar. Thus when the residual is added to the 0-iter map the noise tends to be significantly worse than the sum in quadrature expected for uncorrelated noise. In simulations performed with a 6 arcmin cut-off, the rms noise in the 1-iter map was 50% higher than the 0-iter map for both mean and median. For the next two iterations the rms increased by roughly 20% per iteration.

However, because the residual map is only added to correct the profile of sources in the map, it only makes sense to add those parts of the residual map that do in fact achieve this. The rest is set to zero via a censoring algorithm. The residual values for a pixel and its eight neighbours are summed, giving half-weight to those in the corners. If this sum is below ×1.5 the map rms the value is zeroed, i.e. the nine pixels must be a little above twice the rms on average. This criterion helps to exclude isolated noise spikes, and to retain weak residuals in the vicinity of strong ones, i.e. on the edge of sources. Whilst this censoring ameliorates the adverse noise behaviour of iterative gridding on the background regions, it does tend to complicate the noise characteristics of the map as a whole.

Considering the unfavourable noise behaviour of iterative gridding, it is indeed fortunate that only a single iteration is required to achieve high-precision results for typical gridding parameters. If point sources appear in the 0-iter map at ratio g of their true peak height, then they would be expected to appear in the 1-iter map at ratio g + g(1 − g)f, where f is the loop gain factor. A factor f = 1.16 is thereby indicated for g = 86% (as above). However, this factor only applies to point sources—slightly extended sources would be overestimated. Sources that appear in the 0-iter map at ratio h (>g) will appear in the 1-iter map at r = h + h(1 − h)f. The worst case occurs for h = (1 + f)/2f. With f = 1.16 here, h = 0.93, whence r = 1.006, i.e. a precision of 0.6%.

In simulations using HIPASS data with a 6 arcmin cut-off, the integrated flux density and peak height of Gaussian test sources of various sizes were indeed recovered to within 1% with a single iteration using a gain of 1.2 for both the median and mean. In fact, acceptable results, to within about 3%, were obtained with gain factors in the range 1.1 to 1.3. The implied HPBW, i.e. $\sqrt{(}F^2 - T^2)$ where F is the measured FWHM and T = 14.4 arcmin is the FWHM of the test source, was 14.3 arcmin for median gridding (indicating a slight tendency to super-resolve), and at 14.6 arcmin was only slightly greater than it for mean gridding.

Minor edge effects are associated with iterative gridding because spectra that contribute to edge pixels, though lying outside the map, cannot be reliably corrected. The width of the edge strip affected is slightly greater than the cut-off radius.

3.5.1 On-axis ripple

The scaled template method of correcting the CBS described by Barnes et al. (Reference Barnes2001) relied on the first-order constancy of the shape of the CBS. For each data cube, the first step was to determine the normalised CBS empirically from the spectra of strong continuum sources. Then for each spectrum in the cube, the appropriate scale factor was measured directly from the data and the scaled template CBS subtracted.

Essentially the same method is used here, the difference being that the normalised CBS is determined separately for each beam and polarisation, this being done once and for all using a large subset of the HIPASS survey data. The appropriately scaled template CBS is then subtracted from each spectrum by livedata during post-bandpass processing.

To determine the CBS templates, the entire HIPASS/ZOA data set was reprocessed using robust polynomial bandpass calibration, with the continuum preserved as per Section 3.2, but without Doppler shifting nor spectral baseline removal. Preliminary investigation found that the CBS is produced only by point sources, extended continuum emission makes no contribution and so interferes with the normalisation of the spectra by the continuum level. Nor could any significant dependence on elevation nor feed rotation angle be discerned. Consequently, in order to exclude strong extended emission in the Galactic plane, only a subset of the HIPASS data with |b| > 15° was used to derive the CBS templates. For each beam and polarisation, using only spectra with a continuum level of at least 0.25 Jy beam⁻¹, the spectra were normalised by the measured continuum level, and the template then formed as the weighted median of these normalised spectra, with the continuum flux density used as the weight.

Galactic HI emission was mostly removed from the CBS templates by taking advantage of the annual Doppler shift, which pushes the line around sufficiently so that each of the relevant channels is unaffected for at least part of the year. The number of spectra used to generate the templates varied between 74 062 and 120 000 spectra depending on beam (mainly) and polarisation. This number greatly exceeds that used in the original HIPASS processing with a consequent reduction in the noise to the extent that small birdies became visible. These were removed via a nine-point running median smooth as was done by Barnes et al. (Reference Barnes2001).

3.5.2 Baseline curvature

Having derived the CBS templates, the task remains of determining the appropriate scaling factor for removing the CBS from each individual spectrum.

Preliminary investigation revealed that simple scaling of the template CBS by the continuum level, measured as the median value of the spectrum (excluding Galactic HI), was adequate for weak sources for which the correction is barely significant anyway. However, a small residual of the ripple component often remained for very strong sources. Strong, extended continuum emission is a complicating factor as it increases the continuum level without augmenting the CBS.

A different, more computationally intensive strategy was therefore adopted for the 10% of spectra with a continuum level of 1 Jy beam⁻¹ or stronger. This method optimised removal of the ripple component at the expense of possibly leaving a small residual of the curvature component, which, however, could easily be removed later with a polynomial baseline fit. The first step therefore was to separate the ripple from the baseline curvature component.

The ripple has a basic frequency of 5.7 MHz, which corresponds to 91 channels in the HIPASS/ZOA spectra. Applying a running mean of this width to the CBS templates therefore removes the ripple leaving behind only the curvature component. The ripple component may then be recovered by subtracting the curvature component from the CBS. This decomposition is shown in Figure 4.

Figure 4.

Continuum baseline signature (CBS) templates for each beam and polarisation decomposed into the ripple (left) and curvature components (right). Channel 1 024 is the low frequency end. Beam 1 is at the bottom and successive beams are offset by 0.02 as indicated by the dashed horizontal lines. The two polarisations are shown for each beam as black and grey traces. For the ripple component there is such a close correspondence between them that the two traces are often indistinguishable.

For stronger sources, the scale factor for the CBS template was determined from the ripple component alone. A nine-point median smooth was applied to the spectrum being corrected to match what was done in deriving the CBS templates. This eliminates H-recombination and narrow RFI lines and reduces the noise a little. Smoothing with a running mean over 91 channels then removes the ripple, leaving only the baseline which was then subtracted from the spectrum to isolate the ripple component. This was then divided by the template ripple component shown in Figure 4, and the scale factor determined as the weighted median, with the weight taken as the ripple template value. In fact, RFI that occupies up to one-quarter of the band can significantly affect the computation of the median across the whole band so this potential source of contamination was minimised by taking the median of the medians of each three-quarters of the band.

As expected, spectral cubes produced from data processed with this method of CBS removal compare favourably with the older scaled template method. Because separate templates are used for each beam and polarisation, the ripple is more effectively removed. The baseline noise is also significantly reduced because the CBS templates are constructed from a much larger number of spectra and consequently are virtually noise-free. Also, because the CBS correction is applied before gridding, rather than after, there is less scatter in the data used to compute each pixel value in the gridded cube.

3.5.3 Off-axis ripple

As we have seen, strong on-axis point-source continuum emission creates highly reproducible baseline distortions that can be characterised and removed. Not so for strong off-axis continuum sources, which often manifest themselves via irregular, quasi-periodic baseline disturbances that evolve with position as the telescope scans. Principal amongst these is the Sun, which may affect daytime observations. Although it has a variable effect for each specific point on the sky and so does not survive robust gridding, nevertheless any increase in the scatter in the data acts to reduce the accuracy of the final result. On the other hand, strong fixed cosmic continuum sources, of which there are many, especially in the Galactic plane, have a static effect, which can survive robust gridding. Thus for spectral line work it is always advantageous to remove such baseline distortions as far as is practicable, although with no benefit at all for continuum work.

When the scan for a particular beam and polarisation is represented as an image of channel versus position (i.e. time for a scanning observation), off-axis ripple manifests itself as an irregular, quasi-periodic pattern, usually with harmonics oriented at an angle to the horizontal or vertical as seen in Figure 5a. This suggests the use of a low-pass 2D Fourier filter to model the ripple pattern.

Figure 5.

Steps in off-axis ripple filtering. The spectral axis has frequency decreasing to the right from 1426.5 to 1362.5 MHz; the vertical axis is scan position, 500 s or equivalently 500 arcmin in extent: (a) the uncorrected scan; (b) masking of channels containing Galactic HI and RFI, as well as isolated discrepant pixels; (c) after linear interpolation and nine-point median smoothing in each direction; (d) masked pixels in (b) replaced with those of (c) after low-pass Fourier filtering; (e) ripple model obtained by low-pass Fourier filtering of (d); finally, (f) the corrected scan obtained by subtracting (e) from (a).

The principal difficulty is that such filtering is not robust against strong RFI nor line emission, which must therefore be excised. Figure 5 shows schematically the steps involved. Because the ripple pattern is broad-scale, whereas the RFI or line emission usually occupies a narrow range of channels or integrations, the strategy is to interpolate across it.

Bandpass calibration effectively ensures that the mean value in each spectral channel is close to zero. However, line emission and RFI still clearly manifest themselves via the dispersion, as seen in Figure 5a. As a prelude, isolated narrow- and broad-band RFI transients are masked on a channel-by-channel basis by censoring pixels outside 3×S_n for the channel, where S_n is the robust measure of dispersion discussed in Section 3.4. This discriminant corresponds to 3σ for a normal distribution. As it is not sufficient simply to zero these pixels, their values are replaced by linear interpolation across neighbouring values.

Discrepant channels are then identified by a discriminant based on the dispersion of the dispersion, i.e. channels that have a value of S_n much higher than normal, such that they correspond to 4σ outliers. The channel mask is then broadened by flagging immediately neighbouring channels if they have S_n outside 1σ. Linear interpolation across the channels provides values for the masked channels, followed by nine-point median smoothing to eliminate any small patches of narrow RFI that may have survived to this point, as in Figure 5c. This first-pass interpolation is then subjected to low-pass Fourier filtering using a Gaussian filter function with twice the FWHM of the required final filter function. The result is used to provide values for the masked pixels in the original unsmoothed image, and this then subjected to low-pass Fourier filtering to obtain the ripple model as in Figure 5e.

A Gaussian with FWHM of 16 harmonics in the spectral direction and 8 in the scanning direction proved to be an acceptable compromise for removing the ripple whilst retaining compact spectral line sources, though inevitably creating shallow moats around them.

3.7.1 HIPASS declination zone overlaps

Except for the two circumpolar zones, the HIPASS declination zones were stepped by 8° (=480 arcmin), i.e. ( − 87°, −82°, −74°, −66°, . . ., +14°, +22°). Typically the scans are 100×5 s integrations at 1 arcmin s⁻¹, (500 arcmin) so naïvely we might expect a 10 arcmin overlap at each end. However, as seen in Figure 3, the inner beams contribute another 29 arcmin, and the outer beams another 51 arcmin at each end of the scan. Consequently, the usable overlap between zones is more like 45 arcmin.

However, at 1°.5, the overlap between the − 87° and − 82° zones is twice as wide, the data is also more uniformly distributed in this region of the sky because of grid convergence effects (see Figure 3), and there happens not to be any significant sources of emission to upset the measurements. Consequently, the offsets between these two zones are better determined than elsewhere and are considered not to require correction.

If the − 87° zone had extended to the SCP, or preferably slightly beyond it, then the southern edge of the zone would effectively have overlapped itself, and it would have been possible to set offsets for the zone by reference to the common point, the SCP, in each scan. However, as the Parkes telescope is based on a master equatorial drive system, the dish cannot scan through the SCP and necessarily stopped a short distance from it. Consequently, offsets for the − 87° zone were set by averaging integrations in the three map rows 8 arcmin, 12 arcmin, and 16 arcmin north of the SCP. Small offsets of up to 57 mJy beam⁻¹ were required to achieve uniformity in this narrow strip.

In the first instance, the procedure was to map the overlapping strips separately using weighted median gridding onto a plate carrée projection of size 5401×11 pixels (or ×23 for the southern zones) spaced by 16 s in RA (4 cos δ arcmin) and 4 arcmin in declination. Iterative gridding was not used because adverse edge effects would have affected too high a proportion of each map. Instead the whole process was iterated as described later.

Overlapping strips were subtracted to produce a difference map, e.g. the northern strip of zone − 34° minus the southern strip of zone − 26°, which ideally should have constant brightness in declination. In practice, such constancy usually was observed to a remarkable degree, but in certain locations, particularly where the strip crossed the Galactic plane, there might be significant departures. Often this could be traced to a strong compact source that did not cancel exactly between the two strips; although it may have been a small fraction of the source strength, the residual would still be unacceptably high in absolute terms. This resulted in the streamers that will be discussed later. The zone offsets were computed by averaging the 11 (or 23) pixels across the width of the strip, potentially with censoring of such outliers.

With as few as 11 measurements, it was important to use the data as efficiently as possible. Offsets were computed for each RA as the median of the difference between the zone overlaps, southern minus northern. To improve statistics, $\lfloor 5 \sec \delta _{\mathrm{z}} \rfloor$ image columns spanning approximately 20 arcmin were aggregated, where δ_z is the mid-zone declination. However, as these samples are not independent, the improvement is somewhat less than it would otherwise be.

The offsets were then subtracted from the difference map leaving a residual map in which outliers could clearly be identified. These tended to cluster near the southern or northern edges, or in localised areas of strong emission that failed to cancel exactly between the two zones. S_n computed for the residual map as a whole was used to censor these outliers. Starting from the SCP, the values obtained for S_n in each zone were: 15, 24, 27, 29, 40, 36, 35, 35, 37, 38, 41, 41, 41, 45, and 48 mJy beam⁻¹. Only 0.27% of normally distributed data are expected to lie outside the 3σ confidence interval. However, averaging over all zones, 3.7% were found to lie outside 3×S_n , indicating the degree of departure from normality. Because one discrepant edge pixel could reasonably be expected in a scan of length 11 pixels, the offset was recomputed if up to 10% of the data was outside 3×S_n . Of the 81 000 offsets being determined, 60% were computed from data that had no outliers, and 29% were recomputed with a small number of outliers censored. The remaining 11% were not recomputed on the grounds that the value of S_n for the whole scan probably was not applicable to them and there was no way to improve on the median value.

The process of determining the offsets was iterated, with the offsets determined on one pass used for zipping the overlap strips on the next. The offsets provide everything required here and were used in preference to the levels to avoid complications that might arise from the accumulation of error in computing the latter. Iteration results in a small relative change because the offsets computed from the gridded maps are necessarily a smoothed version of the offsets that must be applied to the ungridded data. In practice, however, it mainly only served to demonstrate self-consistency. The offsets obtained are referred to as the core set to which reference is always made in subsequent processing.

3.7.2 Computing the HIPASS levels

The level for each of 5 400 RAs in each of the 15 declination zones was then computed by summing the core offsets from zone − 87°.

Negative levels of up to −133 mJy beam⁻¹ then resulted in zone − 82° over an extended range of right ascension. These could not be attributed to noise alone, especially when, as previously explained, the offsets between these two zones are better determined than elsewhere. Also, there is no reason to suppose that the minimum in the continuum emission occurs at the SCP or anywhere near it. Consequently, the offsets in zone − 87° were adjusted upwards uniformly by 133 mJy beam⁻¹, which has the effect of increasing all levels uniformly.

The levels were then smoothed lightly using a Gaussian of FWHM $4 \sec \delta _{\mathrm{z}}$ image columns, or approximately 16 arcmin. Considering that zone levels arise from emission that is significantly extended with respect to the 8°.5 scan length, much heavier smoothing in RA would be justified on the basis that there are no long, narrow sources of continuum emission that are oriented north–south on the sky. However, that was unnecessary and probably would have been counterproductive at this stage.

Determination of the zone levels by integrating the offsets amounts to a simple one-dimensional random walk, the variance of the level at any point being equal to the sum of the variances of the steps preceding it. Noting that the values of S_n computed for each zone above were for the dispersion of the data used to compute the offsets, the equivalent of the standard error in the mean should be obtained by dividing by $\sqrt{n}$ . However, we have used the median, which is less efficient than the mean, but on the other hand, aggregated image columns spanning 20 arcmin, though they did not provide independent samples. Taking the overlap width divided by the HPBW, n = 3 (or 6), as a lower limit for n, and omitting the − 87° zone, which defines an arbitrary zero level, the 1σ uncertainty in the levels in each zone should be less than 0, 10, 18, 25, 34, 40, 45, 49, 53, 58, 62, 67, 71, 75, and 80 mJy beam⁻¹. These provide target figures for the potential accuracy of the zipping operation. However, as indicated previously, S_n is abnormally high for about 11% of the measured offsets, mainly for those scans that traversed the Galactic plane. Thus the uncertainty in the levels computed from these offsets can be much higher and this is what gives rise to streamers.

The zone levels obtained are shown in Figure 9 with obvious streamers extending northwards from the Galactic plane, particularly in the vicinity of the Galactic centre. The result of applying them when gridding may be seen in Figure 11a. This is essentially what would be obtained by adding Figure 9 to Figure 8 (on the same projection), the zone-level correction being additive. Streamers are clearly evident. The next step is to refine these levels by crossing HIPASS with ZOA.

3.7.3 Crossing HIPASS and ZOA: (a) computing the ZOA levels

After bandpass calibration, ZOA scans have an arbitrary offset just as do the HIPASS scans. The first step in improving the HIPASS zone levels is to derive these ZOA zoan levels. Figure 10 illustrates that process graphically. Of course, the zoan levels are also needed so that the large ZOA data set can be included in the final map, substantially reducing the noise in the interesting Galactic plane region.

Figure 10.

Graphical depiction of the first iteration of ZOA zoan-level determination based on HIPASS. In each case, the data are gridded onto a plate carrée projection in Galactic coordinates, + 53° ⩾ ℓ ⩾ −165°, |b| ⩽ 16°. The inputs are (a) HIPASS zipped using levels deduced from the declination zone overlaps, and (b) ZOA without levels, analogous to Figure 8. The difference between these, (c), is used to derive the levels for ZOA, which when applied yield (d). In practice this is done separately for each ZOA zoan, with the levels derived by averaging in Galactic longitude. Logarithmic greyscale as for previous figures.

For each ZOA zoan, zone-adjusted HIPASS data were gridded onto a plate carrée projection in Galactic coordinates, dimensioned so that it just covered the zoan. ZOA data was also gridded onto an identical projection. The image width, which is the critical dimension, was 129 pixels spaced by 4 arcmin. The difference between these two maps, HIPASS minus ZOA, should be constant in Galactic longitude, though varying in Galactic latitude. This is the zoan level, i.e. the offset that was lost during bandpass calibration.

As in Section 3.7.1, iterative gridding (Section 3.3.1) was not used in order to avoid edge effects. However, the whole procedure was iterated as described in Section 3.7.5.

In practice, because of errors in the HIPASS zone levels, the difference maps are not quite constant in Galactic longitude, particularly in being affected by residual streamers, as seen in Figure 10c. However, as the streamers cross at an angle to the parallels of Galactic latitude, mostly they only affect a part of the scan in Galactic longitude from which the zoan level is determined.

Unlike the HIPASS levels, which must be derived by summing offsets, the ZOA levels are obtained directly. As for the HIPASS zone offsets, they are computed as the median value across the scan in the difference map (Figure 10c). In contrast to HIPASS, there is a full 8°.5 of overlap to work with, rather than 45 arcmin, and smoothing of the levels was not warranted. However, censoring of outliers was performed as before. That is, the zoan levels were subtracted from the difference map to produce a residual map. Pixels outside 3×S_n in this map were deemed to be outliers, where S_n was computed from the map as a whole, and the zoan levels then recomputed. Visual inspection showed that this was particularly effective at removing outliers associated with areas of strong emission that did not cancel due to small relative errors between the HIPASS and ZOA maps.

Figure 10d was produced from zoan-adjusted ZOA data, a remarkable development of the uncorrected map seen above it. It bears a close resemblance to the first-pass HIPASS map of the Galactic plane in Figure 10a, but much less affected by streamers. This suggests that it may be used for improving the HIPASS zone levels.

3.7.4 Crossing HIPASS and ZOA: (b) refining the HIPASS levels

Differencing Figures 10a and 10d produces a map, seen in Figure 12a, that consists mostly of residual artefacts—streamers—in the HIPASS map, together with strong sources that do not quite cancel. However, Galactic coordinates are not the best choice because the streamers are oriented at different angles depending on Galactic longitude. In equatorial coordinates on a cylindrical projection the residual streamers all run north–south, as seen in Figure 11a. When the zoan-corrected ZOA data is gridded onto an identical projection, and the difference taken, the residual streamers manifest themselves as vertical stripes as seen in Figure 11c, and as such are amenable to measurement and subsequent correction.

Figure 11.

Graphical depiction of the first iteration of HIPASS zone-level correction based on ZOA. In each case, the data are gridded onto a plate carrée projection but this time in equatorial coordinates, 24^h ≥ α ≥ 0^h, − 90° ⩽ δ ⩽ +26°. The inputs are (a) HIPASS zipped using levels deduced from the declination zone overlaps, as per Figure 10a, and (b) ZOA with levels deduced as per Figure 10d. The difference between these, (c), is used to derive level corrections for HIPASS. In practice this is done separately for each HIPASS zone, with the correction derived by averaging in declination. Logarithmic greyscale as for previous figures except for (c), which is a linear scale from −1 to +1 Jy beam⁻¹.

The procedure was similar to the forgoing. Maps on a plate carrée projection, this time in equatorial coordinates, were produced separately from HIPASS and ZOA data for each HIPASS zone that intersects the ZOA, namely − 66° and northwards. These are long thin strips, 5401×161 pixels, of which the ZOA data only fills a portion. A correction to the levels was obtained as the median value across the scan in declination, with censoring of outliers applied as before. Again, visual inspection confirmed the validity of the censoring. A correction was computed if, after censoring, at least five pixels remained in the declination strip.

At this point we have a set of corrections to the zone levels. In fact, what is required is a correction to the core set of offsets that were derived in Section 3.7.1. Whilst there was only one place to apply the 133 mJy beam⁻¹ offset introduced to ensure non-negativity in zone − 82°, namely in the zone − 87° offsets (Section 3.7.2), for more northerly zones the residual could be applied to any of the zones further south. In practice, levels were adjusted by distributing the residual amongst the offsets in zones south of the level being corrected on a pro-rata basis according to the variance (S ² _n ), excluding zones − 87° and − 82°. This tended to place almost all of the correction in the Galactic plane.

This process provided corrections to 13.8% of the HIPASS levels. Wherever no correction was obtained from ZOA, the offsets were replaced with the corresponding offset from the core set. In particular this included all offsets to the east, north, and west of the ZOA survey area.

3.7.5 Crossing HIPASS and ZOA: (c) iteration and convergence

After refining the HIPASS levels it seems obvious that they could be used to produce a better estimate of the ZOA levels. Accordingly, the procedure described in Section 3.7.3 was repeated. In turn, the new estimate of the ZOA levels could then be used to improve the HIPASS levels. In effect such an iterative process exploits the fact that HIPASS scans may form a bridge between adjacent ZOA zoans from which the relative zoan levels can be determined. Likewise, the ZOA scans may form a bridge between adjacent HIPASS scans. However, unlike the HIPASS zone overlaps where the overlapping scans are parallel to each other, these bridges cross at an angle. Thus, instead of providing the offset between a pair of scans on opposite sides of a discontinuity, they provide the offset between sets of scans on either side. Complicated as the geometry may be, it seems reasonable to expect that convergence should be obtainable. That is, we expect that there is a set of HIPASS and ZOA levels for which the difference between the HIPASS and ZOA maps is zero plus noise, and we anticipate that iteration will find them. However, there are complications.

In successive iterations, as the HIPASS levels are expected to be better determined, the streamers should reduce in size and generally have less of an effect on the determination of the ZOA levels. Indeed, very few streamers survived even a single iteration. However, it was found that if the streamers were wide enough on the first pass, and the angle acute enough, then they could cause the iteration to fail to converge over part of the map. The problem is evident in sections of the ZOA north of the Galactic centre region in Figure 12.

Figure 12.

Progress of the iteration depicted graphically—HIPASS minus ZOA after (a) zero, (b) one, (c) two, (d) four, and (e) ten passes of refining the HIPASS levels. (Figure (a) is equivalent to Figure 11c.) Most of the narrow streamers have disappeared after a single pass leaving only intransigents that emanate from the Galactic centre region. These are slowly eaten away in subsequent passes. The subtle saw-tooth residual, which appears below the Galactic centre region, becomes more noticeable as the streamers are corrected, but is gone in the final pass with special handling. Same projection as Figure 10. Linear greyscale from −1 to +1 Jy beam⁻¹.

The characteristic blocky structure seen in Figure 12b indicates that wide streamers have corrupted the zoan levels over a range of Galactic latitude in adjacent zoans. In fact, this can be seen directly in Figure 10d, whilst close inspection of Figure 12b reveals that all zoans between 344° and 48° were affected to some degree. These intransigents persist through successive passes, indicating that they form part of a stable, yet clearly incorrect solution of the iteration. The cure, applied in the second and third passes, was simply to identify and excise the affect ranges of Galactic latitude in each zoan of the ZOA maps from the next pass of the iteration. The bad latitudes are those associated with the dark blocks in Figure 12b. The bright corners are so because they correctly identify the positive excess due to the positive streamer (in this case). Effectively these bright corners eat into the streamer on the next pass, eliminating the smaller intransigents, and reducing the width of the larger ones in Figure 12c. After four passes the intransigents have been completely eaten away in Figure 12d.

The HIPASS maps produced in each pass always applied the best estimate of the levels available, namely those obtained from the previous pass. Thus subsequent iterations only provided corrections to the levels from the previous pass. As mentioned in Section 3.7.1, iterating corrects for the subtle effect that the levels computed from the gridded maps are necessarily a smoothed version of the levels that must be applied to the ungridded data. However, until the intransigent streamers had been completely removed, the ZOA maps used to compute the zoan levels (Section 3.7.3) were all produced without levels. That is, the uncorrected map shown in Figure 10b was used each time. This provided a fixed reference point from which the convergence behaviour could be assessed. The remaining iterations were then done with zoan levels applied.

As the difference between the HIPASS and ZOA maps should consist solely of noise together with a few outliers associated with strong sources, Figure 12 shows that, when present, the effect of streamers or other defects is readily apparent on visual inspection. Thus the convergence behaviour could be monitored qualitatively at a glance. After dealing with the intransigent streamers, a new type of artefact assumed prominence albeit at a lower level. It appears in Figure 12d below the Galactic centre region as a saw-tooth pattern with an amplitude of about 100 mJy beam⁻¹. This pattern arises from a different mechanism to the intransigent streamers and disappears slowly in successive iterations. However, after eight passes it was apparent that the convergence was slow enough to warrant investigation of a more direct solution.

Thus far we have not considered the overlap between adjacent ZOA zoans. Although they offer an independent mechanism for tying together the ZOA zoans, and hence the HIPASS zones, these overlaps are problematic for two reasons. Firstly, unlike HIPASS where the scans all converge on the SCP, ZOA scans when tied together have no common point so there is no way to relate scans in neighbouring Galactic latitudes. Secondly, strong sources within a degree or two of the Galactic plane render the determination of offsets very uncertain, with the formation of prominent streamers in Galactic longitude akin to those in declination in the HIPASS levels. However, on further investigation it was apparent that the saw-tooth pattern was generated by steps in the levels between adjacent zoans. Only one value is needed per zoan to eliminate the pattern, and there is sufficient information in the ZOA overlaps to measure these steps directly.

Although a faint pattern is also visible in the Galactic anti-centre region, it seemed best to allow iteration to deal with that naturally. Thus we focus on the Galactic bulge region, measuring the offsets for each latitude only in the extensions where |b| > 6°, well away from bright sources in the plane. These measurements were averaged into one single offset for each pair of zoans from 336° to 32°. The offsets were then converted to zoan-level corrections as in Section 3.7.2, and renormalised to zero for zoan 336° and the zoans preceding it. Corrections for the 48° and 40° zoans were set equal to that for the 32° zoan. Thus only seven independent corrections for the ZOA zoan levels were computed, amounting to 300 mJy beam⁻¹ from zoan 336° to zoan 32°. Once applied, the saw-tooth disappeared quickly; on the next iteration the difference in the levels had dropped by a factor of ×15, with a further factor of ×3 in the iteration after that, reducing it to a negligible value.

Iteration continued until there was no substantive change in the difference maps shown in Figure 12. Convergence was considered to have been achieved after ten passes of refining the HIPASS levels and one further round of computing the ZOA levels from them. Bright source ghosts form the most prominent residual artefacts visible in Figure 12e. These are expected as previously explained. Fine-scale HIPASS streamer noise, the leftovers of imperfect streamer removal, can be detected over most of the difference map. Away from the Galactic centre it has a peak height of about 50 mJy beam⁻¹, but may become more prominent where it crosses bright extended sources. There is no evidence of any residual from the ZOA levels.

HIPASS zone boundaries are also visible in the difference map in the Galactic centre region, mostly a positive excess of up to 100 mJy beam⁻¹ but systematically going negative where the boundary crosses the plane. This behaviour seems to be confined to regions of very strong extended emission. It was not removed by iterative gridding and currently its cause is not understood. Again, following treatment of the saw-tooth residual, there is almost no evidence of the ZOA zoan boundaries. In areas well away from residual artefacts the rms of the difference map is 20 mJy beam⁻¹. Assigning the relative contribution to this from HIPASS and ZOA in the ratio $\sqrt{5}:1$ , the rms of the component maps would be 18 mJy beam⁻¹ and 8 mJy beam⁻¹ as best case noise figures.

As previously explained, the error in the levels determined from the HIPASS zone overlaps in Section 3.7.2 is expected to be highest in the + 22° zone, and this was confirmed, with corrections in the range ±1.5 Jy beam⁻¹, an rms of 320 mJy beam⁻¹, and a mean value consistent with zero.

Figure 13 presents the uncorrected HIPASS levels obtained from the declination zone overlaps together with those resulting from ten passes of crossing HIPASS and ZOA. The worst of the streamers originating in the Galactic plane have disappeared from Figure 13b, but some prominent streamers remain, clearly originating from strong extra-Galactic continuum sources, notably Cen A. The final step is to fix these individually and then smooth the levels.

Figure 13.

HIPASS levels (a) before correction (same as Figure 9), (b) after correction by crossing with ZOA, and (c) the final result after conditioning. Strong Galactic streamers have all disappeared in (b) leaving the prominent Cen A streamer visible just left of centre, together with low-level streamer noise. Same projection and linear greyscale as Figure 9.

3.7.6 Conditioning the HIPASS levels

Figure 13b shows the HIPASS levels obtained after correction by crossing HIPASS and ZOA. Residual streamer noise is visible with peaks to about 100 mJy beam⁻¹, though with several stronger. The source of five of these was readily identified, including Cen A, and it was verified that the offset computed from the relevant HIPASS zone overlaps in Section 3.7.2 had a high value of S_n due to these sources. As they lie outside the ZOA survey, none of them were amenable to correction by the previous methods. However, as these streamers have an obvious source, it was straightforward to apply an offset that would make them disappear into the background streamer noise. For Cen A this required an adjustment of −550 mJy beam⁻¹ (Gaussian peak) over 19 levels (FWHM) spaced by 16 s in RA; for 3C273, +300 mJy beam⁻¹ over 9 levels; for 30 Doradus with two separate streamers, +200 mJy beam⁻¹ and +140 mJy beam⁻¹ both over 10 levels; for PKS J1935-4620, −300 mJy beam⁻¹ over 12 levels; and for PKS J0303-6211, +130 mJy beam⁻¹ over 10 levels. The correction was applied entirely to the offset between the zone containing the source and its neighbour to the north.

After dealing with these outliers, the remaining streamers appear to have a noise-like distribution, with as many positive as negative and no obvious correlation between them. As explained previously, the process of computing HIPASS levels amounts to a simple one-dimensional random walk for which the variance at any point is equal to the sum of the variances of all steps leading to it. Hence the streamer noise is expected to be lower in the south and higher in the north, as is observed qualitatively. Indeed, all of the processing to this point has carefully treated positive and negative streamers equally so as not to introduce biasses that might disturb this random distribution.

The final step then was to smooth the levels in right ascension. Previous processing only used very light smoothing, but considering that zone levels arise from emission that is significantly extended with respect to the 8°.5 scan length, much heavier smoothing in RA is justified on the basis that there are no long, narrow sources of continuum emission that are oriented north–south on the sky. Bear in mind that only the levels are smoothed, not the final map. The question then arises of the maximum valid smoothing length, and what length is sufficient to remove the streamers.

The ZOA levels were used for guidance in assessing a legitimate smoothing length. The strong peak that occurs right on the Galactic plane was found to be the only narrow feature. The zoan levels on either side of it change gradually, with only a few, relatively weak features with a scale length in the range from 1° to 2°. It was also apparent that the ZOA levels are well determined and do not require smoothing. Considering that the HIPASS sampling rate is overwhelmed by a factor of ×5 in the region of the ZOA survey, the choice of smoothing length for the HIPASS levels is not a significant concern in this region. Moreover, in light of the results of Section 3.7 in using ZOA to eliminate HIPASS streamers, smoothing of the HIPASS levels within the ZOA survey area should be kept to a minimum. In practice it was only done to prevent discontinuities on the boundary of the survey.

Several smoothing strategies were investigated, the simplest being to smooth the levels in right ascension. In fact, this is the least effective method because it does nothing to stop streamer formation. By smoothing the offsets, streamers are attenuated close to their source, before they can propagate northwards. However, unless the ZOA region is smoothed identically, offsets do not balance between zones northwards of the Galactic plane, leading to the formation of streamers. Thus the most effective smoothing method was found to be a combination of the two approaches. Starting from the south, the levels for one zone are smoothed, then the levels for the next zone (outside the ZOA) are recomputed from these smoothed levels and the original offsets. The next zone is then smoothed in turn. This strategy attacks streamers at their source, and also has the benefit of smoothing a quantity that will appear directly as emission on the final map.

In practice an empirical approach was used to determine the smoothing length. Essentially the levels were put through a high-pass Gaussian filter and the smoothing length adjusted to best effect, giving little weight to the ZOA survey area. Starting with a short smoothing length, only streamer-like noise made it through the filter—long filaments oriented north–south with no obvious structure that could be related to any celestial emission. As the smoothing length was increased, shorter and wider features appeared that could be related to genuine cosmic emission. A length of 4° FWHM was found to be the best compromise for streamer removal and retention of genuine emission in regions outside the ZOA. The rms of the high frequency streamer components was about 100 mJy beam⁻¹ in the + 22° zone being the worst case. Having determined the appropriate filter length, the levels were then smoothed using a low-pass Gaussian filter of the same filter length. Within the ZOA, a length of 1° was sufficient to avoid discontinuities at the edges of the survey area.

As can be appreciated from Figure 11, the ZOA overlaps the HIPASS scans to a variable extent. Some HIPASS scans and their neighbours have the benefit of a complete intersection, whilst others intersect only partially. In particular, only a small corner of the 200° and 48° ZOA zoans intersect the + 22° HIPASS zone, with the 48° corner proving troublesome. As stated in Section 3.7.4, a correction was computed if, after censoring, at least five pixels remained in the overlap. However, compared with peak values of around 130 pixels, corrections based on such a small number of pixels cannot be said to be fully within the ZOA survey area, nor be treated with the same degree of confidence. Thus, in order to prevent discontinuities on the boundary of the ZOA survey and to clean up nuisance streamers generated in the boundary regions, it was found necessary to use a graduated definition of inclusion. A level was considered to be fully inside the ZOA if the correction was computed from a certain minimum number of pixels, n ₀, and fully outside if there was no correction (in practice, less than five). Progressive smoothing operated within the margins, with the FWHM of the smoothing length varied pro rata between 1° (inside) and 4° (outside) according to the number of pixels used to compute the correction. Noting the gradual effect that changing n ₀ had on the results, a value of n ₀ = 50 pixels was determined empirically.

The final HIPASS levels are shown in Figure 13c. At this stage the levels ranged from −0.34 to +9.1 Jy beam⁻¹.

3.8.1 Blanking unreliable pixels

As can be seen in Figure 3, the reliability of pixels north of δ = +25° reduces as the density of the data gradually diminishes. These pixels were blanked by imposing a cut-off in the value of the relative sensitivity over the whole of the brightness map. A value of 4.0 units was found to be an effective level. On this basis the northern boundary of the map has been allowed to remain ragged. This filter also blanked eight pixels at the SCP, and 60 pixels known to be associated with saturated sources.

As explained in Section 3.1, data flagged by the multibeam correlator were properly handled in this analysis. Pixels affected by flagging were found by producing a map representing the proportion of flagged data that would have (but did not) contribute to each pixel in the brightness map, this type of map being more effective at highlighting the influence of flagged data than the sensitivity map. Apart from source saturation, data might also be flagged because of strong RFI, and its effect was certainly evident in this map. However, the criterion set to exclude obviously saturated pixels also reliably ignored all RFI-affected pixels. Thus, 256 pixels (0.004% of non-blank pixels) were found where more than 15% of the data was flagged. These pixels convincingly matched all of the bright sources suspected of being saturated.

Next, new maps were produced as before, but this time with a 5 arcmin cut-off radius for the gridding kernel instead of 6 arcmin. The density of data in the Galactic plane (ZOA) region may be high enough even with a 2 arcmin radius to be above the relative sensitivity cut-off of 4.0 units. Reducing the cut-off radius should exclude flagged data for pixels on the periphery of strong sources, and indeed 6 of the 256 pixels were reclaimed. Matching pixels in the sensitivity map were updated with the reduced value from the smaller gridding kernel. The radius was further reduced to 4 arcmin, 3 arcmin, and then 2 arcmin with the reclamation of a modest 2, 3, and 5 pixels respectively. Inevitably, a remainder of 240 pixels (additional to the original 60) were too close to strong peaks to be recoverable. These were blanked and the matching pixels in the sensitivity map were zeroed. Saturated sources and the associated number of blanked pixels are identified in Table 3.

Table 3.

A total of 300 pixels were blanked from the following 16 saturated sources.

3.8.2 Brightness calibration

The raw HIPASS data is calibrated reasonably accurately for flux density; all brightness values quoted so far are with respect to this nominal scale. The calibration was checked via Gaussian fits to the peak height of two sources in the compact source map (Figure 1): 1934-638 (14.9 Jy), and Hydra A (40.6 Jy). The mean of the calibration factors (1.11, 1.07) was 1.09.

The final steps required to obtain a fully calibrated flux density image are: (a) to determine and apply a zero level; and (b) to apply the overall flux density scale. Non-negativity required a global offset of +310 mJy beam⁻¹, with the minimum occurring on the northern edge of the north polar spur, close to the NGP. The above scale factor of ×1.09 was then applied.

The final map prior to scaling is presented in Figure 15 for comparison with the previous maps. It should be considered in conjunction with the relative sensitivity map of Figure 16.

Conversion to absolute brightness temperature depends on the unknown zero level and the Jy to K scale factor. The former can be measured using calibration horns; the latter ratio can be calculated if the beam has been accurately measured out to distant sidelobes. Fortunately, we can obtain both values by comparison with the lower resolution all-sky 1420 MHz mapFootnote ⁹ derived from the surveys of Reich (Reference Reich1982), Reich & Reich (Reference Reich and Reich1986), and Reich et al. (Reference Reich, Testori and Reich2001). This plate carrée map was regridded onto a Hammer-Aitoff projection, it being necessary to use an equiareal projection for the purpose. The present map was first regridded onto an equatorial plate carrée projection; then blurred to 35.4 arcmin resolution, first in RA with a 32.3 arcmin FWHM Gaussian kernel (the number of pixels therefore varying with declination) then in declination; and then regridded to match the other map. As the least-squares analysis of offset and scale was distorted by the preponderence of data below 5 K, the data was sorted into 10 mK bins. A factor of 0.44 K/(Jy beam⁻¹) with offset 3.30 K was obtained and used to convert the final map to units of absolute brightness temperature.

The resulting cross-plot of some 470 000 pixels is shown in Figure 14 and shows excellent agreement. Like Haslam et al. (Reference Haslam, Salter, Stoffel and Wilson1982), the Reich et al. maps use a full-beam brightness temperature computed for a very extended beam of 7°. The brightness temperature scale for the Parkes map is therefore consistent with this. A conversion value of T_B /S = 0.44 K/(Jy beam⁻¹) also allows compact source fluxes to be measured accurately (though it is recommended to use the compact source map in Figure 1 for this purpose). Nevertheless, the implied main beam efficiency is only 51% with respect to a Gaussian beam with a HPBW of 14.4 arcmin. This is even lower than the quoted Stockert beam efficiency of 55% (Reich, Reference Reich1982). Values of T_B /S = 0.80 K/(Jy beam⁻¹) are typically used in HI observations of extended objects at Parkes, e.g. Staveley-Smith et al. (Reference Staveley-Smith, Kim, Calabretta, Haynes and Kesteven2003). Some of the lower efficiency is due to the use, in the present work, of all the off-axis beams, the outer of which is 26% less efficient than the central beam. As discussed in Barnes et al. (Reference Barnes2001), there are similar factors that arise from the nature of the gridding function and the subsequent variation of beam size with the angular scale of the source, some of which was absorbed in the previously discussed flux calibration factor of 1.09. Nevertheless, a conversion value closer to T_B /S≈0.57 K/(Jy beam⁻¹) might have been expected. It therefore remains possible that some of the ~30% residual difference is due to sidelobe power on intermediate scales ( < 7°). Structures on this scale may be slightly less prominent than they would be with a telescope with a pure Gaussian beam. However, the excellent match with the lower resolution data (Figure 14) after application of a single overall scale factor, implies high consistency with previous work in the field.

Figure 14.

Cross-plot of the present HIPASS/ZOA map blurred to $35\hbox{$.\hspace{-2.22214pt}^\prime $}4$ resolution versus the 1420 MHz all-sky map of Reich et al. (Reference Reich, Testori and Reich2001).