Skip to main content

Enabling Near Real-Time Remote Search for Fast Transient Events with Lossy Data Compression

  • Dany Vohl (a1) (a2), Tyler Pritchard (a1) (a3), Igor Andreoni (a1) (a3) (a4) (a5), Jeffrey Cooke (a1) (a3) (a5) and Bernard Meade (a1) (a6)...

We present a systematic evaluation of JPEG2000 (ISO/IEC 15444) as a transport data format to enable rapid remote searches for fast transient events as part of the Deeper Wider Faster programme. Deeper Wider Faster programme uses ~20 telescopes from radio to gamma rays to perform simultaneous and rapid-response follow-up searches for fast transient events on millisecond-to-hours timescales. Deeper Wider Faster programme search demands have a set of constraints that is becoming common amongst large collaborations. Here, we focus on the rapid optical data component of Deeper Wider Faster programme led by the Dark Energy Camera at Cerro Tololo Inter-American Observatory. Each Dark Energy Camera image has 70 total coupled-charged devices saved as a ~1.2 gigabyte FITS file. Near real-time data processing and fast transient candidate identifications—in minutes for rapid follow-up triggers on other telescopes—requires computational power exceeding what is currently available on-site at Cerro Tololo Inter-American Observatory. In this context, data files need to be transmitted rapidly to a foreign location for supercomputing post-processing, source finding, visualisation and analysis. This step in the search process poses a major bottleneck, and reducing the data size helps accommodate faster data transmission. To maximise our gain in transfer time and still achieve our science goals, we opt for lossy data compression—keeping in mind that raw data is archived and can be evaluated at a later time. We evaluate how lossy JPEG2000 compression affects the process of finding transients, and find only a negligible effect for compression ratios up to ~25:1. We also find a linear relation between compression ratio and the mean estimated data transmission speed-up factor. Adding highly customised compression and decompression steps to the science pipeline considerably reduces the transmission time—validating its introduction to the Deeper Wider Faster programme science pipeline and enabling science that was otherwise too difficult with current technology.

Corresponding author
7 Email:
Hide All
Andreoni I., Jacobs C., Hegarty S., Pritchard T., Cooke J., & Ryder S. 2017, PASA, 34, 37
Astropy Collaboration, et al. 2013, A&A, 558, A33 10.1051/0004-6361/201322068 2013A&A...558A..33A
Belmon L. 1998, PhD thesis, University of Paris XI Orsay
Belmon L., Benoit-Cattin H., Baskurt A., & Bougeret J.-L. 2002, A&A, 386, 1143 10.1051/0004-6361:20020225 2002A&A...386.1143B
Bradley L., et al. 2016, astropy/photutils v0.2.2
Brueckner G. E., et al. 1995, SoPh, 162, 357 10.1007/BF00733434 1995SoPh..162..357B
Clark A. 2008, IEEE ISPM, 25, 146 10.1109/MSP.2008.918681 2008ISPM...25..146C
Cohen A., Daubechies I., & Feauveau J.-C. 1992, Communications on pure and applied mathematics, 45, 485
Diehl T., Dark Energy Survey Collaboration 2012, PhPro, 37, 1332 10.1016/j.phpro.2012.02.472 2012PhPro..37.1332D
Flaugher B. L., et al. 2012, in Proc. of SPIE, Vol. 8446, Ground-based and Airborne Instrumentation for Astronomy IV, eds. McLean I.S., Ramsay S.K., & Takami H. (Bellingham: SPIE), 844611 10.1117/12.926216
Flaugher B., et al. 2015, AJ, 150, 150 10.1088/0004-6256/150/5/150 2015AJ....150..150F
Folk M., Heber G., Koziol Q., Pourmal E., & Robinson D. 2011, in Proc. of the EDBT/ICDT 2011 Workshop on Array Databases. AD ’11 (New York: ACM), 36
ISO/IEC 15444-1:2000 2000, Technical report, Information technology – JPEG 2000 image coding system – Part 1: Core coding system
Kitaeff V., Cannon A., Wicenec A., & Taubman D. 2015, A&C, 12, 229 10.1016/j.ascom.2014.06.002
Klebesadel R. W., Strong I. B., & Olson R. A. 1973, ApJ, 182, L85 10.1086/181225 1973ApJ...182L..85K
Labrum N., McLean D., & Wild J. 1975, Methods in Computational Physics, Vol. 14, Radio Astronomy (New York: Academic Press)
Li J. 2003, Modern Signal Processing, 46, 185
Li L.-X., & Paczyński B. 1998, ApJ, 507, L59 10.1086/311680 1998ApJ...507L..59L
Lorimer D. R., Bailes M., McLaughlin M. A., Narkevic D. J., & Crawford F. 2007, Science, 318, 777 10.1126/science.1147532 2007Sci...318..777L
Meade B., et al. 2017, PASA, 34, 23 10.1017/pasa.2017.15
Miller J. M., Fabian A. C., & Miller M. C. 2004, ApJ, 614, L117 10.1086/425316 2004ApJ...614L.117M
Morii M., Ikeda S., Sako S., & Ohsawa R. 2017, ApJ, 835, 1 10.3847/1538-4357/835/1/1 2017ApJ...835....1M
Pence W., Seaman R., & White R. L. 2011, in ASP Conf. Ser., Vol. 442, Astronomical Data Analysis Software and Systems XX, eds. Evans I. N., Accomazzi A., & Mink D. J., & Rots A. H., 493
Pence W., White R. L., Greenfield P., & Tody D. 2000, in ASP Conf. Ser., Vol. 216, Astronomical Data Analysis Software and Systems IX, eds. Manset N., Veillet C., & Crabtree D. (San Francisco: ASP), 551
Pence W. D., White R. L., & Seaman R. 2010, PASP, 122, 1065 10.1086/656249 2010PASP..122.1065P
Peters S. M., & Kitaeff V. V. 2014, A&C, 6, 41 10.1016/j.ascom.2014.06.003 2014A&C.....6...41P
Rabbani M., & Joshi R. 2002, Signal Processing: Image Communication, 17, 3 10.1016/S0923-5965(01)00024-8
Rest A., et al. 2005, ApJ, 634, 1103 10.1086/497060 2005ApJ...634.1103R
Rissanen J., & Langdon G. G. 1979, IBMJ, 23, 149 10.1147/rd.232.0149
Shannon C. 1948, Bell System Technical Journal, 27, 379
Smithsonian Astrophysical Observatory 2000, SAOImage DS9: A utility for displaying astronomical images in the X11 window environment, Astrophysics Source Code Library (ascl:0003.002)
Tomasi M. 2016, A&C, 16, 88 10.1016/j.ascom.2016.04.004 2016A&C....16...88T
Valdes F., Gruendl R., & DES Project 2014, in ASP Conf. Ser., Vol. 485, Astronomical Data Analysis Software and Systems XXIII, eds. Manset N. & Forshay P. (Honolulu: ASP), 379
Vohl D. 2013, Master’s thesis, Université Laval
Vohl D., Fluke C., & Vernardos G. 2015, A&C, 12, 200 10.1016/j.ascom.2015.05.003
Wells D. C., Greisen E. W., & Harten R. H. 1981, A&AS, 44, 363 1981A&AS...44..363W
White R. L., & Percival J. W. 1994, in SPIE Conf. Ser., Vol. 2199, Advanced Technology Optical Telescopes V, ed. Stepp L. M. (Bellingham: SPIE), 703
Wu Z., & Chin O. B. 2014, Big Data Research, 1, 1 10.1016/j.bdr.2014.08.002
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Publications of the Astronomical Society of Australia
  • ISSN: 1323-3580
  • EISSN: 1448-6083
  • URL: /core/journals/publications-of-the-astronomical-society-of-australia
Please enter your name
Please enter a valid email address
Who would you like to send this to? *



Altmetric attention score

Full text views

Total number of HTML views: 3
Total number of PDF views: 16 *
Loading metrics...

Abstract views

Total abstract views: 101 *
Loading metrics...

* Views captured on Cambridge Core between 5th September 2017 - 17th November 2017. This data will be updated every 24 hours.