1 - Introduction
Published online by Cambridge University Press: 05 September 2016
Summary
Why Compact Data Structures?
Google's stated mission, “to organize the world's information and make it universally accessible and useful,” could not better capture the immense ambition of modern society for gathering all kinds of data and putting them to use to improve our lives. We are collecting not only huge amounts of data from the physical world (astronomical, climatological, geographical, biological), but also human-generated data (voice, pictures, music, video, books, news, Web contents, emails, blogs, tweets) and society-based behavioral data (markets, shopping, traffic, clicks, Web navigation, likes, friendship networks).
Our hunger for more and more information is flooding our lives with data. Technology is improving and our ability to store data is growing fast, but the data we are collecting also grow fast – in many cases faster than our storage capacities. While our ability to store the data in secondary or perhaps tertiary storage does not yet seem to be compromised, performing the desired processing of these data in the main memory of computers is becoming more and more difficult. Since accessing a datum in main memory is about 105 times faster than on disk, operating in main memory is crucial for carrying out many data-processing applications.
In many cases, the problem is not so much the size of the actual data, but that of the data structures that must be built on the data in order to efficiently carry out the desired processing or queries. In some cases the data structures are one or two orders of magnitude larger than the data! For example, the DNA of a human genome, of about 3.3 billion bases, requires slightly less than 800 megabytes if we use only 2 bits per base (A, C, G, T), which fits in the main memory of any desktop PC. However, the suffix tree, a powerful data structure used to efficiently perform sequence analysis on the genome, requires at least 10 bytes per base, that is, more than 30 gigabytes.
The main techniques to cope with the growing size of data over recent years can be classified into three families:
Efficient secondary-memory algorithms. While accessing a random datum from disk is comparatively very slow, subsequent data are read much faster, only 100 times slower than from main memory. Therefore, algorithms that minimize the random accesses to the data can perform reasonably well on disk.
Information
- Type
- Chapter
- Information
- Compact Data StructuresA Practical Approach, pp. 1 - 13Publisher: Cambridge University PressPrint publication year: 2016
References
and (1964). Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables. Dover, 9th edition.
,, and (2015). Succinct: Enabling queries on compressed data. In Proc. 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI), pages 337–350.Google Scholar
, , and (1974). The Design and Analysis of Computer Algorithms. Addison-Wesley.
, , , and (2009). Introduction to Algorithms. MIT Press, 3rd edition.
and (2006). Elements of Information Theory. Wiley, 2nd edition.
and (2011). Space-efficient preprocessing schemes for range minimum queries on static arrays. SIAM Journal on Computing, 40(2), 465–492.Google Scholar
and (2007). The cell probe complexity of succinct data structures. Theoretical Computer Science, 379(3), 405–417.Google Scholar
(2011). Compressed Suffix Trees: Design, Construction, and Applications. Ph.D. thesis, Ulm University, Germany.
and (2014). Optimized succinct data structures for massive data. Software Practice and Experience, 44(11), 1287–1314.Google Scholar
, , and (1994). Concrete Mathematics – A Foundation for Computer Science. Addison-Wesley, 2nd edition.
and (2013). Design of practical succinct data structures for large data collections. In Proc. 12th International Symposium on Experimental Algorithms (SEA), LNCS 7933, pages 5–17.Google Scholar
(1997). Algorithms on Strings, Trees and Sequences: Computer Science and Computational Biology. Cambridge University Press.
and (2012). Computer Architecture: A Quantitative Approach. Morgan Kauffman, 5th edition.
(1988). Succinct Data Structures. Ph.D. thesis, Carnegie Mellon University.
, editor (2016). Encyclopedia of Algorithms. Springer, 2nd edition.
(1998). The Art of Computer Programming, volume 3: Sorting and Searching. Addison- Wesley, 2nd edition.
, , , and (2013). Accurate and compact large vocabulary speech recognition on mobile devices. In Proc. 14th Annual Conference of the International Speech Communication Association (INTERSPEECH), pages 662–665.Google Scholar
and (2010). Fast and accurate long-read alignment with Burrows-Wheeler transform. Bioinformatics, 26(5), 589–595.Google Scholar
, , , and (2015). Genome-Scale Algorithm Design. Cambridge University Press.
(1984). Data Structures and Algorithms 1: Sorting and Searching. EATCS Monographs on Theoretical Computer Science. Springer-Verlag.
(1996). Tables. In Proc. 16th Conference on Foundations of Software Technology and Theoretical Computer Science (FSTTCS), LNCS 1180, pages 37–42.Google Scholar
(2005). Data Streams: Algorithms and Applications. Now Publishers.
(2013). Bioinformatics Algorithms: Sequence Analysis, Genome Rearrangements, and Phylogenetic Reconstruction. Oldenbusch Verlag.
(2015). Encoding data structures. In Proc. 9th International Workshop on Algorithms and Computation (WALCOM), LNCS 8973, pages 1–7.Google Scholar
(1992). Compared to What? An Introduction to the Analysis of Algorithms. Computer Science Press.
(1999). Parallel Processing and Parallel Algorithms: Theory and Computation. Springer.
and (2013). An Introduction to the Analysis of Algorithms.Addison-Wesley- Longman, 2nd edition.
and (2011). Algorithms. Addison-Wesley, 4th edition.
and (2011). Unary data structures for language models. In Proc. 12th Annual Conference of the International Speech Communication Association (INTERSPEECH), pages 1425–1428.Google Scholar
(2008). Algorithms and Data Structures for External Memory. Now Publishers.
Accessibility standard: Unknown
Why this information is here
This section outlines the accessibility features of this content - including support for screen readers, full keyboard navigation and high-contrast display options. This may not be relevant for you.Accessibility Information
Accessibility compliance for the PDF of this chapter is currently unknown
and may be updated in the future.