Centered on the fusion of probabilistic modeling and network representations, this chapter introduces an information-theoretic lens for network learning. It illuminates probability decomposition, clique tree inference structures, and network representation learning, offering insights into probabilistic modeling of intricate relationships in complex systems.

The intricate dynamics of dependence and correlation are explored in this chapter. The discussion spans from linear to nonlinear correlations, encompassing regression and information-theoretic measures. This chapter lays the groundwork for deciphering the complex interplay between variables.

Delving into the fundamentals of machine learning, this chapter dissects supervised, unsupervised, and reinforcement learning approaches. The chapter discusses training, validation, and testing models, furnishing readers with the requisite skills for harnessing machine learning in data-driven modeling.

Expanding on estimation techniques, this chapter focuses on parametric approaches such as maximum likelihood estimation and Bayesian parameter estimation. It unveils methods to fit parametric distributions to empirical data, providing complementary modeling tools to the nonparametric ones discussed in the previous chapter.

Delving into multivariate probabilities, this chapter explores both nonparametric and parametric estimation techniques. From Pearson’s covariance matrix to multivariate normal distribution, readers gain insights into modeling relationships among multiple variables.

Starting from the cornerstone of probability theory, this chapter elucidates essential concepts such as quantiles, expected values, and moments of distribution. It lays the groundwork for understanding both univariate and multivariate probabilities, which are fundamental for probabilistic modeling endeavors.

This chapter explores the pivotal role of modeling as a conduit between diverse data representations and applications in real, complex systems. The emphasis is on portraying modeling in terms of multivariate probabilities, laying the foundation for the probabilistic data-driven modeling framework.

The data landscape has changed almost beyond recognition over the last 30 years. Established methods of collecting, compiling and publishing infor - mation have either been made redundant or radically transformed. We have looked at the drivers of these changes in the previous chapter, with the internet and, more recently, AI reconfiguring data value chains and business models in the process. This chapter explores the current state of the information landscape and considers how much data is being created in the mid-2020s, where it is coming from and how it is being used.

How much data is there?

Precisely mapping and measuring the global information economy is impossible as countries measure things differently and many of the inputs and outputs are hidden behind firewalls and corporate networks. However, approximations can be made based on publicly reported data, government statistics and technology sales. Broadly, we can attempt to measure the monetary value of data and information products produced each year as well as the quantity of data produced, distributed and stored. For the purposes of this book, the utility of considering such figures is to help us better understand broader trends in the production, distribution and use of data and what this might mean for the future of information professionals from a range of disciplines. Understanding the shifting sands of our data-driven environment can help us make better decisions about where to invest our time and resources going forward.

One of the first comprehensive and rigorous attempts to measure the volume of information being generated each year was carried out by economists Peter Lyman and Hal Varian at Berkeley University in 2000 (Lyman and Varian, 2000).

This chapter explores the rapidly developing policies of national and international governmental bodies relating to data and AI, the emerging resulting legislation and the ethical concerns leading to these initiatives. As policies, laws and rulings are evolving to deal with the new challenges posed by AI and the data it feeds off, this chapter presents a snapshot of the legal situation in late 2024; however, the issues underpinning policy and legislative changes are ongoing. These centre on concerns with privacy and data protection, copyright, monopolistic practices and trust as well as efforts to stimulate economic growth and international competitiveness. The chapter focuses on developments in the UK and the US as well as across the EU, which allows for policies and legislation to be compared and highlights different national concerns and priorities.

Ethical concerns

Scientific breakthroughs and new technologies have long been a source of alarm for philosophers, writers and politicians. This is particularly true where questions of what it is to be human and how far machines can replace human activities emerge. Mary Shelley's 1818 creation of the Frankenstein monster highlighted concerns emerging from the enlightenment and the move from a society based on superstition to one more founded on the principles of science (Gunkel, 2024, 3). Science fiction novels and films – from Metropolis in 1927 and Brave New World in 1932 to 2001: A Space Odyssey in 1968 and Bladerunner in 1982 – developed these ideas in different and unsettling ways. While public policies and laws change fairly frequently in response to events and differing priorities, the values and ethics that underpin them are more constant.

The title of this chapter may be provocative to some as it suggests radical change driven by data and AI are leading a revolution in how organisations will operate in the near future. Caution over analysts and pundits calling technological advances ‘revolutions’ is wise (as we shall see in Chapter 5) and there is no shortage of examples where irrational exuberance has got the better of writers and forecasters. As the management guru, Peter Drucker, is quoted as saying, ‘Trying to predict the future is like trying to drive down a country road at night with no lights while looking out the back window’ (Guber, 2014). This is particularly true with trying to forecast how new technologies will evolve and whether they will succeed in the marketplace. Established technology vendors may have a vested interest in playing down the significance of a new innovation developed outside their own company, while businesses launching new products may go in the opposite direction.

Over-confidence in the existing order was demonstrated in 1876 when the president of the telegraph giant, Western Union, was offered the patent on the telephone for US$100,000 and stated, ‘This “telephone” has too many shortcomings to be seriously considered as a means of communication’ (Yeuh, 2019). Less than 30 years later, his company was bought by telephony ‘upstart’ AT&T. On the other side, we have WeWork, once valued at US$65 billion by Goldman Sachs for its innovative use of technology to revolutionise officeletting but sold for approximately one hundredth of that figure a few years later (Neate, 2019; Sherman, 2024). Hubris and a confused business model were to blame there.

While acknowledging the difficulties of predicting technology adoption curves, the contention of this book is that we are at the start of a revolution in business practices driven by data and AI. Chapter 5 will explore how this may take shape over the coming five to ten years. This chapter examines a number of current developments and how organisations are already experimenting with and deploying data-driven AI services.

A brief history of the information industry

Before considering the business and social impacts of the current data-driven AI revolution, it is worth considering how our use of information has evolved over the last 5,000 or so years. In many ways, the forces that led to the rise of information being transcribed for the first time are not too dissimilar to those driving change today. Commerce and the desire to maintain power over disparate groups of citizens have always required the creation, storage and distribution of information in multiple forms and across a variety of media.

One of the earliest forms of transcription was discovered in 1929 by Julius Jordan, a German archaeologist who excavated a collection of clay tablets over 5,000 years old in what is now Iraq. It took almost 50 years for researchers to decipher the markings on the tablets, which turned out to be records of the sale of commodities such as sheep, grain and honey. What many archae - ologists had thought were early forms of poetry or personal correspondence were, in fact, much more mundane, but vital to the orderly flow of goods across the region (Harford, 2017). Further study of the tablets revealed an unexpected sophistication in the way information was transcribed in the clay. As higher quantities of goods needed to be recorded, new ways of representing larger numbers were needed. For example, the sale of three sheep involved the sheep being pressed into the clay three times, but when quantities grew above ten, other symbols were required. These symbols allowed records of ever-larger exchanges to take place.