Data from the real world

Data analysis is an essential part of scientific research and engineering applications. A proper data analysis method will provide a better understanding of the process under consideration, for example, extracting useful parameters to validate an existing theory or to inspire a new theory. However, data from the real world, such as field-observation data, well-controlled laboratory experiments, or numerical simulation, generally possess several problems. For instance, the data length may be too short to satisfy stationary or ergodic conditions, or the mechanism behind the data may be nonlinear, etc. In Figures 1.1 through 1.3 are several examples from the real world, demonstrating common problems of real datasets.

Figure 1.1a displays a collected oxygen saturation index obtained from a MAREL network (Automatic Monitoring Network for Littoral Environment, Ifremer, France) from the period January 1 to December 31, 2010. The large variation of the measured oxygen saturation index shows the nonstationarity of the data. For example, a high intensity of oxygen saturation index was observed at September 3, 2010. As shown in Section 1.3, to mimic this nonstationary event, high-order Fourier harmonic components are required. Moreover, sometimes the sensor fails to collect data, due to maintenance problems or failure of the system. This missing data problem is typical of field observation data. To emphasize this, we replotted a 10-day portion of the data in Figure 1.1b, which shows a discontinuous curve due to the problem of the data missing. This imposes a difficulty for the Fourier-based data analysis method, for which a uniform time step is often required. To see the irregular time step more clearly, the time interval δt has been shown between two successful measurements in Figure 1.1c. Visually, the time step δt demonstrates a strong intermittent distribution with large values of δt. Another aspect of this data set is that the underlying physical mechanism is nonlinear. It means that if one can write a governing equation for the oxygen saturation index, this equation must be nonlinear. Therefore, the difficulties of this type of data set are missing data/irregular time steps, both nonstationarity and nonlinearity.

Turning to another aspect of the real world data, namely varying sample size, considered here is an example from the global drifter program. Figure 1.2a illustrates several Lagrangian trajectories obtained from the global drifter program.