Background and problem statement

Thiele and Wilhelm (Reference Thiele and Wilhelm2004) argue for a novel research discipline dedicated to time-predictable embedded systems. Berg et al. (Reference Berg, Engblom, Wilhelm, Thiele and Wilhelm2004) outline key design principles for a time-predictable processor:

Computer architects and researchers specializing in real-time computing are investigating processors and architectural designs tailored specifically for real-time systems (Edwards and Lee Reference Edwards and Lee2007). When we refer to “optimized for real-time systems,” we mean architectures that exhibit some form of time predictability. This may mean statically determining a tight upper bound on the worst-case execution time (WCET).

To compare various approaches, it becomes essential to quantify this time predictability. In other words, we need methods to assess and gauge it accurately. This paper explores diverse techniques for conducting such quantification.

Schoeberl (Reference Schoeberl2009) describes the architecture issues that make WCET analysis hard or even impossible. He then describes a time-predictable architecture. This paper does not define what time predictability means. However, its Figure 1 explains intuitively how a time-predictable processor differs from a commercial-of-the-shelf processor. We adopt this figure also as our Figure 1 to discuss the problem we want to solve with this paper.

Figure 1. Distribution of the best-case, average-case and worst-case execution times and the WCET bound of a task on different architectures.

Edwards et al. (Reference Edwards, Kim, Lee, Liu, Patel and Schoeberl2009) define an architecture with repeatable timing. Liu (Reference Liu2012) presents in his PhD thesis a precision timed machine (PRET) that is an implementation of an architecture with repeatable timing. Repeatable timing is a strong property where every execution of a sequence of instructions with the same input data leads to the same execution time. When we include the same initial internal state of a processor (e.g., clearing its caches), a time-predictable processor will also deliver repeatable timing.

Figure 1 shows the different execution times of a single program on four different processors (A, B, C, and D). The different execution times are: The best-case execution time (BCET), the average-case execution time (ACET), and the worst-case execution time (WCET). Execution times between BCET and WCET are possible. The execution time depends on several factors: the program (task), the state of the processor, and input data. Here, we consider the execution times of uninterrupted tasks.

The figure also has an entry for the WCET bound. The WCET bound is an upper, hopefully safe, bound on the execution time a WCET analysis tool can give. There are several reasons why it is usually not possible to give the exact WCET: (a) the complexity of software analysis, e.g., proving infeasible execution paths, and (b) the simplifications on hardware models required to make the analysis computationally practical. Regarding software, infeasible execution paths may significantly impact the WCET bound because the static analysis cannot prove that these paths may never be executed. Similarly, on the hardware side, dynamic features such as branch prediction and cache contents often need to be modeled conservatively to prevent an explosion of the analysis complexity. The difference between the WCET bound and the real WCET is called pessimism. This pessimism is usually not known because it is practically infeasible to know the real, ground-truth WCET. The general aim of WCET analysis tools is to provide a tight upper bound of the real WCET.

Let us focus on the first three processors and their distributions of the execution times. In the general purpose computing view, Processor A is the best processor, as it has the lowest ACET. Due to the complexity of processor A, we have a relatively high WCET bound and pessimism. Processor B is slower than processor A. It is slower in the ACET and also in the WCET. However, the WCET bound of processor B is lower than that of processor A. Therefore, even with a higher WCET than processor A, processor B is a better choice for a real-time system. A processor for real-time systems aims to optimize the WCET bound, even when ACET suffers. Therefore, processor C, now with a lower WCET and WCET bound than processors A and B is the preferred choice for a real-time system. For hard real-time systems, the likely increase in the ACET and BCET is acceptable, because the complete system needs to be designed to reduce the WCET. Note that a processor optimized for low WCET will be generally slower in the average case than a processor optimized for low ACET. Those are two different design optimizations.

Note that those timing distributions are just for a single task, not as a measurement of a processor for all tasks. The variability of the software is included in that figure.

Time-predictability definitions

Several groups have defined different metrics to talk about time-predictable architectures. We describe them and discuss them in chronological order of publication.

Thiele and Wilhelm (Reference Thiele and Wilhelm2004) introduced a foundational definition of time-predictability. A WCET analyzer is assumed to provide an upper bound on the WCET rather than the exact WCET. Furthermore, with complex hardware, it is often impossible to prove that the upper bound is accurate since the timing model used in the analysis is usually based on guesses from reverse engineering the undocumented timing behavior of the processor.

Their approach involves assessing predictability by evaluating the divergence between the actual WCET and the bounded WCET, commonly referred to as the pessimism of the analysis. They also incorporate the disparity between the lower bound and the best-case execution time (BCET) into their framework for predicting execution times. The main issue with this definition is that we usually do not know the exact WCET. Therefore, we cannot compute that predictability metric. Another issue with this definition is that it does not include the execution time. Therefore, assume two architectures, e.g., C and D from Figure 1, that we compare. Assume C has caches that result in a conservative WCET bound. Assume that D is a simple architecture without caches and a simple sequential processor implementation. Therefore, we can better analyze that architecture, and the WCET bound is close to the WCET. However, due to its simplicity, architecture D’s WCET is higher than that of architecture C. The factor between the WCET Bound and the WCET would favor processor D, which is slower than processor C.

However, the question of which processor to choose might also depend on other factors, e.g., power consumption.

Schoeberl (Reference Schoeberl2009) defines:

However, how useful is this definition when we do not know the WCET? And is this the only useful property? Let us discuss processor D in Figure 1. For processor D, the bound is closest to the real WCET. However, it is also the processor with the highest bound and WCET. Is this the better processor for a real-time system? For practical reasons, probably not.

He proposes time-predictable solutions for common architectural artifacts in the rest of the paper. E.g., special caches (method, stack, and special cache), and TDMA-based access to external memory for chip multiprocessors. Pitter and Schoeberl (Reference Pitter and Schoeberl2010) use TDMA-based memory sharing of a multicore version of the JOP Java processor.

Kirner and Puschner (Reference Kirner, Puschner, Min, Pettit, Puschner and Ungerer2011) introduce various definitions of time-predictability. The initial definition centers on the interval between the BCET and the WCET denoted as [BCET, WCET]. A narrower interval signifies enhanced predictability, with BCET equating to WCET indicating the highest predictability. More informally, they also define that the time-predictability of a model is the ability to calculate the execution time of a concrete system. Specifically, they define predictability by multiplying analyzability with stability, including weighting factors. The stability is given by the quotient of ${{BCET} \over {WCET}}$ of the timing model. For WCET-predictability, the stability is ignored, and its factor is set to zero. The analyzability is now equal to the WCET-predictability, as the quotient of WCET and WCET bound. The analyzability of a model is the computational effort needed to calculate the relevant timing properties of the model. For example, a program with data-dependent control flow is less analyzable than one without. Likewise, a processor with memory caching will be less analyzable than one with scratchpad memories. No attempt is made to quantify the analyzability of a timing model. The stability of a timing model is the inherent variability of the model.

Grund et al. (Reference Grund, Reineke, Wilhelm, Lucas, Thiele, Triquet, Ungerer and Wilhelm2011) delve into the realm of predictability and introduce a framework for its definition. This template encompasses three fundamental elements: (1) the property subject to prediction, (2) the origin of uncertainty, and (3) a metric for assessing quality. Their work focuses on the property of time and defines time-predictability, expressed as the ratio between the minimum and maximum execution times. A system achieving a quotient of 1 is considered perfectly predictable. They argue that every system inherently possesses a predictability factor independent of WCET analysis. Therefore, they argue that this assessment will not include the WCET analysis. However, the challenge lies in the typical absence of knowledge regarding real WCET and BCET, rendering the “measurement” of time-predictability impractical.

Program dependency

Time-predictability metrics usually depend on the actual program under consideration. For example, there is a large difference in the WCET bound dependent on where the data is allocated. If the program uses statically allocated data only, a WCET tool can compute the static addresses and model the data cache behavior. However, if the data is allocated on the heap, there is no static information on the data addresses available. Therefore, the cache analysis cannot model the cache behavior and needs to assume cache misses. This will lead to a large difference in the WCET bounds for almost the same program. It is often possible to write a more or less WCET analysis-friendly program, implementing the same logic.

Puschner and Burns (Reference Puschner and Burns2002) present single-path programming as an extreme approach to minimize program-dependent execution time. To produce single-path code, the compiler takes any WCET-analyzable code and converts its control flow to only have a single execution path. If-conversion is used to eliminate data-dependent branching (Allen et al. Reference Allen, Kennedy, Porterfield and Warren1983), while loops are forced to iterate a fixed number of times (the maximum possible.) Predication or conditional-move instructions ensure the program maintains the same behavior. Addition techniques efficiently remove execution-time variation from function calls and memory accesses (Maroun et al. Reference Maroun, Schoeberl and Puschner2023). While it can only be used for WCET-analyzable code, e.g., requiring all loops to have a known upper iteration bound, single-path code is a code-generation technique that is general enough for any real-time application. The limiting factors for single-path code are its hardware requirements, including predicated instructions or conditional-move support and fixed memory access latency.

Paired with a sufficiently time-predictable processor, like Patmos, single-path code can achieve constant execution times. Simply measuring a program’s runtime will thus always produce the same execution time regardless of the input data. The WCET and BCET are thus the same. However, that programming approach has so far been challenged by lower performance than what can be achieved using traditional analysis. More optimization research is therefore required to establish the true performance potential of the single-path approach. However, single-path code exemplifies how two toolchains can differ in predictability only based on the code generation (i.e., the compiler) and analyzer used: analyzed traditional code versus measured single-path code.

Time-predictable processor projects

Several research and commercial projects aim to build a time-predictable processor for real-time systems. Here we list just a few examples; an exhaustive list would be a survey paper on its own.

Dinechin et al. (Reference Dinechin, van Amstel, Poulhies and Lager2014) designed the Kalray manycore processor for time-critical computation. The processor is organized into 16 clusters of 16 cores. The clusters are connected by a NoC with rate control at the sender to support time-predictable message passing with rate control in the sender. Each core within a cluster and the NoC are connected to an SPM with 16 independent memory banks. By carefully selecting the allocation of data and access to the memory banks, the notion of ownership can be implemented in software. The individual processor cores have been designed according to the proposal of Wilhelm et al. (Reference Wilhelm, Grund, Reineke, Schlickling, Pister and Ferdinand2009).

Zimmer et al. (Reference Zimmer, Broman, Shaver and Lee2014) and Zimmer (Reference Zimmer2015) extend the PRET architecture by Lickly et al. (Reference Lickly, Liu, Kim, Patel, Edwards, Lee and Altman2008), calling it FlexPRET to support mixed-criticality systems with fine-grain multithreading. FlexPRET supports two different thread types, hard and soft real-time threads, in the hardware. Both thread types have fixed slots assigned in the fine-grained thread scheduling. However, slots not used by a thread (e.g., because of stalling or because a thread has finished its current release) can be used by the soft real-time threads. FlexPRET implements the RISC-V instruction set, as defined by Waterman et al. (Reference Waterman, Lee, Patterson and Asanovic2011).

May (Reference May2012) present XCore, a multicore architecture developed by XMOS in the vision of PRET architectures. XCore supports fine-grained multithreading similar to the FlexPRET architecture. XMOS supports XCore with a WCET analysis tool.

Schoeberl et al. (Reference Schoeberl, Puffitsch, Hepp, Huber and Prokesch2018) present Patmos, a processor specially designed to be time-predictable. All features, e.g., caches, have been chosen to be WCET analysis friendly. In this paper, we use Patmos as an example for the calculation of the worst-case performance.

Time-predictable processor components

If it is impossible to determine a quantitative assessment of time-predictability for whole processors, it may be useful to look at processor components. For example, caches can be designed for a tighter WCET bound than caches optimized for average performance. One way of doing so is by choosing the cache line replacement strategy. Reineke et al. (Reference Reineke, Grund, Berg and Wilhelm2007) investigate the predictability of various replacement policies. They classify predictability as how fast a policy regains information about the cache’s contents after an information-losing event, such as accessing an unknown address. They establish that the least recently used LRU replacement policy is the most predictable. This is because every access at least establishes information about the cache line accessed (it becomes the most recently used).

The predictability definition used by Reineke et al. (Reference Reineke, Grund, Berg and Wilhelm2007) hinges on the existence of an analyzer that uses knowledge about the replacement policy to track what can and cannot be in the cache at a given program point. This means that the predictability of a cache component is dependent on the analyzer, without which the predictability classification has no effect.

Reineke et al. (Reference Reineke, Grund, Berg and Wilhelm2007) also does not account for the influence of specific programs on the cache predictability. A program performing only dynamic memory accesses will prohibit an analyzer from ever establishing any knowledge about cache contents, regardless of replacement policy. Alternatively, a program that does not use the cache or has a small data memory footprint where all data fit into one way of the cache is not affected by the replacement policy. Another example is a WCET analysis tool, such as platin, which we use in the evaluation. It does not contain a data cache analysis and, therefore, needs to assume a miss on all access to data that is not allocated on the stack. Therefore, the theoretical knowledge that a specific policy is “usually” most beneficial is ineffective in such cases.

In less extreme program examples, there would be a ratio between “predictable” and “unpredictable” cached memory accesses, which should be incorporated into the evaluation of a system, as the ratio would affect how useful the predictability of the cache replacement policy is to the overall system.

Again, we cannot quantify the time-predictability in absolute terms for individual components. We can only state that replacement policy A gives a tighter WCET bound than policy B with a given program and a given analyzer.

Ferdinand et al. (Reference Ferdinand, Heckmann, Langenbach, Martin, Schmidt, Theiling, Thesing, Wilhelm, Henzinger and Kirsch2001) have shown that the perfect analyzer is impractical, as practical analysis speed is best achieved with modular analysis. However, for practical and modern systems, an integrated analysis “is the only one that can give useful results” (Heckmann et al. Reference Heckmann, Langenbach, Thesing and Wilhelm2003). Analyzers have to therefore trade accuracy for practicality, never quite achieving the perfect result.

Axer et al. (Reference Axer, Ernst, Falk, Girault, Grund, Guan, Jonsson, Marwedel, Reineke and Rochange2014) also investigate the predictability of specific components of a system, from architectural components like the pipeline or cache to the compiler and system design. They provide methods of determining the predictability (a value between 0 and 1) of system components, like the cache, and the system as a whole. They recognize that compositionality would be a useful property of predictability, i.e., by composing the predictability metrics of the system components, the predictability of the whole system can be determined. However, they identify no way of achieving compositionality and recognize that the interactions between components might cancel out the predictability of some of them. For example, an unpredictable pipeline would negate the predictability of the cache. Lastly, the predictability measure used is assumed in the presence of “a system-specific optimal analysis” (Axer et al. Reference Axer, Ernst, Falk, Girault, Grund, Guan, Jonsson, Marwedel, Reineke and Rochange2014). Without an analyzer with such an optimal analysis implementation for each system component, the predictability measures cannot be used. Therefore, even when ignoring the lack of composition of component predictability, without a perfect analyzer, everything breaks down.

Worst-case performance

As we have seen, time-predictability is hard to define to be used in practice. For many real-time systems, the WCET bounds of tasks are more important. We propose to quantify this as the worst-case performance (WCP) of a processor and its tooling. For average-case performance, industry and academia have converged on certain benchmark suites, such as the SPEC benchmark. The same can be done for the WCP. We propose using the TACLe benchmark suite (Falk et al. Reference Falk, Altmeyer, Hellinckx, Lisper, Puffitsch, Rochange, Schoeberl, Sørensen, Wägemann, Wegener and Schoeberl2016) to quantify WCP.

Similar to SPEC benchmarks, WCP benchmarks constitute an end-to-end quantification that includes the program, the compiler, the processor, and the memory (hierarchy). As in SPEC benchmarking, we use the geometric mean for the WCP benchmarks as it provides a more balanced comparison. However, while the SPEC benchmarks are executed on actual hardware or emulations of hardware, the WCP benchmarks are being performed by a WCET analysis tool, whose bounds are used for the quantification. A WCP benchmark score for a particular processor will depend on the WCET analysis tool and the processor’s available timing model. This will incentivize designers of time-predictable processors to provide accurate timing models to achieve the highest possible score on the WCP benchmarks. A separate verification flow is needed to verify that the timing model is correct (meaning that the WCET bound is always greater than the actual WCET).

Example WCP

For example, we perform WCP benchmarking for Patmos. Hepp et al. (Reference Hepp, Huber, Knoop, Prokesch and Puschner2015), Huber et al. (Reference Huber, Puffitsch and Puschner2011), and Maroun et al. (Reference Maroun, Dengler, Dietrich, Hepp, Herzog, Huber, Knoop, Prokesch and Puschner2024) provide the WCET analysis tool platin, which is part of the T-CREST platform (Schoeberl et al. Reference Schoeberl, Abbaspour, Akesson, Audsley, Capasso, Garside, Goossens, Goossens, Hansen, Heckmann, Hepp, Huber, Jordan, Kasapaki, Knoop, Li, Prokesch, Puffitsch, Puschner, Rocha, Silva, Sparsø and Tocchi2015). It is integrated with the compiler (LLVM) to generate flow facts (e.g., loop bounds). Platin has a timing model for Patmos, including memory timings for different field programmable gate array (FPGA) boards.

Sieh et al. (Reference Sieh, Burlacu, Honig, Janker, Raffeck, Wagemann and Schroder-Preikschat2017) have extended platin to support the ARM and RISC-V instruction sets. Therefore, we also envision exploring the usage of platin with FlexPRET.

We target Patmos in the version implemented in the low-cost FPGA board DE2-115 which contains an Intel Cyclone-IV FPGA. We configure Patmos with 4 KB method cache, 2 KB stack cache, and 4 KB data cache. Note that our WCET analysis tool does not include a data cache analysis; therefore, each data access is considered a miss with a penalty of 21 clock cycles. However, platin includes a method cache and stack cache analysis.

In Table 1, we show in the second column the WCET bounds for each of the chosen benchmarks produced by platin. We scale the raw cycle counts platin produced to the default 80 MHz frequency of our setup. We only include the programs compiled, run, and analyzed correctly. For two of the sequential programs, loop bounds were invalid and so could not be analyzed. For seven programs platin failed to produce WCET bounds. That leaves us with 16 programs. At the bottom, we have added the geometric mean, which would be the WCP of the combination of our end-to-end toolchain and Patmos processor.

Table 1. WCET bounds and single-path execution times in microseconds for each of the chosen benchmarks and their geometric mean

Another option is to use single-path transformation to achieve constant execution time. Table 1’s third column shows the execution time of each program using single-path code instead of WCET analysis. Because the compiler is immature and has bugs, it failed to compile some of the benchmarks to correct single-path code, so only those compiled and executed correctly are used. Since the execution time of single-path code is both the best and worst-case, it can be directly compared to bounds from tools like platin.

Looking at the actual execution time numbers, we can see that single-path code does not include values for the most long-running programs. Therefore, we do not present the geometric mean for the single-path measurements. Single-path code is longer-running than traditional, analyzed code for all but the two application programs. Therefore, for any fair comparison using WCP, every program chosen as part of the benchmark must be represented in all comparison setups.

The results highlight a problem: We can see that analyzers and compilers can have bugs and fail to produce results. However, do we even know if the results we do have are correct? A WCP would be irrelevant if it were wrong, meaning the first task of any benchmarking is to provide some confidence in the results’ correctness.

One way to gain confidence in WCET analysis results is to run the program repeatedly on different input data and measure the execution time. With sufficient data variation and no instances of execution times exceeding the WCET bound, we can have confidence that the bound is valid. An analogous thing could be done for single-path code, where we ensure the execution is identical for all runs. We propose that a future version of TACLe benchmarks should facilitate such confidence-building measures as crucial for trusting the eventual WCP.

The TACLe benchmark does not provide any such facility. Every program comes with one set of input data and no more. Therefore, it cannot be used to validate the correctness of the bound produced by an analyzer or the execution time of single-path code. We imagine the possibility of building a benchmark that provides diverse input data sets for each program or an automated way of generating data on the fly. For the latter, care must be taken that all the data generated is valid and fits the constraints expected by the program. Otherwise, invalid data might cause errors, e.g., by making a loop iterate an unreasonable number of times.

One observation from comparing WCET bounds with single-path generated code is that better time predictability comes at a cost: a higher execution time. If (average-case) performance-enhancing features, such as out-of-order execution and branch predictors are avoided, we will get lower (average-case) performance. One future challenge is developing performance-enhancing features that help lower the worst-case performance.

In Figure 2, we take the WCET bound produced by platin for each program and compare it to the actual execution time of simply running the program. Notice that for the Dijkstra and mpeg2, the values are incredibly small at close to 0.003 and 0.004, respectively.

Figure 2. Execution time of running each program with traditional code compared to the WCET bound of the traditional code.

We have investigated the two extreme cases Dijkstra and mpeg2. We explored different input data looking to maximize the measured execution time, but it did not increase by more than a factor of 2, which is not a meaningful change compared to their respective WCET bound. The code of Dijkstra is not WCET-friendly. There are many places where the analysis tool has to assume the worst loop bound when, most often, that loop is taken only a few times. For example, the queue of waiting nodes can be up to 1000 nodes long, but most often, it is not. It makes sense that platin produces a bound that is very pessimistic.

The code of mpeg2 is similarly unfriendly to WCET analysis. It consists of many branches where one path does almost nothing while the other performs many computations. Additionally, these branches are nested. To find a safe WCET bound, the analyzer must assume the paths taken are the ones where all the complex branch paths are taken. However, it is likely that such a path will be very rare or impossible to take at runtime.

In summary, these two examples are probably not written to be used in real-time systems where the WCET bound is of primary importance. General guides should be followed when writing programs for real-time systems (Harmon et al. Reference Harmon, Schoeberl, Kirner and Klefstad2008). Therefore, updating the TACLe benchmark suit with programs written for real-time systems would be a good idea. Furthermore, it would be interesting to have a mechanism to vary the input values for some research questions.

We also compare, in Figure 3, the WCET bound to the execution time of the program when using single-path code. We also only include the programs that compile and run correctly. Recall that single-path code has constant execution time, so it can be directly compared to the WCET bound, unlike execution times, which do not necessarily represent the WCET. One would assume that the execution time of single-path code is usually higher than the WCET bound produced by WCET analysis. The two application benchmarks are more efficient in single-path code. This shows options for improvement in platin, and concrete the cache analysis.

Figure 3. Execution time of running each program with single-path code compared to the WCET bound of the traditional code.