3.1. Frameworks for service development

Most closely related to the present publication is Maree and Bagle, (Reference Maree and Bagle2022), in which a service-based approach to create digital twins of buildings is presented. Similar to our work, the article strives to develop a framework. The main difference is that their work is much broader, that is, the framework covers not only forecasting and optimization but also data storage, thermal models of buildings, and how these can be learned from data. Consequently, Maree and Bagle, (Reference Maree and Bagle2022) do not handle the aspect of forecasting and optimization services at a comparable depth as in the present work. For example, it does not contain any requirements analysis, a detailed discussion about the technical design and implementation of the services, nor does it provide a systematic approach to derive new forecasting or optimization services from existing code. Furthermore, their work does not contain any hint about a potential publication of the corresponding source code. Thus we conclude that their work, unlike ours, is not a reasonable basis for deriving forecasting and optimization services for energy management applications.

On the other hand, larger Machine Learning (ML) frameworks, for example, PyTorchFootnote ³ and MLflow,Footnote ⁴ provide the functionality to expose an ML model as a web service with a Representational State Transfer (REST) API. However, to the best of our knowledge, no solution exists that supports triggering the training models from API calls out of the box,Footnote ⁵ that is, that allows fitting system-specific parameters as our framework does. It thus appears that utilizing the framework developed in the present work is significantly more advantageous for service developers. This is particularly the case if one considers that our framework has been explicitly designed to minimize the necessary effort for the development and operation of forecasting and optimization services for EMSs at scale.

3.2. Forecasting and optimization services

Several publications have been identified, beyond Maree and Bagle, (Reference Maree and Bagle2022), that utilize the concept of forecasting and/or optimization components wrapped into services (Mohamed et al., Reference Mohamed, Al-Jaroodi and Jawhar2018; Lenk et al., Reference Lenk, Arnoldt, Rösch and Bretschneider2020; Marinakis et al., Reference Marinakis, Doukas, Tsapelas, Mouzakitis, Sicilia, Madrazo and Sgouridis2020; Dengler et al., Reference Dengler, Lalbakhsh, Bazan, Dayaratne, Liebmann and German2023; Galenzowski et al., Reference Galenzowski, Waczowicz, Meisenbacher, Mikut and Hagenmeyer2023; Hill et al., Reference Hill, Pieper, Bruhn, Schönfeldt and Penaherrera Vaca2023). Regarding approaches not documented in academic publications, one should first consider that several providers exist operating web APIs for the retrieval of weather-related data or forecasts, partly as free or commercial offerings, for example, Bright Sky,Footnote ⁶ Open Meteo,Footnote ⁷ SoDa,Footnote ⁸ Solcast,Footnote ⁹ and Forecast.Solar.Footnote ¹⁰ The latter two offer additional services related to forecasting PV power generation. Closely related to the latter is NIXTLAs TimeGPT,Footnote ¹¹ a commercial service for generic time series forecasting. Finally, it is worth mentioning the Building Energy ModelingFootnote ¹² service provided by Schneider Electric as part of their EcoStruxure platform, more details about the latter are in the following section. The service allows users to learn the thermal energy consumption pattern of buildings from data.

In contrast to our work, none of the publications or services referenced in this section present a framework for deriving forecasting and optimization services for energy management applications. Thus, these services are neither in conflict with the present work nor do they provide any substantial input for the requirements analysis or design concept presented below. While some of the mentioned offerings could be reasonably utilized by EMSs, the main issue is that each of these services covers only a fraction of the typically required functionality, while none provides a generic approach to derive the remaining necessary forecasting and optimization services. Nevertheless, the pure existence of these publications and services can be considered as strong advocacy for the general concept of forecasting and optimization services and, thus, the relevance of the present article. Furthermore, it is worth noting that the scientific research underlying the referenced publications would very likely have benefited substantially from using the framework proposed in the present work.

3.3. Energy management systems, platforms, communities and market places

A relevant platform that has been developed and utilized for years in EU projects is FIWARE (Cirillo et al., Reference Cirillo, Solmaz, Berz, Bauer, Cheng and Kovacs2019). FIWARE is a general-purpose IoT Platform (Cirillo et al., Reference Cirillo, Solmaz, Berz, Bauer, Cheng and Kovacs2019) managed by the FIWARE Foundation (Rodriguez et al., Reference Rodriguez, Cuenca, Ortiz, Camarinha-Matos, Afsarmanesh and Rezgui2018). It is used in various use cases, including smart farming (Rodriguez et al., Reference Rodriguez, Cuenca, Ortiz, Camarinha-Matos, Afsarmanesh and Rezgui2018), smart buildings, and smart grids (Blechmann et al., Reference Blechmann, Sowa, Schraven, Streblow, Müller and Monti2023). The heart of FIWARE is the so-called context broker, which receives data from data providers (e.g., sensors), stores the latest information, and provides it to data consumers (e.g., some services). Aside from the context broker, there are various different solutions for data processing and storage that can be connected to the broker, as well as a set of “smart data models” that have been used in different applications. With all these, the FIWARE ecosystem provides many different building blocks that can be used in energy management. These building blocks, however, are very heterogeneous and generic, as their only shared foundation is the integration with the context broker and the underlying Next Generation Service Interface (NGSI). Therefore, FIWARE is less an alternative to the proposed service framework and more a platform into which derived services could be integrated.

On the other hand, several commercial approaches exist that are similar to our service-based forecasting and optimization concept. In particular noteworthy are the platform solutions from Siemens (Building X),Footnote ¹³ Bosch (NEXOSPACE),Footnote ¹⁴ and Schneider Electric (EcoStruxure).Footnote ¹⁵ The latter two appear conceptually similar, that is, the platforms provide several functionalities for smart building operation, including energy management, but require that proprietary hardware (i.e., gateways) must be installed in the building that should be connected to the respective platform. This is a clear contrast to the Siemens solution, which is advertised with an open API concept and connectivity to third-party systems, while seemingly offering similar functionality like the other two. No evidence was found that any of the three companies offer a framework like introduced in this work. However, all three vendors claim that their platform solutions can be extended by third parties and offer a marketplace for applications that can be integrated. However, publishing extensions on the marked places must be explicitly granted by the respective company and is subject to licensing fees. Thus, we conclude that none of the three platforms is indeed a viable alternative to this work as neither empowers third parties to develop and operate forecasting and optimization services for energy management applications that are independent of the respective vendor. Finally, it is worth mentioning that Schneider Electric and Bosch both offer EMSs for private households, for example, Bosch Smart HomeFootnote ¹⁶ and HEMSlogic.Footnote ¹⁷ While both of these systems apparently use some form of cloud-based optimization, there seems to be no possibility to directly interact with these forecasting and optimization services or to integrate third-party services as an alternative. However, we again perceive that the existence of Siemens, Schneider Electric, and Bosch solutions, providing cloud services for smart building operation and energy management, strongly advocates the concept underlying this work.

Finally, we find it important to discriminate our work from the Open Energy PlatformFootnote ¹⁸ as well as from OpenEMS,Footnote ¹⁹ two projects well-known among scientific researchers. The first of these is a community effort to establish a collection of tools supporting the work with and publication of energy-related datasets, with a focus on energy system modeling. This is clearly disjoint from our goal to provide tooling for the implementation of forecasting and optimization services for energy management applications. On the other hand, OpenEMS is a fully functional, open-source, and free-to-use EMS. While it does, in fact, contain a limited number of forecasting and optimization algorithms, providing these is not the essential task of the software. The latter is particularly true as OpenEMS is usually operated on edge devices with little computing power, which limits the applicability of modern ML-based forecasting and optimization methods. However, it is absolutely reasonable to extend OpenEMS with an Energy Service Generics (ESG) compatible client, to allow the integration of forecasting and optimization services derived from our framework, and we plan to demonstrate this in future work.

4.1. Analysis of application areas

The basis for the requirement elicitation process is an analysis of application areas, with which we aim to provide more context and information on the environment, in addition to Section 2. Domain knowledge and an understanding of the application’s context are important for the quality of the requirements (Antonelli et al., Reference Antonelli, Rossi, do Prado Leite and Oliveros2012; Alebrahim et al., Reference Alebrahim, Heisel and Meis2014). Therefore, to aid in the formulation of requirements, this analysis defines the relevant application areas or domains (Loucopoulos and Champion, Reference Loucopoulos and Champion1988), and is carried out by compiling typical applications for EMSs, as well as their characteristics and distinguishing factors. For this purpose, we consider the three application areas private households, commercial buildings, as well as districts and areas separately, and concisely illustrate the individual goals and system specifics. The latter, motivated by ISO 25010 (ISO/IEC, 2023), is achieved by pointing out functional, efficiency, compatibility, interaction, reliability and safety, security, as well as maintainability and flexibility aspects. Table 1 summarizes key differences and similarities between the application areas.

Table 1. Comparison of three areas in which EMSs are used

In private households, nearly two-thirds of the energy demand is used for space heating. They account for 27% of the final energy demand in the EU, of which only 25% is electricity (Eurostat, 2023), but this is expected to rise due to the ongoing electrification of heat and transport due to electric vehicles and heat pumps (Ruhnau et al., Reference Ruhnau, Bannik, Otten, Praktiknjo and Robinius2019). Electricity generation by local PV plants is also growing rapidly, leading to an increase in households that produce parts of their electricity consumption themselves (often called “prosumers”) (Kotilainen, Reference Kotilainen2019; Sovacool et al., Reference Sovacool, Barnacle, Smith and Brisbois2022). The usual goal is to optimize local PV usage and minimize the electricity needed from the public grid by using flexible devices, such as batteries, and shifting flexible electricity demand. In order to do this, private prosumer households often use EMSs to control their batteries, heat pumps, and/or charging processes (Zafar et al., Reference Zafar, Bayhan and Sanfilippo2020). They can work rule-based (Berkes and Keshav, Reference Berkes and Keshav2024), which in simple cases also produces optimal results, for instance when there is a flat electricity tariff by storing all excess PV production and discharging whenever there is a deficit, or use all sorts of optimization algorithms, including mixed integer linear programming, genetic algorithms, particle swarm optimization and more (Henggeler Antunes et al., Reference Henggeler Antunes, Alves and Soares2022; Srilakshmi and Singh, Reference Srilakshmi and Singh2022). Home EMSs used in private households can either be provided as cloud services, for example, by electricity providers or operated locally on an edge device, for example, a Raspberry Pi. In the latter option, EMSs can be operated based on open-source smart home systems like Home AssistantFootnote ²⁴ or OpenHABFootnote ²⁵ which can be installed and used by everyone but may be limited in terms of computing power. Hardware interoperability on the building level is a challenging task, due to a lack of standards and many vendor-specific solutions. The provided user interfaces vary depending on the intended user group. Solutions like OpenHAB allow, for instance, the creation of own control rules, while others provide only simple visualizations. All functions should be provided without interruption, to ensure user comfort, but usually outages would only lead to loss of comfort for the affected household(s). Systems should be designed to avoid damage to devices and users. Especially in private households, the limited computing power of local EMSs makes it favorable to outsource optimization, load prediction, or PV forecast to a cloud service. However, a strong argument for using local systems is the high level of privacy protection, as no data on electricity consumption, which can be used to draw conclusions about residents’ behavior, has to be shared with cloud providers (Boiko et al., Reference Boiko, Komin, Malekian and Davidsson2024). Maintainability and (software) flexibility are crucial for EMS developers and providers, especially for offering their customers continued safe and secure systems and allowing support for more and new hardware.

In commercial buildings, energy management algorithms, for example, proposed by Oldewurtel et al. (Reference Oldewurtel, Parisio, Jones, Gyalistras, Gwerder, Stauch, Lehmann and Morari2012), Chen et al. (Reference Chen, Cai and Bergés2019), and Ding et al. (Reference Ding, Du and Cerpa2019), typically address the optimization of the Heating, Ventilation, and Air Conditioning (HVAC) system, controlled centrally or for rooms individually. Usually, these buildings are equipped with a Building Automation System (BAS) on which a Rule-Based Control (RBC) strategy is implemented. The latter is replaced with an optimization-based approach given an EMS is installed. The efficiency, both from a software and energy perspective, varies with the employed algorithms and depends on the local systems (Al-Ghaili et al., Reference Al-Ghaili, Kasim, Al-Hada, Jørgensen, Othman and Wang2021). Compatibility, like in the residential case, can be a challenge, however, with larger facilities and larger associated investments, customized integrations are more reasonable than in the residential case. In commercial buildings, the correct operation can be of critical importance for the organization utilizing the building. Therefore, the building optimization system might have to be executed on-premise to prevent outages caused by internet failures. Using cloud-based energy management systems in commercial buildings can also come with challenges regarding privacy and security (Anthi et al., Reference Anthi, Javed, Rana and Theodorakopoulos2018). Commercial buildings might be utilized by organizations that are privacy-sensitive and thus do not permit data to be stored in the cloud. Other organizations, however, might be rather price-sensitive and hence prefer to use an optimization algorithm provided as a cloud service while configuring the BAS to fall back to RBC in case of connection issues. From a maintainer and vendor perspective, again, maintenance and flexibility are important for the operation of existing systems and the further development of the product.

Districts and areas differ from those categories due to their size and, most importantly, the involvement of energy grids. One major reason to conduct energy management on the level of facilities, districts, and even on a regional scale is grid operation. With increasing decentralized generation, especially from renewable energy sources, and increasing demands from electrification, the need for monitoring the utilization of the grid and its power quality (see Chawda et al., Reference Chawda, Shaik, Shaik, Padmanaban, Holm-Nielsen, Mahela and Kaliannan2020) and actively influencing energy flows to prevent or resolve undesired situations is rising (e.g., Volk et al., Reference Volk, Lakenbrink, Kurka and Rupp2017). Energy management on an area level, therefore, often considers the associated energy grids, especially in the case of micro-grids. Another reason for area-level energy management is the optimal, for example, cost-minimal, operation of all the generators, storage systems, and flexible loads in the area (e.g., Roccotelli et al., Reference Roccotelli, Mangini and Fanti2022). Control of the different flexible devices and/or buildings in the area can be achieved with more or less direct mechanisms, ranging from direct device access to indirect, highly aggregated control signals (Förderer et al., Reference Förderer, Hagenmeyer and Schmeck2022). The practical implementation and derived qualities like efficiency, reliability, and safety, depend on the selected orchestration mechanism, local regulation, and the characteristics of the area in question, for example, who owns the devices and energy grids and whether there are any fees for using the public grid in a given scenario. Since, in a region, there can be any number of commercial and residential buildings combined, the interoperability challenge is amplified manifold. Standardized interfaces, models, and EMSs for each building can alleviate this challenge (Khalid, Reference Khalid2024). Users may be provided with user interfaces for checking the current regional status and history or making inputs, such as electric vehicle charging settings. In districts and areas, privacy-sensitive data, for example, of many households, may need to be protected, which can be done using data aggregation due to the larger scale (Kursawe et al., Reference Kursawe, Danezis and Kohlweiss2011; Langer et al., Reference Langer, Skopik, Kienesberger and Li2013; Varenhorst et al., Reference Varenhorst, Hoogsteen, Gerards and Hurink2024). Here, a higher level of aggregation and abstraction is additionally beneficial to keep the amount of data that has to be managed and the computation times on an acceptable level, also resulting in a need for different models. Aggregation is especially important on higher grid and system levels. Smart energy-optimized areas may, for instance, aggregate their flexibility on the feeder level (e.g., Volk et al., Reference Volk, Lakenbrink, Kurka and Rupp2017). In such a scenario, control signals need to be disaggregated for their implementation upon reception. Reliable and safe operations are especially important on the regional level, as faults may leave many buildings without energy. For achieving reliable and safe operation, maintainability and flexibility in software are especially helpful on this level, compared to the other two.

From a general perspective, the basic building blocks needed for energy management in all three application areas are very similar, that is, functionality for determining and assessing the current systems state, forecasting of energy-related time series, optimization of load schedules or similar control signals, and controllers implementing the schedules. The implementation, however, may vary due to the different properties and specific demands present in the application areas. In all three areas, privacy concerns need to be taken into account due to the presence of privacy-sensitive information.

4.2. Functional requirements

With the analysis of application areas presented in the previous section and the described typical energy management applications in mind, we now derive functional requirements.

The first requirement directly results from the service and stakeholder concept discussed in Section 2. It is:

Here, the intention of the service provider clearly is to allow EMSs to utilize the existing forecasting or optimization algorithm by interacting with the web API of the service.Footnote ²⁶ Hence the second and third requirements are:

The forecasting or optimization algorithms wrapped by the service framework will usually require some form of input data. Considering a PV power generation forecast as an example, this could be the global position and time of the target system. Furthermore, the data format returned by a forecasting or optimization algorithm will obviously be specific to it and should contain all the information the algorithm needs for processing the expected result. The latter includes constraints that should be obeyed by optimization algorithms. Therefore, the fourth requirement is:

Some services may implement system-specific parameters that must be fitted utilizing historical measurements of the system subject to forecast or optimization as a prerequisite for high-quality results. Note that the algorithm for fitting the system-specific parameters is considered to be a part of the existing forecasting or optimization code. As services should be usable by a large number of EMSs, this fitting process should be manageable via the service API.Footnote ²⁷ Returning to the PV power generation forecast example, one could conceive that the forecasting code contains a small neural network that has been trained using the power generation of several PV systems, thus representing an average system. However, if power generation measurements of a specific PV system are available, it is possible to adapt (fit) the weights and bias terms of the neural network (system-specific parameters) such that the prediction error is minimized for the specific system. The fitting procedure is part of the service and the fitting process can be initiated by calling the respective API endpoint with the required input data, which would be a time series of historic power generation data for the PV power generation forecast example. Furthermore, we need to consider privacy-sensitive users, that is, users that do not accept any of their data to be stored in a cloud database, which implies that it must be possible for EMSs to store the fitted parameters locally.Footnote ²⁸ Finally, we need to consider EMSs that are not capable of reliably storing historic recordings of measurements or fitted parameters on-premise, for example, very likely a large fraction of EMSs operating in private households. While this seems like a major difference at first glance, it turns out that this scenario imposes no additional requirements for the service. The discussion behind this finding is out of scope at this point but can be found in Appendix A. The resulting requirements are thus:

API calls made by an EMS may take a significant amount of time before the result becomes available. Consider, for example, the service providing PV power generation forecasts used as a running example for which fitting the system-specific parameters involves training a neural network that might take several minutes to hours. On the other hand, computing forecasts or optimized schedules might require a decent amount of time too, for example, if computing an optimized schedule for a larger building involves solving a complex linear program. As such response times are different from typical values of web services, we formulate it as an additional requirement:

Finally, it is well known that documentation is important for the widespread adoption of APIs (Hunter, Reference Hunter2017; Jin et al., Reference Jin, Sahni and Shevat2018). Here documentation refers to the description of the functionality of a service, in particular its API and the data format for interactions with the latter. Furthermore, development efforts can be reduced by automatically generating the documentation from the corresponding source code, which is additionally beneficial as it prevents that changes in the code are not reflected in the documentation. The resulting final functional requirement is thus:

4.3. Non-functional requirements

Extending the content above, this section presents non-functional requirements for the service framework. To this end, we first consider the envisioned target state that professional service providers operate services that are utilized by a large number of EMSs. Thus, the availability of these services is very likely of critical importance for the intended functioning of a large number of EMSs. This implies that service providers need to apply state-of-the-art computing cluster techniques for operation. Furthermore, the data exchanged between EMS and services may contain sensitive information and should thus be encrypted,Footnote ²⁹ especially as a large share of EMSs will likely communicate with services over the public internet. Finally, operating services may require significant compute resources and energy. A service provider may, hence, wish to restrict access to services to certain EMSs. This leads to the following requirements:

Beyond the commercial aspect, an important intended application of the service framework is to empower academic researchers to derive functional services from existing forecasting or optimization code. For this, one needs to consider the limited resources typical for academic research, which implies that the task of deriving a service should require minimal effort.Footnote ³⁰ Furthermore, it should be regarded that some service developers might not have strong expertise in applied informatics but should still be able to utilize the proposed framework. An example to illustrate this demand could be a project with public funding dedicated to energy management in private households carried out by a consortium of research groups. In such a case, it may appear beneficial to integrate a research group dedicated to energy meteorology to develop a forecast service for PV power generation without demanding that this group cares about the operation and implementation details of the service. This leads to the following requirements:

A service operating for a longer time may need continuous development work by both service developer and provider, for example, in order to maintain or even improve performance and usability. Such efforts could include breaking changes, for example, an adaption of the data format which is not backward compatible. In order to give EMS developers time to adjust to those changes, it is common to operate an old and a new version in parallel. However, this implies that the EMS developer must be able to select which version of a service should be utilized, thus leading to the following requirement:

While deriving FR09 above, we have argued that documentation is important for the adoption of APIs by EMS developers. However, beyond the pure existence of documentation, it appears reasonable to demand that the latter should allow EMS developers to rapidly comprehend the API of a service. Furthermore, the main intention of an EMS developer reading the documentation is likely to implement a client in order to interact with the API of a service. For this, we demand minimal effort of implementation again, assuming it will likely support widespread adoption of the corresponding service. Hence, our final two requirements are:

We are convinced that the requirements derived in the present and previous sections are a solid foundation for deriving a framework for provisioning forecasting and optimization algorithms as web services for EMSs, and demonstrate this suitability below.

5.1. API design

As a first step, it is necessary to choose the paradigm on which the API of our proposed service framework should be based. We consider well-established approaches for web-based APIs (FR02). These are REST (Fielding, Reference Fielding2000), Remote Procedure Call (RPC) (in particular, gRPCFootnote ³¹) as well as GraphQL.Footnote ³² As all three candidates are generally suited to satisfy the functional requirements we focus on the non-functional requirements in order to select the best suited paradigm. The relevant requirements are understandability (NFR09) as well as ease of client implementation (NFR10). It is generally perceived that REST is the most favorable approach regarding these demands (Hunter, Reference Hunter2017; Jin et al., Reference Jin, Sahni and Shevat2018), which is therefore selected.

As a next step, we define the functionality of the API that is provided by the service framework. The selection of REST implies that all communication between client and service will use the Hypertext Transfer Protocol (HTTP) and that the functionality must be mapped to Uniform Resource Locators (URLs). It should be noted that we will only note down the relative part of URLs for the sake of brevity and to highlight that the domain is not relevant for the structure of the API, that is, we use /endpoint1/ instead of the full notation https://some-service.example.com/endpoint1/. The selection of REST furthermore implies that we need to define which HTTP method (e.g., GET, POST, PUT, or DELETE) must be used in order to receive a desired outcome while interacting with a specific URL. We will henceforth refer to the combination of HTTP method and (relative) URL as API method. Further introduction about web-based communication over HTTP can be found in the usual introductory texts or as a short summary in Jin et al. (Reference Jin, Sahni and Shevat2018).

FR03 dictates that EMSs must be able to retrieve a forecast or optimized schedule from a service. Furthermore, we need to consider that a service may take minutes or even hours to compute the result (FR08). As especially the latter is far beyond typical timeouts of HTTP serversFootnote ³³ it is infeasible to directly return the computation result. Instead, we define three API methods to overcome this issue:

POST /{version}/request/

GET /{version}/request/{task_ID}/status/

GET /{version}/request/{task_ID}/result/

The intended interaction of an EMS with these API methods is as follows:

1. 1. The EMS issues a POST call to the /{version}/request/ endpoint containing the required input data (see FR04). Note that {version} is a placeholder that must be filled with the desired version of the targeted service, which is required to satisfy NFR08, and could, for example, have a value of v2, see the example provided below. The service checks whether the input data is correct. If that is the case the service starts computing the result in the background and returns an ID, for example, 123, associated to this task (more details about the latter are provided in the following two sections).

2. 2. Using the ID of the request, the EMS should issue calls to the endpoint: GET /{version}/request/{task_ID}/status/.Footnote ³⁴ Note that {task_ID} is a placeholder too. Regarding the example above, the endpoint would be /v2/request/123/status/. For each call, the service will compute and return the status of the computation that is one of queued, running, or ready.Footnote ³⁵

3. 3. Once the service has finished processing the result and a ready status has been observed, the EMS can issue a GET call to the /{version}/request/{task_ID}/result/ endpoint to retrieve the output of the computation, that is, the forecast or optimized schedule.

Additionally, to the ones defined above, it is necessary to specify API methods to allow fitting system-specific parameters of a service in order to satisfy FR05. As potentially long processing times (FR08), as well as versioning (NFR08), need to be considered again, it appears reasonable to take over the concept introduced above and define the respective API methods as:

POST /{version}/fit-parameters/

GET /{version}/fit-parameters/{task_ID}/status/

GET /{version}/fit-parameters/{task_ID}/result/

The intended usage of the /fit-parameters/ endpoints is equivalent to the pattern discussed for /request/ above.

Finally, and in order to satisfy NFR03, we need to consider authorization, which implies authentication, to allow service providers to restrict access to specific clients. Following Kornienko et al. (Reference Kornienko, Mishina, Shcherbatykh and Melnikov2021), Späth (Reference Späth2023), and Saeed and Abdallah (Reference Saeed and Abdallah2022), the current best practice for REST APIs is token-basedFootnote ³⁶ authentication, in particular the utilization of JSON Web Tokens (JWTs). JWTs, as defined in Jones et al. (Reference Jones, Bradley and Sakimura2015), are special tokens that can be cryptographically validated. Regarding the scope of this work, JWTs are issued by a dedicated identity provider (see Section 5.4) to the client software. The signature of the token allows the service to check locallyFootnote ³⁷ whether a request of a client should be granted or not. Further details about the application of JWTs for web security are given in Saeed and Abdallah (Reference Saeed and Abdallah2022).

5.2. Service components

The concept described in this section arises from the requirements FR01 and NFR06, that is, that it should be possible to derive a functional service from an existing forecasting or optimization code with minimal effort. Especially the latter (NFR06) imposes that as much functionality as possible should be realized by the service framework in order to keep the implementation effort for the service developer low. To this end, we define a functional service to consist of several components that can be grouped into three categories as summarized in Figure 4:

1. 1. Base: Containing the components necessary for executing the code of the service.

2. 2. Service Framework: Containing all components generic to all services.

3. 3. Service Specific: Containing all components a service provider must implement to derive a functional service.

Figure 4. Components of a service derived from the service framework.

The primary component of the service framework is the API as defined above. Additionally, we need to consider that computing the result of a request (as well as fitting system-specific parameters) might take significant time (FR08), while the REST API of the service should respond immediately. Hence, it is necessary to decouple the API from the interaction with forecasting or optimization code. Therefore, we introduce the worker, which is a second component provided by the service framework that is executed in a dedicated process and that is responsible for computing the requested results. It is worth noting that this concurrent processing is crucially important, as, otherwise, executing a forecasting or optimization code might block the API from responding to other calls from clients.Footnote ³⁸

The service-specific category contains the actual payload of the service, that is, the forecasting or optimization code. Additionally, the requirements FR04 and FR06 need to be considered, that is, service developers must be able to specify the format of the input data for calls to /request/ and /fit-parameters/ as well as the output format returned by the corresponding /result/ API methods. We will refer to the part of the implementation that defines these formats as a data model. Footnote ³⁹ At this point, we will not further specify possible characteristics of the data model as the latter are tightly connected to the implementation of the API. However, we will proceed to discuss this topic in Section 6.2.

5.3. Service architecture

It was discussed in Section 5.2 that a functional service must contain an API as well as a worker component and that these should be operated in distinct processes for performance reasons. It should be noted at this point that the service-specific components, that is, the data model as well as the forecasting or optimization code, are perceived to be parts of the API and worker components. The service-specific components are consequently not explicitly mentioned in this section to promote readability.

Following from the execution of service components in distinct processes the necessity arises to establish some form of inter-process communication to allow services to operate. Considering the simplest case, that is, a service consisting of two processes, one for the API and one for the worker, communication between the two could be established quite simply using queues. However, we need to consider NFR01, that is, that service providers should be able to operate services with high availability and scalability. From the latter follows that services might consist of multiple instances of API and worker components, while high availability imposes that the service components might be distributed over several machines. The usual approach in such a scenario, which is utilized in this work, too, is to leverage a message broker for communication between components. The resulting communication pattern within a service would be as follows:

• • Every valid POST call to a /request/ or /fit-parameters/ endpoint should lead to the creation of a task, an object carrying the necessary information for computing the result, by the API. The latter should assign an ID to the task and publish the task on the message broker to invoke the workers. Finally, the API should return the task ID to the client.

• • A worker process should fetch the task from the message broker and start computing the result by invoking the forecasting or optimization code. Additionally, the worker should regularly publish status updates on the processing progress on the broker.

• • In case of a call to a /status/ endpoint, the API should fetch the latest status information about the corresponding task from the broker and return this information to the client.

• • If a /result/ endpoint is called, the API should fetch the result from the broker and return the result to the client.

Finally, it must be considered that a result that is fetched from the broker should not be deleted from the broker immediately. For example, consider that the communication between the client and API might be interrupted, in which case the client will likely retry to fetch the result. However, in order to prevent unlimitedly growing memory consumption of the message broker, it is required to operate an additional component, that is, the garbage collector. The duty of the latter is to delete the task-related data from the broker that are likely not to be required anymore.

The resulting internal architecture of a service, including communication channels, is indicated in Figure 5. It should be appreciated that the service architecture introduced above is strictly designed to support scalability and high availability, in particular, as subsequent calls from clients are not required to hit the same API process again. That is, it is not required that a call to the /status/ endpoint is handled by the same API process that created the corresponding task, as all task-related information is shared on the broker. Furthermore, it is possible to scale the number of API as well as worker processes to the actual load induced by clients.

Figure 5. Internal architecture of a service with processes of components and communication between processes.

5.4. Operation concept

Building up on our considerations introduced above, we will conclude the service design in this section by discussing the missing piece, which is the operation concept. To this end, it should be recalled first that (in this work) a service consists of several components (like API or worker) and that in reality operation might require that several instances of each component are executed in parallel, possibly distributed over several machines (see Section 5.3). We have referred to the latter as processes in order to distinguish the running instance from the implementation. The management of these processes, commonly referred to as orchestration, is a tedious activity best left to specialized and well-established applications. In consequence, the first step toward the operation concept is to select an appropriate orchestration software, where it needs to be considered that service providers have different demands here, ranging from cloud computing professionals of commercial companies with a primary interest in high availability and scalability (NFR01) to academic researchers with no experience with the utilization of orchestration software (NFR07). Systematic comparisons of orchestration applications (Jawarneh et al., Reference Jawarneh, Bellavista, Bosi, Foschini, Martuscelli, Montanari and Palopoli2019; Mercl and Pavlik, Reference Mercl, Pavlik, Yang, Sherratt, Dey and Joshi2019; Malviya and Dwivedi, Reference Malviya and Dwivedi2022) suggest KubernetesFootnote ⁴⁰ for the first group and Docker SwarmFootnote ⁴¹ for the second. Both orchestrators require that the processes of the service are wrapped in containers. As DockerFootnote ⁴² containers are supported by both they are selected.

Beyond the execution of the service components as described in the previous sections, provisioning of functional services to EMSs requires that a service provider operates additional supportive applications. The need for the first one of these supportive applications arises from the simple demand that the API containers must be accessible for the EMSs (FR04 and FR06), as well as that requests from clients should be distributed over the API containers of a service, in order to allow highly available and scalable operation (NFR01). We will refer to the application that fulfills this duty as a gateway. However, it should be noted that such applications are often alternatively named reverse proxy or ingress,Footnote ⁴³ the latter in particular in the context of Kubernetes. It is common, too, that the gateway encrypts the communication with the client, that is, by utilizing the Hypertext Transfer Protocol Secure (HTTPS), on behalf of the payload application, in our case, the service. This procedure removes the burden of integrating the security-relevant and vastly complex encryption logic into the code base of the API component while still satisfying the requirement that communication between client and service must be secure (NFR02). The second supportive application is necessary to issue JWT tokens to clients in order to satisfy NFR03 as discussed in Section 5.1. Such applications are usually referred to as Identity Provider (IdP) and the current best practice choice for issuing JWTs is the OpenID Connect (OIDC) protocol (Kornienko et al., Reference Kornienko, Mishina, Shcherbatykh and Melnikov2021; Scott and Neray, Reference Scott and Neray2021). Hence, our operation concept follows this best practice and intends that a well-established IdP software, for example, Keycloak,Footnote ⁴⁴ is used in order to administrate access control and to issue JWT tokens to the client software using OIDC. Finally, it is worth noting that it may not be necessary for every service provider to operate an IdP by themselves. Especially in the context of academic research, it is likely sufficient that one partner operates an IdP, for example, for a project, while services can still be distributed over several partners. This significantly reduces the effort for providing services for those partners that do not serve the IdP, as operating the latter is a complex task requiring specialized knowledge. This possibility should support especially academic researchers with backgrounds other than informatics.

6.1. Worker, garbage collector, and inter-process communication

In order to ensure that our implementation is robust and to minimize future maintenance efforts, we refrain from developing a custom approach for the inter-process communication between API and worker. Instead, we utilize Celery,Footnote ⁵¹ a well-established, stable, and open-source Python library for distributed task execution. It is worth noting that Celery is particularly well suited for use cases similar to ours, that is, to decouple client-facing web components from worker processes in a scalable fashion. Furthermore, Celery supports multiple message brokers, including the well-established RedisFootnote ⁵² and RabbitMQ,Footnote ⁵³ which gives service providers flexibility to choose a broker matching their demands, tech stack, and/or skill set of employees. We provide an implementation of a generic worker, which invokes the service-specific forecasting or optimization code in the ESG package. The worker makes use of Celery to interact with the message broker and implements the functionality described in Section 5.3. Finally, Celery provides a garbage collector functionality. Depending on the choice of message broker the latter may be available without the need for a dedicated process, more details are provided in the respective part of the documentation.Footnote ⁵⁴

6.2. API

The first step toward implementing the API is to select an appropriate framework to build upon. Considering the choice of Python as a programming language, it follows that two well-established web frameworks are available,Footnote ⁵⁵ these are FlaskFootnote ⁵⁶ and FastAPI.Footnote ⁵⁷ Several requirements need to be considered in order to select one of the two candidates objectively. In particular, these are FR09 (service developers should be able to automatically generate documentation), NFR09 (EMS developers should be able to quickly understand the API using the documentation) as well as NFR10 (EMS developers should be able to implement a client with little effort). We have selected FastAPI because it is generally considered to have a better integration of the OpenAPI specificationFootnote ⁵⁸ and tool ecosystem (Krebs and Cruz Martinez, Reference Krebs and Cruz Martinez2022; Singh, Reference Singh2023). OpenAPI schema is a gold standard for documenting REST APIs. FastAPI is capable of automatically generating and hosting OpenAPI schema documents as well as serving Swagger UI.Footnote ⁵⁹ The latter is an interactive API documentation building up on the former, thus satisfying FR09. Furthermore, interactive documentation is considered to be especially helpful for client developers to quickly understand an API (Hunter, Reference Hunter2017), which makes FastAPI a reasonable choice to fulfill NFR09. Another helpful tool from the OpenAPI ecosystem is Swagger CodegenFootnote ⁶⁰ which allows the automatic generation of large shares of client code for an impressive number of programming languages, thus satisfying NFR10.

Using FastAPI, our implementation of the API component is capable of serving the endpoints defined in Section 5.1. It is fully functional, apart from the definition of the data models, that is, the input data for POST /request/ and POST /fit-parameters/ as well as the corresponding GET /result/ API methods, which need to be defined by the service provider. It is worth noting that the definition of the data models is simple and fast to implement in order to fulfill NFR06. A practical example and additional details are provided in Section 8.1.2. Furthermore, our implementation of the API component puts the communication concept defined in Section 5.2 into action, that is, it transposes the HTTP calls of the clients into interactions with the message broker using the Celery framework introduced above, but only after the corresponding JWT of the call has been verified and checked. We utilize the PyJWTFootnote ⁶¹ package for the latter. Finally, it is worth mentioning that our implementation solely utilizes the JavaScript Object Notation (JSON) data format for all data exchange, as the latter is interpretable for humans and machines alike and is furthermore considered to be the best choice for REST APIs (Hunter, Reference Hunter2017).

6.3. Other functionality

Besides the implementation of the API and the process management system, the ESG package provides additional useful functionality for the implementation of and interaction with services. Particularly relevant is a generic client that can be used to trigger calls to services from Python source code. Furthermore, the package contains building blocks for data models, in order to reduce the effort for implementing these as well as to prevent code redundancy, see Section 8.1.2 for further details. Additionally, useful utility functions are available, for example, to parse pandasFootnote ⁶² “DataFrames” from JSON data.

8.1. Deriving services

As a first step toward proving the relevance of this work, we demonstrate how a simple (but fully functional) PV power generation forecast service can be implemented using the proposed framework. The following subsections present and discuss the implementation of the components of the service. It should be noted that the forecasting algorithm of the illustrated service, including the quality of the forecasts the service could compute, is not of particular interest. Instead, this section aims to demonstrate the value of the framework, which can be perceived by realizing how short the code listings below are. Besides the forecasting or optimization algorithm, which must be implemented anyway, deriving the functional example service requires <100 lines of code. On the other hand, at the time of writing, the ESG repository contained about 2400 lines of code, excluding examples and documentation. It is thus obvious that utilization of the ESG framework is much more work-efficient compared to the implementation of services from scratch, as the latter would require the developer to implement a larger fraction of the code provided by the framework. In fact, implementing services from scratch might be far more time-consuming than the proportion of lines of code suggests. This is, in particular, the case as the code using the framework is quite simple, as demonstrated below, while some parts provided by the framework are complex and need to be implemented carefully, for example, due to security concerns or impacts on runtime behavior.

It is worth noting that the code listings provided below are part of the ESG repository,Footnote ⁶⁵ whereby the online version might be adapted to any potential future changes in the ESG framework. We thus suggest all those wishing to reproduce this example, that is, run the example service, to use the files provided in the repository.

8.2. Client implementation

Above, see NFR10, we have argued that a key requirement for the widespread integration of services into EMSs is the ease of client implementation. In order to demonstrate the latter we provide an example for a minimal client, implemented as a shell script, capable of retrieving PV power generation forecasts from the service introduced in the previous section in Listing 7. It should be noted that the example assumes that the service is reachable via the localhost network address, that is, that the client is executed on the same machine as the service. Note further that the latest version of the code listed below, that is, the version adapted to potential future changes of the framework, is provided as part of the ESG repository.Footnote ⁷⁰

One can perceive from inspecting the code above that the client logic is indeed very simple, and thus easy to implement. The script is written in standard Unix Shell syntax and uses only two additional packages, curl for making HTTP requests and jq for parsing JSON. The script follows the API concept derived in Section 5.1. That is, a request is created with the first curl call. The large nested structure underneath defines the input arguments and parameters that are provided to the service. More discussion about the latter, including an explanation of the fields, is provided in Section 8.1.2. Next, the script extracts the task ID of the created request and polls the /status/ endpoint until the task has reached the ready status and finally fetches the result. Note that the interaction with the /fit-parameters/ endpoint of the service follows the same pattern and is thus omitted here for brevity.

While the service integration used in a production EMS would likely require more functionality, for example, to parse the result into the format the EMS expects or to handle network errors, the concept of the client always remains the same. In fact, the ESG package contains a ready-to-use generic clientFootnote ⁷¹ which provides this additional functionality and reduces the implementation effort further.

The code in Listing 8 demonstrates that the generic client class, which is provided as part of the ESG package, reduces the effort to the implementation of the data models as well as configuring the endpoint of the service the client should use. It is worth noting that specifying the data models has been left on purpose for the developer of the client, although it is technically possible that the client fetches the latter automatically from the service. However, the idea is that the implementation of the data model on the client side documents the data structure that the downstream application, that is, the EMS, is designed for. That is, it allows the client application to detect any changes in the format of the data provided by the service. While such changes should not happen in theory, see discussion in NFR08 and about versioning in Section 5.1, they might still occur due to mistakes made by developers of services. Here, it is likely much simpler to debug an error that is thrown directly in the client code than some error deep downstream in the application using the data, which might have no obvious direct connection to the service and its data format.

Finally, we would like to point out that the generic client provided by the ESG package can only be used in applications capable of executing Python programs. Any developer working on an application not capable of the latter will likely need to implement the client logic from scratch. However, the effort for such an activity should be rather limited as the generic client has been implemented in <400 lines of code. As an alternative, the developer could resort to partly automatically generating the client program using Swagger Codegen, see Section 6.2 for further details.

8.3. Scalability of services

One of the core claims of this work is that the presented framework enables the operation of forecasting and optimization services for potentially thousands of EMSs. In order to demonstrate that this claim is indeed legitimate, we present an experiment that assesses the scalability in the present section.

8.3.2. Experiment execution and results

In order to execute the experiment, the scalability tester service was deployed to the single-node Kubernetes instance introduced above. Utilizing the latter, the API and worker instances were scaled to a selected number, henceforth referred to as replication factor, that is, it was taken care that this many instances of each the worker and API container were executed. The API and worker containers were using a single-node RedisFootnote ⁷⁶ instance as a message broker, a popular choice given the implementation of the ESG package.Footnote ⁷⁷ The actual experiment was conducted by executing the code invoking the clients. The latter was run on the personal laptop of one of the authors that was connected over the public internet to the service for extended realism.

The result, as shown in Figure 6, is the average elapsed time starting directly before the program issued the first request and ending after all results have been available and transferred back. The measurements were repeated three times and the difference between the minimum and the maximum of the measured times was for all replication factors 1 s or less, the results appear thus robust and reliable. It can be seen that the time required for processing 10.000 requests shrinks from initially ca. 120 to ~35 s as the replication factor is increased from 1 over 2, 4, 8, and finally 16. It is thus clearly possible to mitigate a growing number of requests by replicating worker and API instances as the total number of requests per time unit is reciprocal to the processing time required for handling a fixed number of requests.

Figure 6. Measured and ideal time for processing 10.000 requests over replication factor.

An interesting observation from the inspection of the measured results is that increasing the replication factor from 8 to 16 yields only a rather small gain of one second in real processing time, while the ideal processing times suggest a significant reduction from 20 to 10 s. Analysis of the CPU and memory consumption of the machine on which the experiment was executed on yielded no evidence that exhausted hardware could have been the reason. However, it appears likely that this behavior has been caused by the high number of threads (16.000 with the largest replication factor!), which very likely imposes a significant overhead for the operating system to switch tasks between these. In the latter case distributing the worker instances over more than one machine should improve the performance even further. While, at the time of writing, we do not possess the technical resources to validate this claim experimentally, we plan to catch up on this matter once we have access to a more sophisticated and better-equipped Kubernetes cluster.

Finally, it is worth mentioning that the processing time of 35 s for 10.000 requests is roughly equivalent to 250.000 EMSs issuing one request per 15 min, which appears a solid foundation for forecasting and optimization services at scale. However, we estimate that using more sophisticated computing resources, it should rather easily be possible to serve several millions of EMSs.

8.4. Comparison of requirements with concept and implementation

As the last part of our evaluation, this section presents a systematic comparison of the requirements derived in this work with the realization of our proposed framework and community concept. This comparison is provided in Table 2 for the functional and non-functional requirements defined in Sections 4.2 and 4.3. One can easily perceive that our contributions, that is, design and implementation of the framework as well as the community concept, do actually satisfy all requirements by inspecting the provided table.

Table 2. Comparison of functional and non-functional requirements with actual realization in service framework and community concept

As the requirements have been carefully derived from the current state of the art of energy management applications, it is concluded that our proposed framework and community concept are indeed valuable tools for the implementation and widespread distribution of forecasting and optimization services.