3.1 Knowledge representation and reasoning

In KAT, the transition diagrams of any domains are described in an extension of the action language $\mathcal{AL}_d$ (Gelfond and Inclezan Reference Gelfond and Inclezan2013). Action languages are formal models of parts of natural language that are used to describe the transition diagrams of any given dynamic domain. The domain representation comprises a system description $\mathcal{D}$ , a collection of statements of $\mathcal{AL}_d$ , and a history $\mathcal{H}$ . $\mathcal{D}$ has a sorted signature $\Sigma$ which consists of actions, statics, that is, domain attributes whose values cannot be changed, and fluents, that is, domain attributes whose values can be changed by actions. For example, $\Sigma$ in the HFO domain includes basic sorts such as $ad\_hoc\_agent$ , $external\_agent$ , agent, $of\!\!fense\_agent$ , $de\!f\!ense\_agent$ , $x\_val$ , $y\_val$ , and sort step for temporal reasoning. Sorts are organized hierarchically, with some sorts, for example, $of\!\!fense\_agent$ and $de\!f\!ense\_agent$ , being subsorts of others, for example, $external\_agent$ . Statics in $\Sigma$ are relations such as $next\_to(x\_val,$ $ y\_val, x\_val, y\_val)$ that encode the relative arrangement of locations (in the HFO domain). The fluents in $\Sigma$ include inertial fluents that obey inertia laws and can be changed by actions, and defined fluents that do not obey inertia laws and are not changed directly by actions. For example, inertial fluents in the HFO domain include

(1) \begin{align} &loc(ad\_hoc\_agent, x\_val, y\_val)\\\nonumber &ball\_loc(x\_val, y\_val)\\\nonumber &has\_ball(agent),\end{align}

which describe the location of the ad hoc agent, location of the ball, and the agent that has control of the ball; the value of these attributes changes as a direct consequence of executing specific actions. Defined fluents of the HFO domain include

(2) \begin{align} &agent\_loc(external\_agent, x\_val, y\_val) \\ \nonumber &de\!f\!ense\_close(agent, defense\_agent) \\\nonumber &far\_from\_goal(ad\_hoc\_agent),\end{align}

which encode the location of the external (i.e., non-ad hoc) agents, and describe whether a defense agent is too close to another agent, and whether the ad hoc agent is far from the goal. Note that the ad hoc agent has no control over the value of these fluents, although the hoc agent’s actions can influence the value of these fluents. Next, actions in the HFO domain include

(3) \begin{align} &move(ad\_hoc\_agent, x\_val, y\_val)\\\nonumber&kick\_goal(ad\_hoc\_agent)\\ \nonumber&dribble(ad\_hoc\_agent, x\_val, y\_val)\\\nonumber&pass(ad\_hoc\_agent, of\!fense\_agent),\end{align}

which state the ad hoc agent’s ability to move to a location, kick the ball toward the goal, dribble the ball to a location, and pass the ball to a teammate. Next, axioms in $\mathcal{D}$ describe domain dynamics using elements in $\Sigma$ in three types of statements: causal laws, state constraints, and executability conditions. For the HFO domain, this includes statements such as:

(4a) \begin{align} move(R,X,Y) ~\mathbf{ causes }~ loc(R,X,Y),\end{align}

(4b) \begin{align} &dribble(R,X,Y) ~\mathbf{ causes }~ ball\_loc(X,Y), \end{align}

(4c) \begin{align} &\neg has\_ball(A1) ~\mathbf{ if }~ has\_ball(A2), ~A1 \neq A2, \end{align}

(4d) \begin{align} &\mathbf{impossible }~ shoot(R) ~\mathbf{ if }~ far\_from\_goal(R).\end{align}

where Statements 4(a-b) are causal laws that specify that moving and dribbling change the ad hoc agent’s and ball’s location (respectively) to the desired location. Statement 4(c) is a state constraint that implies that only one agent can control the ball at any time. Statement 4(d) is an executability condition that prevents the consideration of a shooting action (during planning) if the ad hoc agent is far from the goal. Finally, the history $\mathcal{H}$ is a record of observations of fluents at particular time steps, that is, obs(fluent, boolean, step), and of action execution at particular time steps, that is, hpd(action, step). It also includes initial state defaults, that is, statements in the initial state that are believed to be true in all but a few exceptional circumstances, for example, the following $\mathcal{AL}_d$ statement implies that attackers in the FA domain usually spread and attack the fort:

(5) \begin{align} \mathbf{initial~~default} ~spread\_attack(X)~ \textbf{if}~~ attacker(X).\end{align}

To enable an ad hoc agent to reason with prior knowledge, the domain description in $\mathcal{AL}_d$ is automatically translated to program $\Pi(\mathcal{D}, \mathcal{H})$ in CR-Prolog (Balduccini and Gelfond Reference Balduccini and Gelfond2003), an extension to ASP that supports consistency restoring (CR) rules. ASP is based on stable model semantics and represents constructs difficult to express in classical logic formalisms. It encodes concepts such as default negation and epistemic disjunction, and supports non-monotonic reasoning; this ability to revise previously held conclusions is essential in AHT. $\Pi(\mathcal{D}, \mathcal{H})$ incorporates the relation holds(fluent, step) to state that a particular fluent is true at a given step, and occurs(action, step) to state that a particular action occurs in a plan at a given step. It includes the signature and axioms of $\mathcal{D}$ , inertia axioms, reality checks, closed world assumptions for defined fluents and actions, observations, actions, and defaults from $\mathcal{H}$ , and a CR rule for every default allowing the agent to assume that the default’s conclusion is false in order to restore consistency under exceptional circumstances. For example, the CR-Prolog statement:

(6) \begin{align} \lnot spread\_attack(X) \stackrel{\,\,+}{\leftarrow} attacker(X),\end{align}

allows the ad hoc agent to consider the rare situation of attackers mounting a frontal attack. Furthermore, it includes helper axioms, for example, to define goals and drive diagnosis. Reasoning tasks such as planning, diagnosis, and inference are then reduced to computing answer sets of $\Pi$ . The ad hoc agent may need to prioritize different goals at different times, for example, score a goal when it has control of the ball, and position itself at a suitable location otherwise:

(7) \begin{align} &goal(I) \leftarrow holds(scored\_goal,I) \\\nonumber &goal(I) \leftarrow holds(loc(ad\_hoc\_agent, X, Y), I).\end{align}

A suitable goal is selected and included in $\Pi(\mathcal{D}, \mathcal{H})$ automatically at run-time. In addition, heuristics are encoded to direct the search for plans, for example, the following statements:

(8) \begin{align} &total(S) \leftarrow S = sum\{C,A:occurs(A,I), cost(A,C)\}\\\nonumber &\#minimize \{S@p, S:total(S)\}.\end{align}

encourage the ad hoc agent to select actions that will minimize the total cost when computing action sequences to achieve a particular goal. We use the SPARC system (Balai et al. Reference Balai, Gelfond and Zhang2013) to write and solve CR-Prolog programs. For computational efficiency, these examples programs build on a refinement-based architecture that represents and reasons with knowledge at two tightly coupled resolutions, with a fine-resolution description ( $\mathcal{D}_F$ ) defined as a refinement of a coarse-resolution description ( $\mathcal{D}_C$ ). For example, the available space in the FA domain and HFO domain is organized into abstract regions in $\mathcal{D}_C$ , with each region being refined in $\mathcal{D}_F$ into small grids that are components of this region. $\mathcal{D}_C$ and $\mathcal{D}_F$ are formally coupled through component relations and bridge axioms such as:

(9) \begin{align} &loc^*(A, Rg) ~~\mathbf{if}~~ loc(R, X, Y), ~component(X, Y, Rg)\nonumber\\ &next\_to^*(Rg_2, Rg_1) ~~\mathbf{if}~~ next\_to^*(Rg_1, Rg_2),\end{align}

where location (X, Y) is in region Rg and superscript “*” refers to relations in $\mathcal{D}_C$ . This coupling between the descriptions enables the ad hoc agent to automatically choose the relevant part of the relevant description based on the goal or abstract action, and to transfer relevant information between the descriptions. Example programs are in our repository (Dodampegama and Sridharan Reference Dodampegama and Sridharan2023b); details of the refinement-based architecture are in the paper by Sridharan et al. (Reference Sridharan, Gelfond, Zhang and Wyatt2019).

3.2 Agent models and model selection

Since reasoning with just prior domain knowledge can lead to poor team performance under AHT settings (see Section 4.2), KAT enables the ad hoc agent to also reason with models that predict (i.e., anticipate) the action choices of other agents. State of the methods attempt to optimize performance in different (known or potential) situations by learning models offline from many (e.g., hundred thousands or millions of) examples. It is intractable to obtain such labeled examples of different situations in complex domains, and the learned models are truly useful only if they can be learned and revised rapidly during run-time to account for previously unknown situations. KAT thus chooses relevant attributes for models that can be: (a) learned from limited (e.g., 10,000) training examples acquired from simple hand-crafted policies (e.g., spread and shoot in FA, pass when possible in HFO); and (b) revised rapidly during run-time to provide reasonable accuracy. Tables 1 and 2 list the identified attributes in the FA and HFO domain respectively.

Table 1. Attributes considered for models of other agents’ behavior in FA domain. Number of attributes represent the number of variables in each attribute times the number of agents

Table 2. Attributes for models of teammates and defense agents’ behavior in HFO domain. Number of attributes represent the number of variables in each attribute times the number of agents

Similar to our recent work (Dodampegama and Sridharan Reference Dodampegama and Sridharan2023a), the attributes are identified and the predictive models are learned using the Ecological Rationality (ER) approach, which draws on insights from human cognition, Herb Simon’s definition of Bounded Rationality, and an algorithmic model of heuristics (Gigerenzer Reference Gigerenzer2020; Gigerenzer and Gaissmaier Reference Gigerenzer and Gaissmaier2011). ER focuses on decision making under true uncertainty (e.g., in open worlds), characterizes behavior as a joint function of internal (cognitive) processes and the environment, and focuses on satisficing based on differences between observed and predicted behavior. Also, heuristic methods are viewed as a strategy to ignore part of the information in order to make decisions more quickly, frugally, and/or accurately than complex methods. In addition, it advocates the use of an adaptive toolbox of classes of heuristics (e.g., one-reason, sequential search, lexicographic), and comparative out-of-sample testing to identify heuristics that best leverage the target domain’s structure. This approach has provided good performance in many applications (Gigerenzer Reference Gigerenzer2016).

Specifically, in KAT, ER principles such as abstraction and refinement, and statistical attribute selection methods, are applied to the set of 10,000 samples to identify the key attributes and their representation in Tables 1 and 2; these define behavior in the FA domain and HFO domain respectively. The coarse- and fine-resolution representation described in Section 3.1 is an example of the principle of refinement. In addition to the choice of features, the characteristic factors of AHT, for example, the need to make rapid decisions under resource constraints and respond to dynamic changes with limited examples, are matched with the toolbox of heuristics to identify and use an ensemble of “fast and frugal” (FF) decision trees to learn the behavior prediction models for each type of agent. Each FF tree in an ensemble focuses on one valid action, provides a binary class label, and has the number of leaves limited by the number of attributes (Katsikopoulos et al. Reference Katsikopoulos, Simsek, Buckmann and Gigerenzer2021). Figure 3 shows an example of a FF tree learned (as part of the corresponding ensemble) for a guard agent (Figure 3a) and an attacker agent (Figure 3b) in the FA domain.

Fig. 3. Examples of FF trees for a guard and an attacker in the FA domain.

The ad hoc agent’s teammates and opponents may include different types of agents whose behavior may change over time. Unlike our prior work that used static prediction models, we enabled the ad hoc agent to respond to such changes by automatically revising the current model, switching to a relevant model, or learning new models. Existing models are revised by changing the parameters of the FF trees, and Algorithm 1 is an example of our approach for selecting a suitable model in the context of predicting the pose (i.e., position and orientation) of agents. Specifically, the ad hoc agent periodically compares the existing models’ predictions with the observed action choices of each agent (teammate, opponent) over a sliding window of domain state and the agents’ action choices; in Algorithm 1, this window is of size 1 (Lines 4–5). Also, a graded strategy is used to compute the error, penalizing differences in orientation less than differences in position (Lines 6–7). The model whose predictions best match the observations is selected for subsequent use and revision (Line 10, Algorithm 1). Note that if none of the models provide a good match over multiple steps, this acts as a trigger to learn a new model.

3.3 Partial observability and communication

In practical AHT domains, any single agent cannot observe the entire domain and communication is a scarce resource. To explore the interplay between partial observability and communication, we modified the original domains. Specifically, in the FA domain, we introduced a forest region where attackers can hide from the view of the two guards other than ad hoc agent and secretly approach the fort—see Figure 1b. The ad hoc agent has visibility of the forest region; it can decide when to communicate with its teammates, for example, when: (a) one or more attackers are hidden in the forest; and (b) one of the other guards is closer to the hidden attacker(s) than it. The associated reasoning can be encoded using statements such as:

(10a) \begin{align} holds(shoots(G, AA),I+1) \leftarrow &occurs(communicate(AHA, G, AA),I), \end{align}

(10b) \begin{align} &holds(in\_forest(AA),I) \leftarrow holds(agent\_loc(AA,X,Y),I),~forest(X,Y),\nonumber\\ &not~~holds(shot(AA),I), \end{align}

(10c) \begin{align} -occurs(communicate(AHA&, G, AA),I) \leftarrow ~not~holds(in\_range(G, AA),I). \end{align}

where Statement 10(c) encodes that communication is used only when a hidden attacker is within the range of a teammate (i.e., guard agent); Statement 10(b) defines when an attacker is hidden; and Statement 10(a) describes the ad hoc agent’s belief that a teammate receiving information about a hidden attacker will shoot it, although the teammate acts independently and may choose to ignore this information. If there are multiple guards satisfying these conditions, the ad hoc agent may only communicate with the guard closest to the hidden attacker(s).

Algorithm 1: Model Selection

In the HFO domain, we represent partial observability in an indirect manner using the built-in ability to limit each agent’s perception to a specific viewing cone relative to the agent. Specifically, each agent is only able to sense objects (e.g., other agents, ball) within its viewing cone; objects outside its viewing cone are not visible. Given this use of built-in functions, we added some helper axioms to ensure that the ad hoc agent only reasoned with visible objects; no additional communication action was implemented.

4.1 Experimental setup

In the FA domain, we used two kinds of policies for the agents other than our ad hoc agent: hand-crafted policies and built-in policies. Hand-crafted policies were constructed as simple strategies that produce basic behavior. Built-in policies were provided with the domain; they are based on graph neural networks trained using many labeled examples.

Hand-Crafted Policies.

• Policy1: guards stay near the fort and shoot attackers who spread and approach.

• Policy2: guards and attackers spread and shoot their opponents.

Built-in Policies.

• Policy220: guards stay in front of the fort and shoot continuously as attackers approach.

• Policy650: guards try to block the fort; attackers try to sneak in from all sides.

• Policy1240: guards spread and shoot the attackers; attackers sneak in from all sides.

• Policy1600: guards are willing to move away from the fort; some attackers approach the fort and shoot to distract the guards while others try to sneak in.

The ad hoc agent was evaluated in two experiments: Exp1, in which other agents followed the hand-crafted policies; and Exp2, in which other agents followed the built-in policies. As stated earlier, the ad hoc agent learned behavior models in the form of FF trees from 10,000 state-action observations obtained by running the hand-crafted policies. It was not provided any prior experience or models of the built-in policies.

Our previous work documented the accuracy of a basic AHT architecture that reasoned with some domain knowledge and static behavior prediction models in the FA domain (Dodampegama and Sridharan Reference Dodampegama and Sridharan2023a). In this paper, the focus is on evaluating the ability to select, revise, and learn the relevant predictive models, and adapt to partial observability. For the former, each agent other than our ad hoc agent was assigned a policy selected randomly from the available policies (described above). The baselines for this experiment were:

• Base1: other agents followed a random mix of hand-crafted policies. The ad hoc agent did not revise the learned behavior models or use the model selection algorithm.

• Base2: other agents followed a random mix of hand-crafted policies. The ad hoc agent used a model selection algorithm without a graded strategy to compare the predicted and actual actions, that is, fixed penalty assigned for action mismatch in Line 6 of Algorithm 1.

• Base3: other agents followed a random mix of built-in policies. The ad hoc agent did not revise the learned behavior models or use the model selection algorithm.

• Base4: other agents followed a random mix of built-in policies. The ad hoc agent used the model selection algorithm without a graded strategy to compare predicted and actual actions, that is, fixed penalty assigned for action mismatch in Line 6 of Algorithm 1.

The baselines for evaluating partial observability and communication were as follows:

• Base5: in Exp1, other agents followed hand-crafted policies and ad hoc agent did not use any communication actions.

• Base6: in Exp2, other agents followed built-in policies and the ad hoc agent did not use any communication actions.

Recall that KAT allows the use of communication actions (when needed) under conditions or partial observability. Also, each experiment described above (in FA domain) involved 150 episodes and results were tested for statistical significance.

In the HFO domain, we used six external agent teams from the 2013 RoboCup simulation competition to create the ad hoc agent’s teammates and opponents. Five teams were used to create offense agents: helios, gliders, cyrus, axiom and aut; agents of the defense team were based on the agent2d team. Similar to the initial phase in the FA domain, we deployed the existing agent teams in the HFO domain and collected observations of states before and after each transition in the episode. Since the actions of other agents are not directly observable, they were computed from the observed state transitions. To evaluate the ability to learn from limited data, we only used data from 300 episodes for each type of agent to create the tree-based models for behavior prediction, which were then revised (as needed) and used by the ad hoc agent during reasoning.

We first compared KAT’s performance with a baseline that only used non-monotonic logical reasoning with prior knowledge but without any behavior prediction models (Exp3), that is, the ad hoc agent was unable to anticipate the actions of other agents. Next, we evaluated KAT’s performance with each built-in external team, that is, all offense agents other than the ad hoc agent were based on one randomly selected external team in each episode. In Exp4, we measured performance in the limited version, that is, two offense players (including ad hoc agent) against two defense agents (including goalkeeper). In Exp5, we measured performance in the full version, that is, four offense players (including ad hoc agent) played against five defense agents (including goalkeeper). In Exp6 and Exp7, we evaluated performance under partial observability in the limited and full versions respectively. As the baselines for Exp4-Exp5, we used recent (state of the art) AHT methods: PPAS (Santos et al. Reference Santos, Ribeiro, Sardinha, Melo, Marreiros, Melo, Lau, Lopes Cardoso and Reis2021), and PLASTIC (Barrett et al. Reference Barrett, Rosenfeld, Kraus and Stone2017). These methods considered the same external agent teams mentioned above, allowing us to compare our results with the results reported in their papers. For Exp6-Exp7, we used the external agent teams as baselines. We conducted 1000 episodes for each experiment described above, and tested the results for statistical significance.

4.2 Experiment results

We begin with the results of experiments in the FA domain. First, Table 3 summarizes the results of using our model selection algorithm in Exp1. When the other agents followed the hand-crafted policies and the model selection mechanism was not used by the ad hoc agent (Base1), the team of guards had the lowest winning percentage. When the ad hoc agent used the model selection algorithm with a fixed penalty assigned for any mismatch between predicted and actual actions (Base2), the performance of the team of guards improved. When the ad hoc agent used KAT’s model selection method (Algorithm 1), the winning percentage of the team of guards was substantially higher than the other two options. These results demonstrated that KAT’s adaptive selection of the behavior prediction models improved performance.

Table 3. Wins (%) for guards with hand-crafted policies in FA domain (Exp1). Model adaptation improves performance

Next, the results of Exp2 are summarized in Table 4. We observed that KAT enabled the ad hoc agent to adapt to previously unseen teammates and opponents that used the FA domain’s built-in policies, based on the model selection algorithm and the online revision of the behavior models learned from the hand-crafted policies. KAT provided the best performance compared with not using any model adaptation or selection (Base3), and when model selection assigned a fixed penalty for action mismatch (Base4). These results and Table 3 support H1 and H2.

Table 4. Wins (%) for guards with built-in policies in FA domain (Exp2). Model adaptation improves performance

The results from Exp1 under partial observability, with and without communication (Base5), are summarized in Table 5. Recall that the other agents used the FA domain’s hand-crafted policies in this experiment. When the communication actions were enabled for the ad hoc (guard) agent, the winning percentage of the team of guards was substantially higher than the winning percentage of the team of guards when they could not use the communication actions. Policy2 was a particularly challenging scenario (because both guards and attackers can shoot), which justified the lower (overall) winning percentage.

Table 5. Wins (%) for guards with hand-crafted policies in FA domain (Exp1). Communication addresses partial observability

Next, the results from Exp2 under partial observability, with and without communication (Base6), are summarized in Table 6. Recall that the other agents used the FA domain’s built-in policies. We observed that when the guards (other than the ad hoc agent) followed policies 650, 1240, or 1600, the winning percentage of the team of guards was comparable or higher when communication actions were enabled compared with when these actions were not available (Base6). With Policy 220, the performance of the team of guards was slightly worse when the communication actions were enabled. However, unlike the other policies, Policy 220 results in the guards spreading themselves in-front of the fort and shooting continuously. Under these circumstances, partial observability and communication strategies were not important factors in determining the outcome of the corresponding episodes. These results support hypothesis H3.

Table 6. Wins (%) for guards with built-in policies in FA domain (Exp2). Communication addresses partial observability

We next describe the results from the HFO domain. Table 7 summarizes results of Exp3, which compared KAT’s performance with a baseline that had the ad hoc agent only reasoning with prior knowledge, that is, without any learned models predicting the behavior of other agents. With KAT, the fraction of goals scored by the offense team was significantly higher than with the baseline. These results emphasized the importance of learning and using the behavior prediction models, and indicated that leveraging the interplay between representation, reasoning, and learning leads to improved performance, which supports hypothesis H1.

Table 7. Fraction of goals scored (i.e., games won) by the offense team in HFO domain with and without the learned behavior prediction models (Exp3). Reasoning with prior domain knowledge but without the behavior prediction models has a negative impact on performance

Next, the prediction accuracy of the learned behavior models created for the limited version (Exp4) and full version (Exp5) of the HFO domain are summarized in Tables 8 and 9, respectively. Recall that these behavior models were learned for the agents other than the ad hoc agent using data from 300 episodes (for each external agent type). This translated to orders of magnitude fewer training samples than the few hundred thousand training samples used by state of the art data-driven methods that do not reason with domain knowledge. The prediction accuracy varied over a range for the different agent types. Although the accuracy values were not very high, the models could be learned and revised quickly during run time; also, these models resulted in good performance when the ad hoc agent also reasoned with prior knowledge.

Table 8. Prediction accuracy of the learned agent behavior models in limited (2v2) version of the HFO domain (Exp4)

Table 9. Prediction accuracy of the learned agent behavior models in full (4v5) version of the HFO domain (Exp5)

The results of Exp4 and Exp5 comparing KAT’s performance with the state of the art baselines for the HFO domain (PPAS, PLASTIC) are summarized in Table 10. Recall that these data-driven baselines required orders of magnitude more training examples and did not support reasoning with prior domain knowledge. The fraction of goals scored (i.e., games won) by the team of offense agents including our ad hoc agent was comparable with the goals scored by the baselines for the limited version, and substantially better than goals scored by the baselines for the full version. These results strongly support hypotheses H1 and H2.

Table 10. Fraction of goals scored (i.e., games won) by the offense team in HFO domain in the limited version (2v2, Exp4) and full version (4v5, Exp5). KAT’s performance comparable with the baselines in the limited version and much better than the baselines in the full version

The results of evaluating KAT under partial observability (in HFO domain) are summarized in Table 11 compared with teams of external agent types without any ad hoc agent. Although the results indicate that KAT’s performance was slightly lower than the baseline teams without any ad hoc agents, the difference was not significant and mainly due to noise (e.g., in the perceived angle to the goal during certain episodes). The ability to provide performance comparable with teams whose training datasets were orders of magnitude larger strongly supports hypothesis H3.

Table 11. Goals scored (i.e., games won) by offense team in HFO domain under partial observability (Exp6, Exp7). KAT’s performance comparable with baseline that had no ad hoc agents in the team but used training datasets that were orders of magnitude larger

In addition to the experimental results documented above, videos of experimental trials, including trials involving unexpected changes in the number and type of other agents, are provided in support of the hypotheses in our open-source repository (Dodampegama and Sridharan Reference Dodampegama and Sridharan2023b).