2. Scope of evaluation criteria

This paper is an example of the assessment of the performance of MA as it applies to a subset of the International Maritime Organization (IMO) Convention on the International Regulations for Preventing Collisions at Sea (COLREGs) (U.S. Coast Guard, 1999) between two vessels approaching one another on the open ocean.

COLREGs provide specific guidance, ‘Rules of the Road’, regarding two vessels interacting within sight of one another, and there exists a Risk of Collision, as defined in Rule 7. Rule 8 (Action to Avoid Collison) is a general rule that provides guidance on qualities of good seamanship. Specific rules that cover One-on-One vessel interactions are Rule 13 (Overtaking), Rule 14 (Head-on), Rule 15 (Crossing), Rule 16 (Action by Give-way Vessel), Rule 17 (Action by Stand-on Vessel) and Rule 18 (Responsibility Between Vessels).

Other non-vessel hazards such as non-moving, both charted and uncharted, are not considered. Non-vessels that may be drifting are considered, since rudimentary perception systems classify such objects that have both a speed over ground and a direction as an actual vessel, albeit a relatively slow one in comparison to an underway vessel.Footnote ¹

This paper only applies to a subset of manoeuvre operations that ocean-going vessels might be engaged in with emphasis on safety, primarily collision avoidance and hazard avoidance, and efficiency/performance of manoeuvres initiated for safety reasons. The focus is on the manoeuvring rules contained within the IMO COLREGs Rules 7, 8 and 13–18, with certain limitations.

Two aspects of manoeuvring rules, which are intertwined with these COLREGs rules, are not directly scored using the methodology of this paper: Safe Speed and In-ExtremisFootnote ² situations. Safe speed is assumed to be continuously evaluated, and the speed of the vessel when the encounter begins is assumed to be considered safe. In Extremis situations, because MA is designed to maintain a minimum Closest Point of Approach (CPA) outside close quarters, In-Extremis manoeuvring is assumed to be handled and evaluated separately.

Additionally, this methodology does not address COLREGs Rules 19–37, the capabilities and implications of interpreting nautical lights, sound signals, day shapes and Bridge-to-Bridge communications with other vessels. Lastly, the evaluation criteria covered here are not meant to cover the implications of Rule 2, Special Circumstances (Multiple Vessel Encounters), which gives seamen leeway to use good judgment in handling special circumstances rather than forcing strict adherence to the COLREGs rules:

The target system boundary for assessments is drawn around the software component(s) responsible for hazard and collision avoidance, assuming that is sufficiently separable from components requesting desired manoeuvres without consideration of hazards and contacts. This boundary can be readily established in simulation-based testing. During at-sea testing, it may be difficult to isolate the target system from supporting systems. However, contributions to performance by supporting systems will be consistent and the root cause of poor performance may be determined through analysis. For example, if a perception system has a tracking latency, this is isolated through analysis, and these latency effects can be accounted for when assessing MA performance.

3. Nautical rules covered and explained

3.1 Covering just the manoeuvring rules

Since the general use of terminology tends to be non-standard in this subject area, this section defines the meaning and usage of key terms within this document.

3.1.1. Route or transit behaviour

Route or transit behaviour in MA is used to navigate a vessel from one geographic location to another using waypoints. When navigating for long durations, waypoints are planned in sequence along a route. The vessel will automatically transit between sequential waypoints, achieving each, until reaching the last planned waypoint, as illustrated in Figure 1. Upon achieving the last waypoint, the vessel will automatically transition to the next planned behaviour. In the case when no task is planned following the last waypoint, the vessel would execute a default behaviour.

Figure 1. General route illustration.

3.1.2. One-on-one vessel COLREGs interactions

COLGREGs Rules 7, 8 and 13–18 define the ‘Rules of the Road’ for One-on-One vessel interactions. These rules are then the basis for which vessel MA must be able to follow in practice and spirit for safe operations.

Ranges, relative positions, orientation, and time and distance to the CPA of the contact vessel are extremely important to the mariner for proper manoeuvres to occur in an expected seaman-like manner. As such, these considerations are assumed to be ensconced in the MA design and software code. Relative positions and orientation are specifically accounted for in the COLREGs rules. However, apart from ‘in sight of one another’ and ‘so close that collision cannot be avoided’, COLREGs do not generally define ranges in which manoeuvring should occur or ranges for avoidance. Therefore, it is incumbent on the mariner to determine these ranges. Every situation is different and relies on a mariner’s experience for the determination of ranges. Ship captains provide standing orders to their manoeuvring officers regarding the ranges expected to be observed, and these ranges can change. Likewise, MA developers are in need of similar range parameters, which are outlined below (see Figure 2). The defined ranges used by the MA are not necessarily the same ranges the other vessel is using; in other words, one mariner’s estimation of what is close may not be the same as other mariners.

Figure 2. Range definition.

Reference distances:

• Observation Area: The area within the vessel’s sensor Line of Sight (LOS) or Evaluation Range.

• Minimum CPA: Minimum range that should be achieved during an encounter. Vessel will manoeuvre to keep contacts outside of this minimum distance. Minimum CPA is a configurable value set by the operator.

Range definitions:

• Evaluation Range: Maximum range at which the vessel detects and begins to evaluate contacts. Evaluation Range is variable, based on sensor performance and environmental conditions.

• Give-Way Range: Distance at which a vessel may begin to manoeuvre as the give-way vessel. Give-Way Range is variable, based on the known and/or estimated characteristics of the vessels involved.

• Doubt Range: Distance at which the vessel, despite being the stand-on vessel, will begin to take action to remain outside of Minimum CPA. Doubt Range is variable, based on the known and/or estimated characteristics of the vessels involved.

• Extremis Range: Distance at which the vessel will take action to avoid or reduce the effects of a collision. Extremis Range is a fixed value based on vessel characteristics.

All regions are shown as circular in Figure 2 for the purposes of simple visualisation; however, manoeuvre autonomies are not precluded from establishing complex-curve ranges to account for varying uncertainty in differing meeting situations and relative bearings.

4. Metrics development

4.1 COLREGs manoeuvring rules compliance evaluation

This COLREGs scoring algorithm uses position and vector data from ownship and contact vessels to calculate individual metric scores on a scale of 0 to 1. A final overall run score is then calculated using a weighted average of the metric scores, which results in an overall evaluation between 0 and 1.

4.1.1. COLREGs manoeuvring metrics

It is important to quantify the performance of the MA beyond strictly subjective subject matter expert (SME) evaluation. Early in the development of a prototype MA, SME evaluation is often used to provide guidance to developers regarding what manoeuvres a mariner would find acceptable or unacceptable. However, as MA matures, it becomes challenging to assess, with SME evaluation alone, whether a newer version of the MA is better than the previous version: a binary, qualitative evaluation may not be adequate. Quantifiable metrics are needed to provide a more granular level of evaluation and a means of comparing identical simulation runs or nearly identical on-water runs between MA versions.

COLREGs engagements can be categorised into separate cases in the analysis: Overtaking, Being Overtaken, Crossing Stand-on, Crossing Give-way, Head-on and manoeuvring for Multiple Vessels Encounters, as listed in Table 1. For each, the analysis includes scenarios in which the other vessel is compliant and noncompliant with the rules. Regardless of the type of interaction with another vessel, COLREGs Rule 8 describes ‘Action to Avoid Collision’. These actions represent what would be considered good practice and were used to assess the recommended actions of the MA during all engagements.

Table 1. COLREGs engagement types

To evaluate each situation, the proposed scoring methodology defines 12 metrics across three categories: COLREGs Performance, Mission and Safety. Each metric is scored on a scale of 0 to 1. An overall score is calculated using a weighted average of these scores, as shown in Table 2.

Table 2. COLREGs metrics

(1) $$CC = min\left( {{{\left| {Course{\rm{\;}}Change} \right|} \over {30}},1} \right)$$

(2) $$SC = {\rm{\;}}min\left( {{{\left| {Speed{\rm{\;}}Change{\rm{\;}}\left( {kts} \right)} \right|} \over 5},1} \right)$$

(3) $$\# CC = \mathop {\max }\limits_{} \left( {\cos \left( {{{Num{\rm{\;}}Course{\rm{\;}}Changes} \over 6}} \right),0} \right)$$

(4) $$\# SC = \mathop {\max }\limits_{} \left( {\cos \left( {{{Num{\rm{\;}}Speed{\rm{\;}}Changes} \over 6}} \right),0} \right)$$

(5) $$CI = \cos \left( {{{2\,{\rm{*}}\pi {\rm{*}\,}Indecisive{\rm{\;}}Time} \over {Min{\rm{\;}}Range{\rm{\;}}Time}}} \right)$$

(6) $$SI = \cos \left( {{{2\,{\rm{*}}\pi {\rm{*}\,}Indecisive{\rm{\;}}Time} \over {Min{\rm{\;}}Range{\rm{\;}}Time}}} \right)$$

(7) $$DIS = {{Straight{\rm{\;}}Line{\rm{\;}}Distance} \over {Actual{\rm{\;}}Distance{\rm{\;}}Traveled}}$$

If goal waypoint not achieved, $DIS = 0$

(8) $$T = {{Straight{\rm{\;}}Line{\rm{\;Travel\;Trime}}} \over {Actual{\rm{\;}}Travel\;Time}}$$

(9) $$MinRng = {{Range} \over {Range{\rm{\;}}Threshold}}$$

(10) $$CB = \mathop {\min }\limits_{} \left( {{{Range{\rm{\;}}\left( {nm} \right)} \over {Contact{\rm{\;}}Speed{\rm{\;}}\left( {kts} \right){\,\rm{*}\,}0.1}},1} \right)$$

4.1.2. COLREGs manoeuvring measures of performance

Safety is emphasised by increasing the weight of the minimum range metric and crossing bow metric by a factor of 3 and 2, respectively. All other metrics are weighted equally. The overall score is described in Equation (11). Additionally, the score will be automatically failed (0) if the T, MinRng or CB score is 0.

(11) $$Scor{e_{1v1}} = {{\left( {{\rm{CC}} + {\rm{SC}} + {\rm{numCC}} + {\rm{numSC}} + {\rm{CI}} + {\rm{SI}} + {\rm{DIS}} + {\rm{T}} + 3{\,\rm{*\,MinRng}} + 2{\,\rm{*\,CB}}} \right)} \over {13}}$$

5. Design of at-sea testing

When designing at-sea tests, every effort was made to create repeatable controlled situations that would enable direct comparisons of the scores for successive MA versions. The vehicle’s world model, or its view of everything around it, was free of other contacts and hazards except for those controlled and included in the test. Test days were chosen based on forecasted environmental conditions that would have minimal impact on vessel actions, i.e. low sea state and light winds.

Test scenarios included head-on, crossing and overtaking engagements of Ownship (OS)Footnote ⁷ in both give-way and stand-on positions as well as scenarios in which the Contact Vessel (CV) is compliant and non-compliant. Additional scenarios included the edge casesFootnote ⁸ in which these engagement types converge (i.e. where crossing becomes head-on). Care was taken in the test setup as small variations in parameters can have dramatic effects on the outcome of the test.

The initial relative bearings, headings and speeds of OS and CV were chosen to result in a CPA that violates the OS minimum CPA threshold. To isolate MA from perception, the initial range from OS to CV were distant enough that the SUT had a fully populated world model and had a stable track on CV before it enters the COLREGs range.

6. Data collection

Guidance for data collection during on-water testing is necessary to ensure accurate scoring can be performed. Omitted or missing data for contact vessels that may have impacted MA decision making will lead to inaccurate scoring results. Therefore, the following data were collected for all vessels within detection range of the OS or within the full operational area, whichever is larger.

• Position: Latitude and Longitude [World Geodetic System 1984 (WGS84)] in positive and negative decimal degrees to at least six decimal places.

• Velocity: Speed over ground in meters per second and course over ground in clockwise positive degrees relative to True North.

In the absence of position/vector data, Automatic Identification System (AIS) data were used. Additionally, the locations of buoys and other static hazards were recorded. However, in certain scenarios, AIS position and reporting intervals may not be precise enough or frequent enough to be relied upon to generate an accurate score.

7. Example results

Example results are shared below to demonstrate the application of the scoring algorithm. These are simulated results using a basic ‘if-then’ decision tree for autonomy and ‘cookie cutter’ sensors.

Figure 3 shows a crossing scenario example in which OS is the give-way vessel and CV is the stand-on vessel. The small numbers on the track lines indicate time steps and the CPA is noted. The individual metric scores and overall score are shown to the right of plot.

Figure 3. Crossing give-way Scenario 1.

In this example, a collision is avoided and the OS correctly gives way, but the behaviour is not ideal. OS initiates several manoeuvres back and forth, and several speed changes between the start and time point 3 before passing behind the contact and coming back on course. The exhibited behaviour could cause confusion for the other vessel, injecting more risk into the situation, which shows unfavourably in the COLREGs score. The large number of course and speed changes in a short time affects the ‘number of course changes’ (numCC), ‘course indecision’ (CI), ‘number of speed changes’ (SC) and ‘speed indecision’ (SI) metrics. Additionally, OS violates the 2-nm standoff range and comes within 1.89 nm of CV, resulting in a lower MinRng metric score. The overall score of 0.74 suggests there is room for improvement in the handling of the scenario.

Figure 4 shows a similar crossing scenario. In this example, OS makes a single, obvious course change, maintains speed, passes behind the contact and then comes back on course. OS does violate the 2-nm standoff range by coming within 1.97 nm of CV, but the overall score of 0.99 reflects a more favourable behaviour performance.

Figure 4. Crossing give-way scenario 2.

The metrics and overall score calculations were developed for the purpose of evaluating results relative to each other. For example, assume scenario 1 above is version 1.0 of an MA. The overall score reveals a problem in the MA, but the individual metrics reveal the deficiencies in the course and speed change logic of the MA. Changes are made to the MA and OS is tested again in scenario 2. The overall score shows that the issues have been addressed and the individual metric scores confirm that the changes did not affect the other metrics.

8. Next steps

8.3. Evaluation of safe speed and in-extremis

Lastly, scoring will need to be developed to apply to other manoeuvring situations such as Safe Speed and In-Extremis. In general, if two vessels come close to each other at sea, adherence by both to the COLREGs manoeuvring rules should prevent a collision. There are certainly exceptional circumstances, such as equipment casualties, reduced visibility, confined waters or combinations of such factors. When only one of the vessels is following the rules, the probability of collision is much higher. The COLREGs accounts for this possibility; in particular, Rule 17(a)(ii) authorises a stand-on vessel to take action as soon as it becomes apparent that the give-way vessel is not taking appropriate action. Moreover, Rule 17(c) states ‘A power-driven vessel which takes action in a crossing situation in accordance with 17(a)(ii) of this Rule to avoid collision with another power-driven vessel shall if the circumstances of the case admit, not alter course to port of a vessel on her own port side’. At a later point in the developing encounter, collision can only be avoided by simultaneous action from both vessels. At that point, the vessels are said to be ‘In-Extremis’, and both vessels have the responsibility to do whatever it takes to avoid the collision. This In-Extremis situation is covered by Rule 17(b): ‘When, from any cause the vessel required to keep her course and speed finds herself so close that collision cannot be avoided by the action of the give-way vessel alone, she shall take such action as well best aid to avoided collision’. This essentially means that both vessels must manoeuvre to avoid collision, and presumably there is a coordinated pairing of best courses and speeds by the two vessels to do so.

The range at which the vessels are considered to be In-Extremis depends on the ‘circumstances of the case’ with factors such as manoeuvrability, size and relative speed of the vessels being critical. Since we are considering the problem from the perspective of the vessel with MA, we assume that the vessel with MA is the stand-on vessel and the other vessel is the give-way vessel. While the other vessel’s manoeuvring characteristics are unknown, the vessel with MA can quickly compute the outcomes of multiple course and speed combinations using the assumption that the other vessel does not manoeuvre. By definition, if there is a course and speed combination for the vessel with MA that avoids collision when the other vessel maintains course and speed, the vessels are not yet In-Extremis. Once no such course and speed combination exists, the vessels are In-Extremis.

An alternate method is to assume conservatively that the other vessel’s tactical diameter is fiveFootnote ⁹ times that vessel’s length and that the turning characteristics of both vessels are symmetrical. Additionally, avoiding collision requires that no part of either vessel will touch during the manoeuvre. Given these assumptions, Extremis range can be computed by Equation (12). Figure 5 illustrates the outer boundary of Extremis range.

(12) $${\rm{Extremis\;Range}} = {\rm{\;}}5 \times {{Own{\rm{\;}}ship{\rm{\;}}length} \over 2} + Own{\rm{\;}}ship{\rm{\;}}length + Contact{\rm{\;}}length$$

Figure 5. Scaled illustration of extremis range.

Guidance on how an In-Extremis manoeuvre is conducted is not entirely clear. The practice of not turning to port for a vessel on the port side has in common practice and training been applied to the In-Extremis situation as described in Naval Ship handlingFootnote ¹⁰ (Crenshaw, 1984). Court rulings have been mixed with a majority of the rulings that have not found at fault the previous stand-on vessel, which is now In-Extremis, for initiating a port turn to a vessel on her port side. However, in some cases, a turn to port was deemed a contributing cause. See Farwell’s Rules of the Nautical Road, pages 101–103 and 223–225 (Allen, Reference Allen2020).

Rule 6 of COLREGs is ‘Safe Speed’, and states essentially a safe speed is necessary to reduce the risk of collision. This should be evaluated prior to any manoeuvre to determine if the entering speed is acceptable. If the speed is too high or ‘unsafe’, any action after that is already insufficient to avoid collision. Rule 6 states ‘Every vessel shall at all times proceed at a safe speed so that she can take proper and effective action to avoid collision and be stopped within a distance appropriate to the prevailing circumstances and conditions’. Also, Rule 19 (b) is ‘Every vessel shall proceed at a safe speed adapted to the prevailing circumstances and conditions of restricted visibility’. A rule of thumb is that safe speed is ‘fast enough to manoeuvre but slow enough to stop in time’.

Adjusting to a safe and reasonable speed should be initiated prior to enacting any manoeuvring; therefore, it is recommended that COLREGs Rule 6 ‘Safe Speed’ be assessed continually by MA.