1. Introduction
Circular Economy (CE) is a framework emphasizing the need to shift production and consumption patterns from linear to closed-loop to reduce used resources and waste (Ellen MacArthur Foundation, 2013). Reference Potting, Hekkert, Worrell and HanemaaijerPotting et al. (2017) presents ten circular strategies with increasing value retention. Beyond Recycle (R8), higher-value approaches include Remanufacture (R6) and Repurpose (R7), which involve reusing existing parts to create a product with the same or different function, respectively. Throughout this paper, “part reuse” refers to both Remanufacture and Repurpose and should not be confused with Reuse (R3), where the product is used again in its entirety. Reference GorgolewskiGorgolewski (2017) introduces the concept of “Form Follows Availability”, inverting the traditional design process. Instead of first creating a prescribed design and then extracting resources and manufacturing the necessary parts, this approach shapes the design from the outset based on the available materials and components. In this perspective, Reference Önalan, Mitropoulou, Triantafyllidis, Hunhevicz and De WolfÖnalan et al., 2025 highlight the need for more computationally efficient and scalable computational methods for circular design with non-standard materials. Although the literature demonstrates successful applications of (mixed) integer linear programming to frame and solve part reuse design problems Reference Brütting, Senatore and Fivet(Brütting, Senatore, et al., 2019; Reference Huang, Alkhayat, De Wolf and MuellerHuang et al., 2021; Reference Zhang and SheaZhang & Shea, 2024), in face of non-linearities embedded in the design space and lack of general purpose, robust and efficient non-linear mixed variable optimizer, such problems are much harder to solve computationally.
An emerging approach to solve engineering design tasks is the application of Reinforcement Learning (RL). RL centers on an agent that observes a state, takes an action, and receives a reward that reflects how well that action contributes to a long-term goal. Over many interactions, the agent learns a policy that maximizes cumulative reward to achieve a given task (Reference Sutton and BartoSutton & Barto, 2018). This strategy, formalized as a Markov decision process and involving sequential decisions over a large search space, has been shown to be an effective mathematical model for solving synthesis tasks in engineering design (Reference Ororbia and WarnOrorbia & Warn, 2023). RL focuses on a long-term goal and so adheres well with the characteristics of the engineering design process in which designers evaluate the feasibility of individual solution alternatives at varying levels of abstraction, and where each assessment is essential to drive the design process but also inevitably narrows down the available search space. In a pure mathematical programming context, Reference Mehta, Taghipour and SaeediMehta et al (2022) emphasize the potential of RL for addressing non-linear, non-convex, and discrete variable optimization problems, demonstrating its effectiveness on a bi-objective travelling salesperson problem.
Motivated by the potential of RL to solve engineering design tasks, this work presents an initial investigation into the development of a computational method for the circular design of planar mechanisms built from available bars and pins using RL. The key challenges in the computational design of mechanisms using available parts include the combinatorial nature of the problem coupled with the scarcity of valid designs. Further, for mechanical products with moving parts, the kinematics is often governed by a set of implicit non-linear motion equations and complex constraints that are difficult to satisfy (Reference Escande and SheaEscande & Shea, 2025). This paper demonstrates how stock-constrained design, meaning mechanisms that are created only from bars and pins already in inventory, can be designed using RL. The work also proposes a bipartite graph representation and an elementary action formulation that can be used to generate diverse topologies, assembling mechanisms step by step. The work focuses on a specific motion generation problem: the force inverter mechanism.
The paper is organized as follows. Section 2 reviews related work and introduces necessary background. Section 3 describes the proposed framework, including the environment setup, representation, and action formulation. Section 4 presents and discusses the results obtained from different training scenarios. Finally, Section 5 concludes the paper and outlines directions for future work.
2. Background and related work
Sections 2.1 and 2.2 introduce relevant works in computational design for the reuse of parts and in planar mechanisms synthesis, respectively. Section 2.2 also provides background on a specific method for static and kinematic determinacy analysis of truss structures that is used in this work. Section 2.3 discusses relevant works on RL in design. The motivation for this work is discussed in Section 2.4.
2.1. Computational design for the reuse of parts
Reference Brütting, Desruelle, Senatore and FivetBrütting, Desruelle, et al. (2019) propose a framework for designing truss structures using a stock of structural components. Strating from a ground-structure based layout, the framework utilizes a Mixed Integer Linear Programming (MILP) to optimally assign the structural components from the stock. However, since the assigned structural components do not necessarily match the corresponding distances between the nodes of the ground structure, the framework uses non-linear programming in a subsequent step to optimize the ground structure to match the lengths of the assigned elements. The iteration between the two steps results in optimized truss designs which are also subject to embodied energy and embodied carbon design criteria.
Reference Escande and SheaEscande and Shea (2025) bring computational circular design to moving structures, namely planar linkages with LEGO® Technic beams. A planar linkage is a mechanism in 2D consisting of links or bars interconnected at joints or pins. When actuated, the mechanism produces a desired trajectory traced at a particular joint. In their work, Reference Escande and SheaEscande and Shea (2025) aim at reusing standardized components to generate a database of linkage designs from an inventory of available parts, and to apply optimization methods to inversely design mechanisms for a user-defined trajectory. Additionally, using a bi-objective optimization setup, the work also examines the trade-offs between the kinematic performance of a mechanism and its CO₂ footprint when incorporating new parts using a bi-objective optimization setup.
2.2. Planar mechanisms synthesis
Planar mechanism synthesis is a classical problem in mechanical design and remains actively researched in the field of deep learning (Reference Deshpande and PurwarDeshpande & Purwar, 2020; Reference Nobari, Srivastava, Gutfreund and AhmedNobari et al., 2022, Reference Nobari, Srivastava, Gutfreund, Xu and Ahmed2024). For example, Reference Deshpande and PurwarDeshpande and Purwar (2020) use an image-based representation and variational auto encoder to sample trajectories traced at individual joints in a mechanism and compare those to the nearest neighbors in a database of four-bar and six-bar linkages based on the learned representation in the latent space. In comparison to Reference Deshpande and PurwarDeshpande and Purwar (2020), the work of Reference Nobari, Srivastava, Gutfreund, Xu and AhmedNobari et al. (2024) applies a more comprehensive approach in which a graph based neural network representation is utilized to learn both joint embeddings of individual designs, as well as their performance, based on a large database of linkages. Their work assumes the lengths of linkages as continuous variables. To ensure manufacturability, MILP is employed to postprocess mechanism designs, resulting in designs that are organized stacked in layers to avoid collisions.
Reference Lumpe and SheaLumpe and Shea (2023) present a two-stage optimization framework for the design of planar pin-joined shape morphing lattice structures. Starting from a ground structure based topology definition, the first stage of the framework finds a topology of a structure for a specific number of degrees of freedom based on static and kinematic determinacy analysis of truss structures according to Reference Pellegrino and CalladinePellegrino and Calladine (1986). The analysis yields the self-stress index
, the mechanism index
and the linearized motions of the structure. The self-stress index
represents the number of independent self-stress states in the structure, which are internal force distributions that satisfy equilibrium without any external loads. The mechanism index
represents the number of independent mechanisms or rigid-body motions that the structure can undergo without changing the lengths of its members (Reference Pellegrino and CalladinePellegrino & Calladine, 1986). Based on the determinacy analysis, the second stage of the framework is initiated in which search methods are used to determine a non-linear motion of the structure. The resulting motion yields the desired target positions or trajectories of the pin-joints.
2.3. Reinforcement learning in design
To support structural design of lightweight truss layouts subject to design constraints, Reference Du, Zhao, Yu, Yao, Song, Wu, Luo, Liu, Zhao and WuDu et al. (2023) propose a two-stage computational framework. The first stage applies search to generate various and feasible truss layout configurations, while the second stage refines the generated truss layouts using RL. The necessity for the two-stage framework stems from the fact that most truss designs are infeasible due to the imposed constraints, resulting in a sparse reward that hinders the RL policy learning process.
Reference Gallego, Muñoz, Viquerat and AguirreGallego et al. (2022) propose a RL-based strategy to optimize four-bar linkages. In their work, the agent observes the state space of linkages characterized by a fixed length feature set, selects an action from a set of predefined actions, and performs the action to define a new linkage. The agent is then updated based on the reward derived from its subsequent actions. Reference Vermeer, Kuppens and HerderVermeer et al. (2018) integrate RL into the kinematic synthesis of planar linkages using a rule-based approach. Their work demonstrates how RL can be utilized to generate mechanisms capable of following complex desired paths.
Closely relevant for this work are the applications of RL for the sequential assembly of predefined building-blocks into objects within physics-based environments. In this context, the objects are designed to achieve the desired functionality (Reference Bapst, Sanchez-Gonzalez, Doersch, Stachenfeld, Kohli, Battaglia and HamrickBapst et al., 2019), meet the shape of target blueprints (Reference Ghasemipour, Kataoka, David, Freeman, Gu and MordatchGhasemipour et al., 2022), or are assembled into structural shells based on the part availability to facilitate material-informed circular strategy (Reference HuangHuang, 2021). Similarly, the work from Reference Chung, Kim, Knyazev, Lee, Taylor, Park and ChoChung et al. (2021) utilizes RL to learn sequential brick construction tasks for 3D object generation using one type of LEGO® brick given a target image.
2.4. Motivation
Structural design of lightweight truss layouts, as well as generic building-block sequential assembly tasks, are characterized with search spaces with complex landscapes and predominantly infeasible designs due to the imposed constraints. The mechanism synthesis problems exhibit similar challenges. The design space is characterized by a complex non-linear landscape with high validity scarcity (Reference Escande and SheaEscande & Shea, 2025; Reference Nobari, Srivastava, Gutfreund, Xu and AhmedNobari et al., 2024) due to the kinematic singularities arising from the actuator motion and invalid mechanism configurations. Despite the challenges posed by sparse rewards stemming from infeasible design configurations, RL learns policies to achieve a long-term goal over a sequence of decisions. This strategy shows to be advantageous in solving mechanism synthesis problems, as well as complex engineering design problems in general (Reference Ororbia and WarnOrorbia & Warn, 2023). The work presented here attempts to build directly on these advantages and extends the work from Reference Escande and SheaEscande and Shea (2025) to explore RL for circular, inventory-constrained mechanism synthesis. In comparison to related works which focus exclusively on parametric design with fixed topology (Reference Du, Zhao, Yu, Yao, Song, Wu, Luo, Liu, Zhao and WuDu et al., 2023; Reference Gallego, Muñoz, Viquerat and AguirreGallego et al., 2022), continuous variables (Reference Nobari, Srivastava, Gutfreund, Xu and AhmedNobari et al., 2024), or sequential brick construction tasks for static structures (Reference Bapst, Sanchez-Gonzalez, Doersch, Stachenfeld, Kohli, Battaglia and HamrickBapst et al., 2019; Reference Ghasemipour, Kataoka, David, Freeman, Gu and MordatchGhasemipour et al., 2022; Reference HuangHuang, 2021), this work combines topology generation and part reuse for synthesis of planar linkages using RL. To facilitate the planar linkages design process, this work employs the same determinacy analysis as Reference Lumpe and SheaLumpe and Shea (2023) based on Reference Pellegrino and CalladinePellegrino and Calladine (1986). Additionally, to simplify the initial attempt of using reinforcement learning (RL) for circular design, this work only considers linearized motions.
3. Method
The computational method for the circular design of planar linkages from available bars and pins using RL is shown in Figure 1. An episode (Figure 1a, 1e and 1f) consists of a sequence of actions (Figure 1d) performed by an agent (Figure 1c) on the environment, given an observation of the environment (Figure 1b). The episode terminates when the desired target motion of the mechanism is achieved (Figure 1g), or when an invalid action is taken. There is no pre-filtering process for the actions. If the maximum number of actions is reached before the episode terminates, it is truncated. The agent’s goal is to maximize the return, which is the cumulative sum of rewards over an episode. The agent interacts with the environment through a policy that maps the current state, represented by the observation, to an action. After the end of an episode, the environment is reset to initiate a new episode. The design task and design representation are explained in Section 3.1 and Section 3.2, respectively. Sections 3.3, 3.4, and 3.5 provide detailed information about the agent’s actions, the physics-based environment, and the observations. Finally, Sections 3.6 and 3.7 explain the reward function, as well as details about the RL policy applied in this work.
Overview of the computational method for the circular design of truss linkages built from available bars and pins using RL. Stock of available bars and pins, scene comprising placed bars and pins, and graph representation for different steps: a. initial conditions, b. first observation, c. policy, d. first action selected by the policy, e. environment after performing this action, f. environment after performing the last action, g. evaluation of the final state (success)
Figure 1 Long description
A diagram illustrating the computational method for circular design of truss linkages using reinforcement learning. Panel A: Initial conditions showing the stock of available bars and pins, the scene with placed bars and pins, and the graph representation. Panel B: Observation features for bars and pins, including their properties and connectivity. Panel C: Policy involving an MLP actor and sampling distribution. Panel D: Action selected by the policy to connect a bar and a pin. Panel E: Environment after performing the first action. Panel F: Environment after performing the last action. Panel G: Evaluation of the final state, including determinacy, kinematics, and motion similarity.
3.1. Design task
The design task involves an initial stock comprising available bars and pins, boundary conditions, a target determinacy, and target motions, as shown in Figure 1a. The available bars are of varying length, while pins can either be support pins (ground or slider) or connectors that can connect two bars. The prescribed target determinacy indices are the self-stress index
, which quantifies the number of states of self-stress in a structure and the mechanism index
, which is the number of possible mechanisms in a structure as defined by Reference Pellegrino and CalladinePellegrino and Calladine (1986). In this work, the task is purely kinematic, with the target determinacy indices set as
and
. Figure 1a shows the design task for a force inverter, and a solution of the task is shown in Figure 1f, with a detailed view in Figure 1g. The target displacements of a mechanism
are queried at
target pins, representing the kinematic modes that the mechanism should exhibit (e.g. “a motion of pin 1 to the left associated with a motion of pin 2 to the right”).
3.2. Design representation
The mechanism design is represented by a complete bipartite graph
, as shown at Figures 1a, 1e and 1f, illustrating the graph’s structure at different stages of an episode.
and
are sets of
bars and
pins, respectively, including both the stock of available bars and pins as well as the bars and pins placed in the scene, and
is the set of edges between bars and pins. Further,
and
denote two sets of distinct classes of vertices in the graph, with their own attributes, as shown in Figure 1b. Every edge in
is inactive by default and gets activated only upon action from the agent (Figure 1e). The reasons for considering both bars and pins as vertices in the graph is to have a generic representation where a bar can connect to more than two pins and a pin can connect to more than two bars (Reference Escande and SheaEscande & Shea, 2025). If a bar or a pin is present in multiple copies in the stock, it is simply repeated in the graph.
3.3. Action
The method assumes that each bar has two holes where the pins can be inserted to connect with other bars from the stock. The choice of the connecting hole is labeled with
, denoting a hole located at the beginning of a bar for
, while
denotes a hole at the end of the bar. The hole location
is an attribute of each active edge in
. Therefore, the action of an agent consists of connecting a bar
with a pin
at the hole location
on the bar. When the action is applied to the environment, the hole location value is assigned to the attribute of the newly activated edge between bar
and pin
. The action is encoded by
, where the first pin connected to a bar is always associated to
and the second pin added to this bar to
. No other pin is expected to connect to the bar, and the space of all possible actions is thus
. Figures 1d and 1e show an example of an action and the associated connection.
3.4. Physics-based environment
This work uses Pymunk, a 2D physics-based solver (Reference BlomqvistBlomqvist, 2025) to determine the positions of bars and pins in the scene when applying a new action. By using a physics solver, each action remains elementary, while the resulting assembly emerges due to the mass-spring interaction modelling over the course of the episode, allowing the agent to implicitly learn the physical interactions governing the mechanism. Bars and pins are modelled as rigid dynamic bodies without collision. Bars have a mass and moment of inertia according to their length, while pins are punctual and have a fixed mass and moment of inertia. The connections between bars and pins of a mechanism are modelled as PivotJoint, which are rigid connections, but allow for some temporary misalignment and push the bars and pins closer. The environment must accept and handle any action performed by an agent from the action space
. Many of such actions can easily be detected as invalid based on the bipartite graph design representation: for example, if bar
and pin
are already connected, or if a bar
already has two pins connected. For the remaining actions in
, the physics-based solver checks if an action is geometrically invalid, for example when a bar is two short or too long to connect two pins. This is accomplished by inspecting possible constraint violations of the PivotJoints at the end of the physics simulation of the action. If the violations exceed a certain threshold, the action is reverted returned as invalid. The physics space has zero gravity, and a global damping term stabilizes the environment. After the solver connected and stabilized all bars and pins in the scene, the truss pin-bar representation is updated. Based on the bipartite graph
, the self-stress index
and the index
indicating the number of possible mechanisms in a structure, as well as the current mechanism modes
, are calculated according to Reference Pellegrino and CalladinePellegrino and Calladine (1986).
3.5. Observation
The observation is a single fixed-length vector (Figure 1b) that concatenates all the bars features, pins features, and connectivity features of the bipartite graph
. Common features shared by both bars and pins include the positions
of their center of gravity in the scene and a usage flag indicating whether a bar or pin is present in the scene. Specific features for bars include the bar length
and the bar orientation, represented as
, whereas for pins they include a one-hot encoding vector for the type of pin (fixed,
-fixed,
-fixed, or free connector), a target flag for target pins, the target motion
for that pin, and the current motion
. All spatial features of bars and pins, such as positions and bar lengths, are normalized by diving them by the reference length
, which represents the maximum extent of the scene. If a bar or pin is in the stock, its position is set to
. If a pin is not a target pin corresponding to the target displacements of the mechanism, then
are set to 0. The connectivity is encoded using the flattened bi-adjacency matrix
. The matrix
is a binary matrix of size
, with elements
equal to 1 if the edge between bar
and pin
is activated.
3.6. Reward
Along with the observation, the environment returns to the agent the reward associated with the action performed. The reward, a value reflecting how much the performance of a mechanism improved or worsened after performing the action, is given in Equation 1:
where
rewards valid action and penalizes invalid ones,
counts the difference in the number of incomplete bars (those not yet connected to two pins) before and after the action,
measures the distance between the determinacy of the current mechanism design in comparison to the desired determinacy, and
measures how closely nodal displacement of the current design match the target nodal displacement. When an action is invalid, it is either not applied to the physical solver or reverted after failure, such that it doesn’t change the state of the environment. A reward of
is returned for invalid actions, and no additional reward terms are calculated in Equation 1. When an action is valid,
and the other terms are evaluated. The term
is defined in Equation 2:
where
is the determinacy error for the current state, in terms of the determinacy of the evaluated mechanism design
and the target determinacy
. If the determinacy is reached
and if all target pins supposed to move are moving, then last term
in Equation 1 is evaluated accordingly Equation 3:
where
is an empirically set weight factor
, to match the magnitude of the other reward terms, and
is the cosine similarity between the current nodal displacements
and the target nodal displacements
, ranging from -1 to 1 and defined as Equation 4:
The ultimate goal of the agent is to learn how to match a certain target motion mode, rewarded by the final term
. The other terms
,
, and
can be seen as intermediate rewards, encouraging the agent to perform valid actions and to go towards the completion of the design task.
3.7. Policy
The agent is trained using Proximal Policy Optimization (PPO) (Reference Schulman, Wolski, Dhariwal, Radford and KlimovSchulman et al., 2017), chosen for its robustness and implementation simplicity. Both the policy (actor) and value (critic) functions are modeled using Multilayer Perceptron (MLP) networks. The policy network takes as input the observation vector, as shown in Figure 1c, and outputs the action probabilities, which is sampled to determine the next action. In the remainder of the work, “MLP agent” is used to denote an agent with an MLP actor and critic, trained using PPO.
4. Results and discussion
The planar linkage design task in this section is the minimal-stock force inverter problem as displayed in Figure 1a. Section 4.1 validates the environment by checking that actions are order-independent and by examining the outcomes produced when actions are selected at random. Section 4.2 investigates the performance of the method by comparing two MLP agents. The first agent is trained on a fixed stock environment, keeping the order of the bars and pins fixed throughout the training, while the second agent is trained on an environment where the bars and pins are randomly permuted between each episode. The difference in performance of these agents is used to assess how well the method generalizes across variations in part ordering. The work presented is implemented in Python using a single core 11th Gen Intel® CoreTM i7-1195G7 processor, and both agents are trained using Stable Baselines3’s PPO policy implementation. The hyperparameters of the MLP agent are presented at the beginning of Section 4.2.
4.1. Validating the environment
To validate the environment, various sequences of actions are tested on the environment. In all these, the length of the episode is
, since the force inverter requires six actions: each of the three bars requires to be connected to two pins. Ideally, for a simple design of the force inverter, the order of the actions should not matter. When only a small fraction of theoretically valid sequences leads the environment to the correct final configuration, the agent’s search becomes much harder as the combinatorial space grows, and the proportion of effective sequences shrinks. A robust physics solver is therefore essential: it should accept any action, attempt to connect the connection pairs, and resolve the structure into a stable, connected configuration whenever the geometry allows it.
To test the order independence of the environment and its physics solver, the
permutations of a known solution action sequence are applied to the environment as distinct episodes. The action sequence
is used as the reference solution. The 720 episodes are solved in 2.3s, corresponding to 3.2ms per episode, or 0.53ms per action. Of these, 660 episodes manage to design the force inverter, corresponding to a success rate of 92%. This indicates that the environment is mostly order independent, and the physics solver successfully resolves most configurations. However, few episodes do not reach the solutions, such as configurations involving mirrored kinematic loops that cannot escape the stable equilibrium and fail to connect the last bar, as displayed in Figure 2.
Success rate of the 720 permutations of a solution action sequence and resulting designs
To establish a baseline for the two RL agents that are discussed in Section 4.2, and to understand better the behavior and reward distribution of the environment, random actions are applied to the environment. The space of all possible actions
is sampled uniformly, ignoring the observations and rewards returned by the environment. This procedure generates
episodes of six random actions. Figure 3e shows the evolution of the reward distribution for each of the six steps, highlighting the paths to solutions. The resulting distribution is highly skewed towards invalid rewards. The figure shows multiple paths to the solution, some of them passing by zero or even negative rewards. In total, there are 42 successful sequences out of 100,000 episodes, resulting in a success rate of
or approximately one success in 2400 attempts. This serves as a baseline for the results of the PPO policy presented in the next section, which is trained on the same environment and same problem.
4.2. MLP agent performance
Figures 3a to 3d show the training sequences of two MLP agents that are trained on a stock with fixed bars and pins order and a stock that is randomly permuted between each episode, respectively. Both agents are trained for 200,000 steps. The PPO policy training is conducted using the default values of standard hyperparameters except for the following adjustments: discount factor
(instead of 0.99), generalized advantage estimate (GAE)
(instead of 0.95), and clipping ratio
(instead of 0.2). The default hyperparameters include a learning rate
, and the MLP based architecture consists of two fully connected layers with 64 units each and the hyperbolic tangent activation function. Key factors identified to improve the training stability and convergence are described as follows: rewards and observations are normalized; episodes terminate immediately after an invalid action; the maximum episode length is limited to 20 actions; and reducing the clipping ratio to 0.1 proves essential for stable training. Figure 3a shows how the episode length evolves during training, Figure 3b the episode reward (return), Figure 3c the explained variance, and Figure 3d the value loss.
Main results; the first row shows various metrics of the MLP agent training: episode length (a), episode reward (b), explained variance (c) and value loss (D); the second row shows reward distribution throughout the steps for: random action
The first agent, trained on the fixed-order stock, is first tested with the same fixed stock (Figure 3f). Then, it is tested on shuffled stocks (Figure 3g). The second agent, that is trained using randomly permuted stocks, is tested on permuted stocks as well (Figure 3h). The first agent converges in about 80,000 steps to the solution (episode reward of 23), while the second agent converges slower, in about 150,000 steps and only reach episode rewards of 15. Both trainings reach explained variances above 0.5, which suggests that their critics learn reasonably accurate value estimates. The weaker and slower convergence of the shuffled-stock agent stems from the added permutation variability.
Figures 3f to 3h offer further insights into the performance of the MLP agents compared to the random agent baseline in Figure 3e. The first agent performs almost perfectly when evaluated in the same setting as its training environment, achieving a 98.5% success rate (Figure 3f). However, when tested on shuffled stocks, the success rate falls to 18.7% (Figure 3g), indicating that it does not generalize beyond the specific ordering observed during training. This suggests that the agent has memorized solution paths tied to bar and pin indices rather than learning the underlying relationships between actions, observations, and rewards. In contrast, the second agent, trained directly on shuffled stocks, reaches a 66% success rate (Figure 3h), which demonstrates a significantly improved generalization and a marked reduction in invalid actions compared to the results in Figure 3g. Overall, all three cases outperform the random agent baseline, which has a success rate of 0.04%, as reported in Section 4.1. In addition, the MLP agents have learned to avoid early invalid actions, as shown in Figures 3f and 3h.
5. Conclusion and future work
This work presents an initial investigation into applying RL for the circular design of planar linkages using available bars and pins. Motivated by the need for computational methods capable of handling combinatorial, inventory-constrained design spaces with complex non-linear kinematics, the proposed method combines a bipartite graph representation of bars and pins with an elementary action formulation. Using an MLP agent, trained with PPO, mechanisms are assembled step-by-step in a physics-based environment, receiving rewards that guide the agent toward achieving target kinematic motions and determinacy. These preliminary results demonstrate that RL can successfully generate valid mechanisms from limited stock: the agent trained on a fixed stock achieves near-perfect performance for the force inverter, while the agent trained with shuffled stocks generalizes moderately well, achieving a 66% success rate. Both agents significantly outperform the random action baseline and learn to avoid early invalid actions, illustrating that RL can effectively navigate sparse, combinatorial design spaces for mechanism synthesis.
The current approach has scalability limitations, as the bipartite graph representation grows linearly with the number of bars and pins, potentially limiting performance in larger inventories and more complex and varied topologies. Scalability will be evaluated and benchmarked against classical optimization methods in a follow-up study. Future work will also explore indexing-invariant architectures, such as graph neural networks and transformer models, to improve generalization across varying stock orders. Additionally, expanding the action space to allow more flexible connection options: either discrete, as in LEGO® Technic-style connections, or continuous for arbitrary hole placements, could enable the design of more realistic mechanisms. These extensions would bring RL-based circular design closer to practical applications in sustainable, inventory-driven mechanical engineering design.
Open access
