A research paper by scientists at Hefei University of Technology presented an intuition-guided deep reinforcement learning framework for soft tissue manipulation under unknown constraints.
The research paper, published on Apr. 14, 2025 in the journal Cyborg and Bionic Systems.
Intraoperative soft tissue manipulation is a critical challenge in autonomous robotic surgery. Furthermore, the intricate in vivo environment surrounding the target soft tissues poses additional hindrances to autonomous robotic decision-making. Previous studies assumed the grasping point was known and the target deformation could be achieved. The constraints were assumed to be constant during the operation, and there were no obstacles around the soft tissue. "To address these problems, we proposed an intuition-guided deep reinforcement learning framework based on soft actor-critic (ID-SAC) for soft tissue manipulation under unknown constraints." said the author Xian He, a researcher at Hefei University of Technology.
The process of this framework can be divided into the following two parts. The first part is selection and evaluation of grasping points in autonomous soft tissue manipulation by robots. A deep Q-networkbased algorithm was proposed to address the initial point selection issue in soft tissue manipulation. First, high-dimensional states were described and used as inputs to the network, which outputs the action value for each grasping point. The grasping point corresponding to the maximum action value was selected as the initial grasping point. Simultaneously, a grasp point quality assessment network was designed to predict the success rate of a selected grasp point for achieving the soft tissue manipulation task. The point selection operation was revised if the success rate during testing fell below a certain threshold. Subsequently, based on the selected grasp points, autonomous robotic manipulation uses a simulated IM mode akin to a physician's approach to execute the manipulation task. Safety constraints were designed to ensure that when the soft tissue undergoes excessive deformation, the robot relinquishes the manipulation task. The network is updated through feedback from the manipulation trajectories of each point selection. A reward function was developed based on expert knowledge.
The second part deep reinforcement learning-based fusion strategy for soft tissue manipulation modes. The SAC algorithm was used to execute soft tissue manipulation tasks after selecting initial grasping points in step 1, addressing the intricate and dynamic environment of soft tissue manipulation within the body. The regulatory factor is set as the action of the reinforcement learning algorithm to autonomously integrate multiple soft tissue manipulation modes, such that the manipulation strategy integrates the advantages of different manipulation modes and improves the robot's ability to make autonomous decisions in complex scenes.
The framework was tested through simulations using a liver model. The results showed that the proposed approach could effectively handle three deformation tasks (position-based, curve-based, and region-based) while accounting for obstacles and dynamic disturbances. It was also demonstrated that the ID-SAC algorithm outperformed traditional SAC algorithms in terms of task performance and exploration capabilities, especially in scenarios with obstacles and unknown disturbances. When comparing the robotic system's performance to human manipulation, the robot exhibited faster task completion times, smoother trajectories, and more efficient manipulation, although human operators sometimes maintained better control in certain complex scenarios. "Our ID-SAC framework provided a new solution for autonomous operation of robots in complex and dynamic soft tissue surgical environments, with broad prospects for medical and surgical applications." said Xian He.
Authors of the paper include Xian He, Shuai Zhang, Jian Chu, Tongyu Jia, Lantao Yu, and Bo Ouyang.
This work was supported in part by the Young Scientists Fund of the National Natural Science Foundation of China (grant no. 52305018), in part by the National Key Research and Development Program of China (grant no. 2023YFB4706000), in part by the Joint Funds Program of the National Natural Science Foundation of China (grant no. U21A20517), and in part by the Basic Science Centre Program of the National Natural Science Foundation of China (grant no. 72188101).
The paper, "Intuition-guided Reinforcement Learning for Soft Tissue Manipulation with Unknown Constraints" was published in the journal Cyborg and Bionic Systems on Apr. 14, 2025, at DOI: 10.34133/cbsystems.0114.
