AI Boosts Realistic 3D Hand-Object Surgery Sims

Abstract

Reconstructing 3Ds of hand-object interaction (HOI) is a fundamental problem that can find numerous applications. Despite recent advances, there is no comprehensive pipeline yet for bimanual class-agnostic interaction reconstruction from a monocular RGB video, where two hands and an unknown object are interacting with each other. Previous works tackled the limited hand-object interaction case, where object templates are pre-known or only one hand is involved in the interaction. The bimanual interaction reconstruction exhibits severe occlusions introduced by complex interactions between two hands and an object. To solve this, we first introduce BIGS (Bimanual Interaction 3D Gaussian Splatting), a method that reconstructs 3D Gaussians of hands and an unknown object from a monocular video. To robustly obtain object Gaussians avoiding severe occlusions, we leverage prior knowledge of pre-trained diffusion model with score distillation sampling (SDS) loss, to reconstruct unseen object parts. For hand Gaussians, we exploit the 3D priors of hand model (i.e., MANO) and share a single Gaussian for two hands to effectively accumulate hand 3D information, given limited views. To further consider the 3D alignment between hands and objects, we include the interacting-subjects optimization step during Gaussian optimization. Our method achieves the state-of-the-art accuracy on two challenging datasets, in terms of 3D hand pose estimation (MPJPE), 3D object reconstruction (CDh, CDo, F10), and rendering quality (PSNR, SSIM, LPIPS), respectively.

Researchers at UNIST have announced the development of a groundbreaking artificial intelligence (AI) technology capable of reconstructing three-dimensional (3D) representations of unfamiliar objects manipulated with both hands, as well as simulated surgical scenes involving intertwined hands and medical instruments. This advancement enables highly accurate augmented reality (AR) visualizations, further enhancing real-time interaction capabilities.

Led by Professor Seungryul Baek of the UNIST Graduate School of Artificial Intelligence, the team introduced the BIGS (Bimanual Interaction 3D Gaussian Splatting), an innovative AI model that can visualize complex interactions between hands and objects in 3D using only a single RGB video input. This technology allows for the real-time reconstruction of intricate hand-object dynamics, even when the objects are unfamiliar or partially obscured.

Traditional approaches in this domain have been limited to recognizing only one hand at a time or responding solely to pre-scanned objects, restricting their applicability in realistic AR and VR environments. By contrast, BIGS can reliably predict full object and hand shapes, even in scenarios where parts are hidden or occluded, and can do so without the need for depth sensors or multiple cameras-relying solely on a single RGB camera.

The core of this AI model is based on 3D Gaussian Splatting, a technique that represents object shapes as a cloud of points with smooth Gaussian distributions. Unlike point cloud methods that produce sharp boundaries, Gaussian Splatting enables natural reconstruction of contact surfaces and complex interactions. The model further addresses occlusion challenges by aligning multiple hand instances to a canonical Gaussian structure and employs a pre-trained diffusion model for score distillation sampling (SDS), allowing it to accurately reconstruct unseen surfaces, including the backs of objects.

Figure 1. Results of reconstructing hand-object interactions from various viewpoints using the 'BIGS' method.