A collaboration between the University of Helsinki and the Finnish company Twinverse has tested how well today's AI can label the individual objects inside 3D models of industrial sites, and where it still falls short.
Digital twins are virtual 3D replicas of real spaces such as factories, warehouses and industrial plants. Companies use them to monitor equipment, plan maintenance and simulate changes without disrupting operations.
For a digital twin to be genuinely useful, the system has to understand what each object in the scene actually is: which part is a pipe, which is a container, which is a machine. Identifying and labelling those objects is known as the segmentation problem.
Ofia Rahman Alif's thesis examines this problem. The work was funded by the project, which supports the adoption of artificial intelligence in small and medium-sized enterprises (SMEs).
The research was carried out in collaboration between the University of Helsinki's and , a Finnish company that builds digital twins of complex industrial facilities.
- This study was an opportunity we could not have otherwise taken. The results were insightful and we remain open to continuing collaboration in the future. I encourage companies to be active and understand what an academic thesis is, how to guide and push thesis writers to do their best. There's little to no cost to the company other than the time investment, with genuinely useful results, says Arttu Julin, CTO of Twinverse.
What works and what doesn't in real factories
Alif's thesis provides a systematic analysis of where this approach succeeds and where it breaks down in real industrial settings. It examines how the quality of the 3D reconstruction affects segmentation accuracy, how the camera setup changes the results, and how the consistency and accuracy of the object masks shape the final outcome.
- My analysis also looks at the conditions that make industrial environments difficult for AI: dark metallic surfaces, complex geometries, objects that are partly hidden from view, and objects that look very similar to one another. Each of these affects different stages of the pipeline, Alif describes.
To make comparisons concrete, the methods for improving AI performance were tested both on real industrial datasets provided by Twinverse and on standard benchmark scenes commonly used with 3D Gaussian Splatting (3DGS) approaches, a technique that reconstructs a real-world space as millions of tiny 3D shapes fitted to photographs taken from multiple angles. This allowed a direct comparison between idealized conditions and the messier reality of an operating industrial site.
The results map out which parts of the segmentation pipeline are ready for practical use and which still need work before object labelling in industrial digital twins can be reliably automated.
Connecting three AI tools into one pipeline
Alif's thesis builds on three existing methods. The first, 3DGS, produces photorealistic 3D reconstructions from regular camera images along with its camera poses. The second is FlashSplat, a research method that assigns each of these 3D shapes to a specific object using a voting mechanism across multiple camera views. The third tool is SAM2 (Meta's Segment Anything Model 2), an AI that can identify and outline objects in images.
- A central contribution of my thesis is a pipeline that connects these tools. SAM2 generates object outlines, or masks, from the images. SAM2's video tracking then keeps those masks consistent across different camera viewpoints, and the results feed into FlashSplat's 3D voting system.
The thesis also introduces an adaptive component that automatically selects the best SAM2 settings for each scene, based on characteristics such as lighting, how reflective the surfaces are, and how complex the scene is.
- The most surprising result was that giving the AI more data actually made it perform worse, not better. A small, carefully checked set of inputs produced a far cleaner result than a much larger but noisier one. It turns out the system has no way to undo a bad piece of evidence once it's factored in, so a few mistakes can outweigh a lot of extra data. It was a good reminder that in AI, quality often matters more than quantity.
- Moving from controlled research datasets to a real industrial environment was a much bigger jump. The tools I started with worked better on clean, controlled scenes, but broke in unexpected ways once I tested them on an actual industrial environment. The biggest lesson was that failure isn't a dead end if you can explain it and understanding why something breaks is often as valuable as getting it to work, Alif sums up.
Thesis details
Ofia Rahman Alif, , Master's thesis, Faculty of Science, supervisor Professor Laura Ruotsalainen, CTO, Dr. Arttu Julin.
