How do you enable a robot to harvest tomatoes in an environment that is constantly changing? Wageningen University & Research (WUR) is exploring how simulation can accelerate and refine the development of harvesting robots for greenhouse horticulture. Researchers are building a simulated greenhouse environment in which robots and tomato plants interact realistically.
Testing robots in greenhouse horticulture is complex in practice. Plants grow, conditions shift, and each harvest permanently alters the situation. As a result, tests are difficult to repeat under identical conditions. WUR is therefore developing a simulation environment that digitally replicates this dynamic setting, allowing robot design and control systems to be tested in a systematic and repeatable way.
"Robot simulation is quite common in controlled environments such as car factories, but in greenhouse horticulture this is genuinely new," says Arjan Vroegop, researcher at WUR and project leader within the Vision+Robotics team. "Here, you are dealing with living plants and significant natural variation. That makes simulation much more complex."
"Robot simulation is quite common in controlled environments such as car factories, but in greenhouse horticulture this is genuinely new."
- Arjan Vroegop
- Project leader within the Vision+Robotics team
From plant science to robot simulation
At the heart of the research is the integration of expertise from across WUR. Within the Digital Twin project, specialists in robotics, simulation, crop physiology and 3D modelling collaborate closely. The aim is not only to simulate robot movements, but also the crop itself, including its structure, variation and physical properties.
"In earlier research projects, we worked on crop modelling and growth simulation," Vroegop explains. "But a realistic visual representation was not essential at that stage. In this project it is, because the robot must be able to detect where fruits are located and how it physically interacts with them."
The plant models are developed by researchers in crop physiology. WUR researcher Maarten van der Meer is responsible for the 3D modelling of tomato plants. "We measure real plants and reconstruct them virtually using fixed measurement protocols," he says. "By allowing variation within measured boundaries, such as plant height, leaf orientation and fruit position, we can generate a wide range of realistic plant models."
According to Van der Meer, this variation is crucial. "A robot must also be able to cope with less-than-ideal situations, where fruits are difficult to reach. We want to be able to test those scenarios explicitly within the simulation.
Application in robot development
The simulation is being used in a joint project with technology company DENSO and its subsidiary Certhon, which are developing a tomato-harvesting robot. Within the simulated greenhouse environment, the interaction between robot and tomato plant can be tested step by step under identical conditions. This enables design adjustments and software updates to be compared systematically.
In a real greenhouse, this is hardly possible because plants continue to grow and every harvest permanently changes the setting. By modelling that variation digitally, development cycles can be completed more quickly than when relying solely on physical trials.
State of the research
The collaboration began in May 2024. Since then, a functioning simulation environment has been established in which both the robot and the tomato crop are modelled. The environment also generates synthetic data, such as semantic and instance segmentations, which are used to train detection models.
In the current phase of the project, the focus is on further automating robot testing and refining the interaction between robot and plant. This allows different development choices to be evaluated more quickly and consistently.
WUR
In addition to a realistic visual environment, the simulation also provides semantic and instance segmentations. These are used to train detection models.
A new approach for greenhouse horticulture
According to Vroegop, the project demonstrates how simulation can open up new possibilities for research in greenhouse horticulture. "In sectors with stable environments, simulation has long been an important tool. This project explores how that approach can be applied in a domain characterised by biological variation and continuous change."
