Software that allows users to control a computer using a range of gestures and movements with just a webcam and PC, invented by academics from UCL Computer Science, was put through its paces by visiting school students and the BBC's Paul Carter.
MotionInput was initially developed by Professor Dean Mohamedally, alongside colleagues and students from UCL Computer Science in 2020 as a means of supporting NHS frontline staff during the COVID-19 pandemic. Since then, it has gone through several generations of improvements, integrating offline and privacy safe Artificial Intelligence for PCs, with the help of industry partners including Intel, IBM and Microsoft. A spinout company from UCL Computer Science, MotionInput Games, was established in 2023.
The AI-enhanced software now includes 24 different features that allow people to interact with a computer in different ways. This includes a user operating the cursor using just their gaze, generating text by writing with a finger in mid-air, leaning in one direction to action a gaming event and hand gestures to perform a range of programmed actions, such as transcribing live text.
The latest functionality has been made possible by artificial intelligence software provided by Intel and the OpenVino toolkit. For example, for users with limb difference or amputees without hands, it can use machine learning models to recognise a user's body and evaluate their motions to correlate to certain actions.
To help test and refine the software, UCL invited BBC technology reporter Paul Carter and pupils from the Richard Cloudesley School in London to give it a go using popular games like Minecraft and Rocket League. It was also tested with Microsoft MakeCode for children learning to code.
