With enormous eyes, a huge mouth and defined, prominent eyebrows, an avatar in the form of a ghost using sign language in real time takes the stage. During 2022, Riksteatern Crea - one of the world's leading sign language theatres - and researchers at Lund University Humanities Lab trialled new innovative and creative solutions in order to make the technology work for this multi-dimensional theatre experience.
We are sitting in the Humanities Lab studio in Lund. On computer screens, research engineer Stefan Lindgren is showing us what it takes to bring an avatar to life. The technique is called motion capture, abbreviated to mocap, and it records and stores movements on a computer. Gaming and movies have long worked on making characters' movements realistic. ABBA made their comeback using the technology, while the film "Avatar," released a decade ago, features astonishing scenes.
Streaming in real time
The basic idea is to create a three-dimensional recording - small round markers are attached to different points on a person's body. These markers are recorded by a large number of mocap cameras, which are placed around the person. When the person moves, the system's computers record their movements in three dimensions. Special software transfers the movements from the person to the corresponding points on a computer-generated body. Until now, the Humanities Lab has never streamed the results in real time, instead they have worked with large volumes of data afterwards.
"We help many different kinds of researchers, such as archaeologists, linguists and behavioural scientists, and we have long experience of making recordings. But when Riksteatern Crea wanted to research whether signing avatars could be live-streamed, I thought blimey - that is a seriously exciting challenge," Stefan Lindgren recalls.
Small details make the difference
Riksteatern Crea is the leading Swedish professional theatre for deaf people - actors and audience alike. Along with Humanities Lab's mocap experts, Artistic Director Mindy Drapsa, the director and actors started testing and working in the mocap studio. One of the challenges is that signers express language through bodily gestures, while grammatical detail is conveyed by the face. How could we get at those tiny details in the pattern of movement and simultaneously stream in real time?
Placing markers on the face did not work, the nuances were difficult to capture. Eventually, the researchers found an app that uses a smart phone to film tiny facial movements. Along with gloves, which the researchers were able to calibrate better, as well as the filming from the mocap cameras, it was possible to use three systems to register all the movements at the same time and stream them to the virtual figure. The results were surprisingly good.
"Everything rests on the system being able to see the signing actor and how well the virtual character can interact. If a little marker falls off the actor, chaos ensues. That might mean that one leg will not move, which looks very strange. Taking the leap and daring to do this live, something that has never been done before, is brave," Stefan Lundgren says.
Riksteatern Crea's technological know-how also gives a lot of opportunities for the avatar to be projected onto any part of the stage, to fly in from the wings or appear behind the actor's shoulder.
Globally unique projects
During the autumn, this unique concept has been tested on the audience for the play "Spöket på Canterville" (The Canterville Ghost), and received a positive response. In 2025, the theatre troupe will embark on a nationwide tour. The signing actor does not even need to travel - the technology can all be taken care of remotely. But how will the public be made to realise that it is live?
"The play might start with the ghost approaching the audience, talking to them in sign language and engaging in conversation. The avatar is not simply an add-on, it actually brings a lot to the scenography," Stefan Lindgren explains.
After a long working process, Artistic Director Mindy Drapsa feels very positive about the development of Riksteatern Crea's ambitions in this globally unique project.
"It will be incredibly exciting to see a signing avatar on the stage. The audience we tested it on were overwhelmed at experiencing an expressive avatar using sign language in real time. That is an experience that you cannot describe."
