"Avatar," the highest-grossing film of all time, took viewers to a new world, Pandora, and it advanced filmmaking to its own new world: developing the field of virtual production.
Leveraging a wide range of technologies such as performance capture, LED virtual environments, and advanced 3D imaging technologies, virtual production is changing the landscape of modern cinema. While millions of people have seen "Avatar," only a fraction of that number understand the magic behind the scenes. Exposing filmmaking students to this magic is what MIT Media Lab alumnus Daniel Pillis SM '24 is all about.
"Motion capture, like that in 'Avatar,' bridges real human movement with digital technology," says Pillis. "In this digital age, and as artificial intelligence becomes more involved in film studios, technology that enables the authenticity of human expression and performance is becoming increasingly important."
That is what Pillis, now an assistant professor at Emerson College, teaches his students in his filmmaking courses. To bring the lesson to life, each semester the class travels across the river to MIT, where Emerson undergraduate and graduate students use the capabilities of the MIT.nano Immersion Lab to create their own virtual productions.
Donning full-body motion-capture suits that pair to the 28-camera OptiTrack system in the Immersion Lab, the students become their own avatars - generating virtual characters that dance, fight, or play the guitar like The Beatles. They see their animation data immediately on a computer screen and can change or add to their character's movements in real time. Later, they take their data back to Emerson to build into short films for their final projects.
"It has been truly gratifying to support this course and to see the curiosity and ingenuity students have brought to the stage," says Talis Reks, who manages the MIT.nano Immersion Lab. "This class highlights the range of what our lab can offer, extending well beyond research and into art and the performing arts."
The MIT.nano Immersion Lab - there's really nothing else like it
Pillis first learned about the MIT.nano Immersion Lab during his time as a graduate student in Professor Hiroshi Ishii's Tangible Media group at the MIT Media Lab. Working with colleague Georine Pierre SM '24, the two collaborated on a Haitian folklore dance project , creating a motion capture-driven simulation of Haitian folkloric dance traditions, specifically the sacred Yanvalou dance. They built a living archive using the capabilities of the Immersion Lab that let participants dance with an interactive AI-driven ancestral avatar animation.
When he became faculty at Emerson, Pillis knew the Immersion Lab was a perfect fit to elevate his students' experiences. "The level of high-end film production that the Immersion Lab supports is out of reach for so many students who would benefit from this technology in their practice," explains Pillis. "The facility is unique, well-equipped, and even accessible to those outside of MIT - there really is nothing else like it in the Boston area."
With the type of mechanical character animation the Immersion Lab technology allows, the final projects end up light-years beyond what these students thought they could achieve, continues Pillis. And they're having fun. "They really get into it," says Reks. "These students are not necessarily trained as actors, but the moment they see themselves as virtual characters, the realistic, granular movement enabled by motion capture, they get fully into performing."
Rewarding professionalism
In the past two years, over 60 Emerson College students have used the Immersion Lab for Pillis' class. Emerson undergraduate student Nick Forsch received an EVVY Award nomination for his project. The Emerson version of an Emmy, EVVYs are awarded to students whose projects are judged and selected by a panel of industry experts looking for creativity, quality, and professionalism.
"Being able to use the MIT.nano Immersion Lab really elevated my project," says Forsch who created "Enter," a short film about a human transported into a digital world to meet an artificial intelligence. "I was excited to submit it for an EVVY, knowing the technology behind my work was on a professional level."
Another undergraduate student, Evan Costa, recently created a virtual recreation of The Beatles on "The Ed Sullivan Show," capturing a version of each musician's performance and reconstructing a simulation of 1950s television. Costa will be joining the MIT Learning Engineering and Practice Group, led by principal research scientist John Liu in the Department of Mechanical Engineering, this summer to continue exploring virtual production as an intern.
"Having the opportunity to gather motion-capture data within the Immersion Lab gave me more than advanced technology for my project; it provided insight into an often-unseen world of creativity," says Costa. "Modern storytelling exists across a wide range of mediums, from film to video games, and witnessing the inner workings of this process has deepened my passion for virtual production."
In the coming academic year, Pillis and Reks plan to leverage advanced Immersion Lab technologies to teach facial animation, hand and finger tracking, multi-modal data capture, and further advances in interactive generative motion capture as they gear up for the next set of productions.
