Make yourself comfortable at home on the sofa and immerse yourself in a symphony orchestra's magical take on Beethoven´s Fifth Symphony as if you were there inside the concert hall. Change camera angles, zoom, cuts, sound quality and access background information about the piece being played. Or chat with fellow members of the digital audience.
"The rapid digitalisation of the performing arts has opened up new worlds in terms of how the public can access culture, worlds where we need to apply new thinking so that consumers become active instead of passive."
So says Jesper Larsson, expertly guiding us around the Malmö Live concert hall, which with its absolutely world class acoustics is home to the Malmö Symphony Orchestra. In this project, the Malmö Live concert hall will serve as an experimental environment in the form of a living lab.
Lund University's Faculty of Fine and Performing Arts, along with the Faculty of Engineering (LTH), the Faculty of Law, the School of Economics and Management and a number of external partners such as AXIS and Amazon, have been awarded SEK three million over three years to fund their efforts to develop new methods of producing the arts and making them accessible. This thematic collaboration initiative is led by Sanimir Resić, dean of the Faculty of Fine and Performing Arts, and coordinated by Jesper Larsson, who has a solid background in the cultural sector through his past management roles at Malmö Live and Stockholm City Theatre and the City Arts Centre.
A digital universe
The planned streaming service will allow you to choose which camera angles or cuts you prefer, to zoom in on an instrument of your choice or even adjust the volume of an instrument in the mix. If you are curious about someone else's version of the concert, you can access that too. Using facial recognition technology, you will receive information about the people on screen and you will also be able to chat with other visitors and ask questions of the performers before the concert and during the interval.
The digital format also opens the door to new visual possibilities.
"When we move the concert experience from the geographical location to a digital universe, we are not limited to how things look inside the concert hall. We might experience the concert as a hike in the Alps or through a psychedelic AI-generated landscape we can lose ourselves in. Or why not from inside a clarinet?"
The aim of this thematic collaboration initiative is to reach new groups that do not usually enter the concert hall at Malmö Live.
"We would be able to connect schools and libraries and create a good platform for children and young people. We would also like to reach out into nursing homes and prisons. While the project has an upside, we do not intend to ruin or change the format for those who already love it."
Individually tailored live broadcast
During the pandemic, Malmö Live used cameras supplied by AXIS communication, an industry leader in network cameras, in order to stream their concerts to the audience. This was a positive experience, and its impact has now grown. The ultimate solution for individually-tailored live broadcasts puts great demands on new digital solutions, including artificial intelligence, the development of new business models and legal questions about copyright and the right to privacy. Making this happen - giving the audience at home an experience that is at least as fantastic as the one enjoyed by the audience in the hall - will require the entire span of Lund University's expertise.
How do we zoom in sound?
"The basic idea is that I, as a listener, choose a seat in the concert hall. Today, we have the technology in place, and lots of solutions for how to zoom within images digitally. New technology is needed though, for individually selected digital zooming of the sound," explains Jesper Larsson.
Researchers at LTH want to investigate a more automated way of handling production that does not require so many technicians, visual producers and sound technicians. The answer is to engage AI.
"We want AI to read the sheet music, to know where the musicians are seated and where the cameras are placed. Let's say I want the trombone to sound louder in a certain part - how do we zoom in sound? We anticipate a streaming service in which the computer will be able to visually zoom in on the harp in relation to the trumpet according to how the sheet music is written, as well as being able to zoom in on sound," says Jesper Larsson.
There are many technical challenges to overcome, so researchers and teaching staff from LTH are involved.
"This project gives me a 'We choose to go to the moon' feeling," says Erik Larsson, professor at the Integrated Electronic Systems unit, who has already begun involving students studying for the Bachelor of Science in Computer Science and Engineering and Electrical Engineering with Automation.
This autumn, students will be working on various projects.
"I hope this can lead to exciting degree projects for the engineering students in the spring, and that we will also be able to get other faculties to see the potential in this exciting project.", says Erik Larsson.
New forms, new demands
Digitalisation brings many new demands, not least when it comes to copyright. How can legislation keep up with the technical development our society is undergoing when at the same time, all music is covered by copyright? Those whose livelihood is their music must be able to get paid.
"It is an enormous question, especially if we think about a global market. In this landscape, there will be an awful lot of conflicts, discussions and concerns. But consider how Spotify has revolutionised musical life. This is where we bring in researchers from the Faculty of Law, who will dig deeper into this complex issue."
When every concert is streamed, and when cameras are more numerous and close-up than earlier TV productions, it puts extra pressure on the musicians. How does it affect the musicians if cameras can zoom in at any time, not just when they are playing.
"Over the past 15 years, we have had great personalised opportunities to watch and film athletes' performances. Spectators in the stadium have also been exposed. The question of privacy is a research field all of its own. When it comes to musicians, I can imagine that sensitive situations may be recorded and distributed, something that could intensify stage fright. We have to bear in mind that if one goes down the path of being a musician, it is not possible to avoid exposure. Unfortunately this could entail a breach of the right to privacy," says Jesper Larsson.
Tighter rein on finances
The initiative also wants to examine a new kind of payment model more closely, and is considering a crypto-currency that means more money ending up in the pockets of the creators.
"Under the normal payment systems, an incredible amount of money leaks out into banking organisations. We want more money to end up with us, the ones streaming the concert. It should be possible to keep a tighter rein on finances," Jesper Larsson predicts.
Jesper Larsson and researchers from the School of Economics and Management envisage a global system, one that attracts members from all over the world and where the consumer could buy a streamed concert from the Berlin or Los Angeles Philharmonic, or the Malmö Symphony Orchestra. There are no limits to the experiences that could be offered and made global.
