We have passed the dystopian 1984. We passed the space odyssey 2001. We are into 2020, and the future is officially now. Our voracious audiences have devoured every gag that we can physically throw at them. They appreciated the drum kit roller coaster. They allowed us to fly drones over their heads. They tolerated the laser beams in their face. They even briefly cheered when we drenched them in Faygo and then coated them with confetti. The bravest ones even came back asking for more. We have reached the outer limits of the live performance universe, and we still need to go further. One viable next step is to take them into the virtual world with us. We can take the audience beyond the limitations of time and physics and deliver them into a digital panacea. Please take a few moments to go on a journey through time and space with myself and our guide, screens producer J.T. Rooney of Silent Partners Studio.
What is Augmented Reality?
Augmented Reality (AR) includes anything that alters, changes, or affects your interpretation of what is happening in front of your eyes. In its simplest forms, it’s the yellow first down line marker you see on live broadcast Monday Night Football. In its more complex manifestation, it’s the dragon that flies into the stadium and sets the scoreboard on fire. AR is any enhanced image or environment as viewed on a display, produced by overlaying computer-generated images on a real-world environment. It is hard to put the boring line marker and the magical dragon in the same category, but neither one exists without an augmented reality. AR is a loose term but, as J.T. says, we are stuck with the name.
Can We Use AR in Live Shows?
AR has been incorporated into a few shows over the past few years with varying levels of success. AR was placed front and center for the 2019 VMA Awards when an entire atmosphere complete with dancing suns, bouncing clouds and shooting rockets was created for J. Balvin and Bad Bunny’s performance of Que Pretendes (see plsn.me/JB-BB). NBC Universal’s coverage of the 2016 Election Night used AR to turn Rockefeller Center into a live scoreboard. AR has conversely been used in much more functional ways when the Weather Channel’s coverage of 2018 hurricane damages included a virtual nine-foot tall wall of water to showcase potential flood damage. AR has also been used to enhance I-Mag or live video applications on concert tours in various forms. “The nice thing about AR being fairly broad at the moment and in its early days is that there’s a lot of opportunity to experiment and try new things,” says J.T. Since the VMAs were already set up to use AR in the performances, a decision was also made to use AR more functionally. AR was used to highlight award category announcements, floating lyrics, virtual set extensions and virtual scenery. Possible Productions designed the show award category announcements, set extensions and some of the performing acts while additional artist performances were designed by Silent Partners Studio, Blink.tv, and others. The AR Engine technical implementation was done by Gravity.
Should We Use AR in Live Shows?
AR started as a tool of necessity. It is cheaper and easier to generate thousands of set pieces in digital space than it is to build them in the real world. Shortly after, AR became a tool of innovation. If five dancers are good, then 500 dancers are better. Placing 500 dancers on a stage is easier in the virtual realm. J.T. was lucky enough to have worked on a recent project where AR became necessary. “There have been few moments that have legitimately needed a tool like AR to tell the story or get the point across,” J.T. noted. AR, however, is a technology that requires a reason. Using a tool like AR just to show off your technological prowess usually falls flat. Without a purpose, AR is just a distraction.
When Should We Use AR in Live Shows?
Whenever a new tool is successfully introduced into the industry, we all scramble to use it in the most creative way. Making a virtual jet land on the stage in a small theater setting seems really easy until the accountant has to approve the cost and the technical director has to approve the logistics. J.T. wants you to know that “the surprising thing to many people is how many different departments and disciplines have to be involved in order to successfully and tastefully integrate AR graphics.” Collaboration is required from the camera team, broadcast team, TD, television directors, creative designers, content teams, animators and artists. Even after the creative process is complete, there is still a whole new level of coordination involved with the tracking of cameras, running game engines, real-time content engines live and sorting out the limitations of new technologies. If AR is to be done properly, it needs to be planned and discussed in a great detail. These are not decisions that can be made quickly or easily. Successful AR requires ample forethought and full cooperation.
Does AR Distract from the Artist?
J.T. responded, “This is an interesting conversation, as always,” before pausing and pondering the point. Every new gag, technique or tool in live performance needs to have a reason and a purpose. Projection mapping was once considered a side gag or as a distraction. When it became affordable, it became popular. I couldn’t scroll YouTube without seeing a projected building crumbling, filling with water or turning into an art piece. “Over time, it has developed and found its way into some very meaningful uses and integrations,” J.T. says. “Of course, there are still moments where it might be overkill or distracting.” The same goes for AR. “There are some moments where I’ve had conversations or pitches about potential projects wanting to use AR, and then as a group, realizing and discussing a similar effect could be better for the audience using another solution.” J.T. gets more excited about the times when he presents an idea that is both necessary and possible. “However, there have definitely been a few ‘eureka’ moments with a new set of possibilities with AR that have led to good results,” he says.
What’s Next for AR?
As AR gets cheaper and more streamlined, I predict that AR will get used to create entire touring atmospheres. I can envision a bare arena with a stage, P.A. and the audience. Every person has AR lenses or a tablet. The AR could be individualized to each attendee’s preferences. Even while listening to the same performer one person would be interacting with a virtual forest of flying fairies and dancing clouds while another guest would be enjoying an underwater oasis with yellow squids and playful purple dolphins. I can also imagine a concert where we can virtually place ourselves on stage with our favorite artists. Using 360° cameras, we could transmit those images live to AR devices that allow hundreds of fans to roam the stage during the show without interfering with the performance. Hologram tours of deceased artists are already becoming a reality. With AR, the only limit is imagination and budget.
