Augmented and Virtual Reality Meet in Real Time
Montreal-based Luz Studios provides lighting, set design and video content services worldwide across the broad spectrum of performing arts. For the past eight years, they have provided the visual design for the Gémeaux awards gala honoring Canadian TV shows and digital media broadcast in French. The gala is sponsored by the Academy of Canadian Cinema & Television, which had also sponsored the Gemini Awards for English-language productions. (While the Gemini and Genie awards galas were combined into the Canadian Screen Awards in 2012, the Prix Gémeaux continues on as the Canadian Francophone equivalent of the Emmy Awards in the U.S.)
In the past, the awards show has typically been broadcast from a variety of public theatrical locations such as Montreal’s Théâtre St-Denis and Place Des Arts. The 2020 edition was different, necessitating a move into a TV studio to allow for augmented reality hardware and to have an open place to redefine the audience space. A regular theater, with seats and proper social distancing, would look empty. “Due to the global pandemic, we had to rethink the way we approach the gala, from the choice of venue, set design and the way we reveal winners,” says Luz Studios’ AR video director David Pawsey.
The process involved creating augmented reality graphics that united nominees on stage. A grandMA lighting console was used to control both the virtual lights lighting the AR graphics while also controlling the physical lights in the studio in real time. The design team also “gathered” multiple artists virtually on the stage with the live nominees, again lit with the combination of real and virtual lighting.
The driving factor behind this approach was two-fold in that it provided a virtual dynamic scenography. Instead of making all the graphic elements actual physical set pieces, Luz Studios created virtual set pieces that were moving and dynamic and key to the production. Beyond that, they had to a requirement of keeping all the nominees at a safe social distance.
“We chose this innovative approach because it required no rendering time,” notes Pawsey. “This way, the additional virtual elements and people were always coherent with the physical set we designed. Additionally, we wanted to stay away from the zoom-like presentations and generate a feeling of closeness.”
Real-world reflections were added to both real people and virtual graphics to create a homogenous look and feel for the viewing audience. The actual studio lights and the reflections on the virtual pieces matched, and that was one of the keys to making the magic happen. “We also gathered (virtually) multiple artists together to create a unique moment for the opening number,” says Pawsey.
Lighting
Luz Studios’ production and lighting designer Matthieu Larivée points out that the creative team used Unreal Engine software architecture in their workflow. “In the Unreal software, we were able to customize a lighting fixture based on the real fixture we had on site. We created a virtual light with RGB, a zoom, a pattern, and then patched it via Art-Net through the lighting console. It was assigned a unit number, regrouped, and patched with the actual fixture.”
Before Luz encapsulated this innovative approach, usually, the AR format followed was that of content people creating their images separately for the lighting designer to create looks after the fact. The obvious layering is evident and has somewhat of a Photoshop feel to it, Larivée believes. “This unnecessarily creates two different camps. We have integrated these two disciplines and formed a new level of visual cohesion.”
For the union operators on the grandMA console, it was a walk in the park. They merely had to patch 24 fixtures, and the Unreal software took care of the rest of the math. When the operator grabs a fixture or creates a cue, both the real and the virtual light simultaneously appear. This way, the entire scene is very cohesive and, for the sake of broadcast, the light levels were all consistent. “The depth of perception achieved, because one cannot discern what is real and what is not, was truly a mind-blowing revelation once the final visual was viewed,” says Larivée. “That is the real magic of what we have accomplished here.”
A combination of virtual artists and live nominees appear on the stage for the opening number of the ceremony. The combined palette of real and 3D lighting was used on the actual nominees and holograms of the artists that appeared virtually on the stage. Previously, in an isolated studio, each of the virtual artists introduced to the award ceremony were captured via a flatly-lit, no-color frame.
“Even, flat lighting gave us the latitude to apply virtual light to the virtual subjects,” says Larivée. “If we were to get too artsy with lighting the original subjects remotely, then we would be stuck with those specifics. It would create complications further down the line when we relight them in 3D. Not all the original parameters may translate. To the audience’s eye, something just doesn’t look quite right. By lighting them originally this way, we have greater range in tweaking the 3D light in the broadcast scene.”
“To do that in real time is what really is the impressive part,” adds Pawsey. “That is really a huge breakthrough. We integrated that footage and treated it in such a way the image could be read in the Unreal engine.” As each nominee was introduced, one by one, “the virtual light worked so seamlessly with the actual light that any sense of layering or separation is eliminated.”
The Set, with Virtual Extensions
A combination of an actual stage and virtual set pieces helped heighten the emotional anticipation of the show and gave rhythm to the multiple sequences occurring throughout the night. The main stage was a circular disc capped with reflective material similar to the Harlequin Hi-Shine flooring utilized in television studios. Hidden behind the stage was an actual riser that rose up to reveal the virtual artists appearing. The actual upstage wall was a 50-foot curved LED wall. Downstage, separating the audience from the performance area, there were two smaller discs that mimicked the main stage, serving as an area for guest host introductions. Thirteen Solotech SLPro8 LED tiles created the façade for the two-foot-high main stage.
A subtle yet impressive touch was the ability to present the reflections on each of the glossy stage floors with both the physically present participants and the virtual attendees. “Within the software, we were able to do a pixel count of what was happening with the live feed,” says Pawsey. With that input, they were then able to calculate within the tracked camera how a virtual object reacts to a live environment, thereby creating 3D objects that mirror the pixel track of the background image.
“It’s pretty straightforward actually,” continues Pawsey. “Even though it sounds complicated, the math all is done with in the software. Because we were able to track cameras and analyze the reflections of the host on the stage material, we could imitate that surface in 3D and composite in real time all of the virtual actors. The software gave us the tool to isolate their reflections and input that information onto the stage floor.”
Other Virtual Set Elements
A virtual trophy symbol, representing the award given, added grandeur to the evening, and it served as a set extension piece. The large element fades in, dominating the screen at the top of the show. Continuing the innovative use of virtual reflection, the 3D wide shot replicates that standard awards show “long shot, over the shoulder, pulling away from the stage kind of look,” says Pawsey.
Appearing throughout the show, it facilitated lead-ins and lead outs, segment breaks and transitions. “We magnified the trophy with its metallic texture,” adds Pawsey. Here again, the set lights were coherent with the reflections on the virtual set piece. The trophy would appear and disappear from the stage floor, which enhanced the minimal stage design. “We accomplished the image by engineering a jib camera,” Pawsey notes. “Created to bring production value, the golden symbol makes a bold emotional statement without being too ‘over the top.’”
A virtual “fence,” as the Luz team refers to it, served as an essential set extension piece. As they would in a normal gala setting, the host named the category, and the nominees walked on stage. Being six feet apart, a virtual metallic fence covered them. Cameras traveled on nominees as they were named, and the virtual fence opened to reveal the nominee with a presentation video insertion next to them. Here again, the reflections on the metallic elements worked with the actual lighting design.
This element helped create a rhythm to the introductions of the five nominees for each awards category. By highlighting them separately, it created an individual moment for each of them. It also masked the fact that the five nominees were standing on an empty stage while creating a reveal that kept them safely six feet apart.
These virtual stage elements were created by Luz Studios art designer Emily Fortier who came up with the 3D elements in a way that complemented the show and was consistent with the set design and overall branding of the event. “It was a standard design approach, just done in a completely different medium,” says Pawsey.
The Show Goes Live
The broadcast event was cue-called and ran live, without any timecode whatsoever. Upstage, a 50-by-10-foot (WxH) semi-circular LED screen served as both a backdrop and surface on which to display the prerecorded nominee scenes, using 300 Absen 2.9mm tiles. These visuals were inserted into a video window in AR from the live input of the VTR.
“Being able to pipe all these 3D animations into show control software gave our show caller (TD) the ability to run the show with a cue list much as she would call lighting cues,” says Pawsey. “That was the cool part of how we integrated the workflow we had into the 3D environment.”
There was a live, socially distanced audience of about 50 people in the studio. Their tables were pin spotted from above. This way, viewers at home had enough ambient light to realize there was an audience, without clearly seeing how sparse it had to be. “You feel the people but don’t actually see them, adding to the architecture of the overall look,” says Larivée.
Additionally, the Luz team came up with a way to do a live broadcast feed in the studio, but just off camera and out of sight. This way, the attendees in the room were able to get an idea of the virtual visual effects that were happening on stage and enjoy the experience even more. The nominees were able to view the feed as well, which helped them better understand their role in the live show.
Says Pawsey, “This is an exciting time to stay with our identity — integrating our shows, keep creating lighting, creating moments. Now we have an extra layer to work with. I think we are looking forward to all the opportunities that are coming our way.”
Adds Larivée, “This is a technology that kind of takes reality and extends it. I believe this will continue to grow and be used for the future, not because of the pandemic, but because of the real emotion that happens and the way it brings people together. This is not only for the moment we are in but will provide a better way to analyze and extend shows.”
