The eXtended Reality World of Katy Perry’s “Daisies” Performance
On her Instagram feed, Katy Perry commented on her May 17 season finale performance on ABC’s American Idol. “Started on the #AmericalIdol stage, ended up in a whole new world,” she wrote. And she wasn’t exaggerating. During the Covid-19 pandemic quarantine, Perry and her team took creativity to an entirely new place. Performing her new single, “Daisies,” with full eXtended Reality (XR) production, Perry’s video introduced a new realm of live entertainment production possibilities.
In the segment, Perry appeared to begin on the American Idol stage, which then dissolved away around her, shifting into an animated house, bringing the over nine million viewers, along with Perry, into an animated world that morphed and shifted as she interacted with virtual surroundings and elements. It was all beautifully realized as Perry’s physical presence was believably tethered into the virtual world thanks to the in-camera XR solution that literally immersed her in the video environment in real time — meaning she saw it all around her, just as the camera did, while performing live. No green screen; no visuals added post-production.
To learn more about that technical wizardry and how they achieved it, PLSN recently spoke with some of the creative and technical team behind this groundbreaking XR performance — Perry’s longtime creative director, Baz Halpin, principal of Silent House Productions; screen producer JT Rooney, of Silent Partners Studio (SPS), the firm that also created the content; lighting designer and Silent House partner Cory FitzGerald; and technical director Stefaan “Smasher” Desmedt.
From Concept to Teamwork
“We knew we wanted to do something different, cutting edge; something exciting and interesting for the performance,” explains Halpin, of the initial concept. “Katy was working at the time on an animated music video, so we discussed taking that idea and changing it into the XR idea with a house that turns into an attic that explodes away falling through a sky. I thought that it would be cool to start out on the American Idol stage and then end returning to the Idol stage. With that concept, my first call was to JT Rooney, with Silent Partners; they’re the content company for Katy’s tours. JT straddles working both with the XR process and the content side; he really knows this stuff. You have to have the person that understands how the content is built, how it reacts, how it’s all interacting. JT knows that. I could say I wanted to do this, and then JT would translate that into XR-speak.”
“Baz and I — as well as others in the industry — have been talking about the use of this technology for a long time, but we haven’t really had places to make it work,” notes Rooney. “The combination of what Katy wanted to do creatively, as well the existing world of Covid, led to this being a good opportunity. Katy had already created an animated lyric music video for “Daisies,” and she really wanted to play off that concept of that cartoon-like world. So we took that and developed it into an actual workable XR world that she could enter and perform in live.”
Halpin and JT also called in Stefaan “Smasher” Desmedt, who has been working with and developing the XR process over the past few years. “We got with Smasher because he obviously has been pioneering this for the longest time. Smasher’s the legend, as he’s so knowledgeable with all this,” comments Halpin. “We also brought in XR Studios, who produced it. They have really been perfecting the XR workflow for some time, and it is so important in this process to have an XR producer. There’s a lot that goes into doing something like this. There has to be that middleman, that producer role, which XR Studios does. They sit between the creative director, the content creator, the gear and the vendors, and bring all those things together to execute the end result.” For his part, Smasher was pleased to be part of this particular team. “Myself, Scott Millar, XR Studios, Notch and disguise, we all have been working on this workflow for a while now,” says Smasher. “I have always believed there’s something special here. And I am so happy we did this first big complete job with Baz, JT and Cory.”
The team leveraged the latest technology using disguise servers, Notch VFX paired with high resolution ROE LED screens to make the XR video work along with real lighting to augment the digital lighting in the content. The video was shot at PRG’s Los Angeles XR studio. PRG also provided much of the production equipment. The production time on this project totaled about three and a half weeks for a three-minute performance. Working in the studio for three days perfecting the setup, angles, etc. with a stand-in. Shooting with Perry herself took just under three hours.
Navigating the Virtual World
They were able to save time because the creative process with Perry had already begun while working on the animated music video. “We were really fortunate, because a lot of time had already been spent on the creative world, building the music video — without which it would have added at least two or three weeks onto our timeline,” explains Rooney. “We were able to work with that team and went through the traditional pipeline of content — animatics and style frames, essentially storyboarding. SPS created the assets, content and animation for the XR shoot. The next challenge was getting this all into a Notch VFX, so that it could run and operate in real time; so when the cameras moved and were tracked we could render content, from different perspectives, at every moment. This is a challenge in that you’re always managing real time performance, but Notch’s performance monitoring tools and familiar workflow in a production environment allowed us to land in a great place.”
In the world of XR, content creation needs to be thought of in new ways, like the idea that you can move and explore different perspectives in the XR-created world; it’s not just content on a flat LED wall. “From a creative perspective, the actual look and feel of the piece was a deceptively difficult effect to achieve,” describes Rooney. “The graphics’ solid flat colors and simple textures actually made it difficult to complete the illusion of making the physical LED [backdrops] disappear and magically combine with a virtual environment. Also, we need to create a world that’s ever-changing; you can view it from different angles. There were angles shot in the final version that we weren’t planning on shooting originally, but it looked so much better once we discovered them on camera. From a design standpoint, you really need to think in a more holistic way in terms of what you’re creating. You have to be smart about what kind of textures to use or how the 3D content is being designed in a way that makes it all work, but still has a creative aesthetic and still is telling a story. There are a lot of technical design challenges.”
Combining Technologies
For the XR shoot, there were two main LED walls as backdrops of ROE Black Onyx BO2 2mm LED modules, and one LED floor consisting of ROE Black Marble BM4 4mm LED modules. In conjunction, the technical team chose to use Brompton Technology processing for this project. “For the LED from ROE, we used Brompton’s 4K SX40 LED processors,” states Smasher. “It was the obvious option because they are fail-proof and I knew how we can get what we needed out of them.”
Rooney comments on the software, which “connects all those together in the virtual space and then actually extends the world past those LED screens. That’s the beauty of the disguise xR workflow that we’re using. It’s a virtual set extension to kind of continue the world off the LED screens. disguise have been a huge part of this workflow for the past couple of years, and they’ve really just been pushing it.” Scott Millar, a disguise xR Workflow Specialist, who provided creative technology support and was also credited as technical director along with Smasher, adds that “the seamless extension of the real-world LED screens to the virtual world environments could only be done by disguise’s xR camera registration workflow, allowing switching between camera perspectives and the LED content Multi-Camera switching between perspectives also allowed us to switch cameras and the LED content.”
To track the cameras in the 3D space so they could feed their positions into the disguise servers, the team relied on stYpe’s RedSpy Optical Tracking Systems. “stYpe, a company out of Croatia, has created a camera tracking system that mounts on top of a camera,” notes Smasher. “That tells the system where the position of the camera is in space. You define a zero point and then the stYpe system can say to the disguise servers, ‘Okay, I’m over here now with this camera pointing in this direction.’”
Bridging Two Worlds with Lighting
Smasher describes the lighting as “equally important as the content. You can have the most gorgeous 3D scene, but if it’s not lit well, it won’t work. You need the right lighting designer who really understands the subtleties needed for the XR workflow.”
“Lighting is absolutely critical,” agrees Halpin. “That is why we had Cory FitzGerald as the lighting designer. He really understands both the lighting in the physical world, but also the lighting in the virtual world. And that’s an integral part to make this all worked, I didn’t want this to look like film; I wanted this to look like a live performance. I wanted this to look like she was on the American Idol stage. Cory and XR made it look like it was real, like it was live.”
FitzGerald describes some of his process for the shoot. “It was mostly about key light. In this type of environment — especially this performance — it was 99 percent about making sure she was lit — knowing where she was going to be, and how the lighting affected the floors and the environment around her. So, there was a lot of balancing, chasing of the key light and the levels. A challenge was that studio was very small in comparison to a giant stage where you have a lot of distance between the fixtures and the artist. Here the lights were probably 10, 12-feet away; maybe 20-feet at the farthest. If she moved four feet downstage, she’d double the foot-candles. Ultimately, it was about trying to find balance and all the little subtleties and cues. That kept the levels fluctuating five percent, 10 percent between upstage and downstage. Just that small little percentage made a big difference on the foot-candles hitting her.”
Working with an in-camera XR workflow, FitzGerald could see his lighting — and changes — in real time. He could keep the balance going as they went along. “Because, again, five percent’s too much — a little bit too hot — and her face blows out; or the background, the shadow becomes super dominant, and you see that she’s standing in the middle of nowhere. You’d have to really be aware of the whole picture — everything that was going on, all at the same time. It’s definitely a new environment to try to express lighting in, because you have a lot of constraints. You have to keep everything right where it needs to be, so the lighting makes sense. That’s the balance. It’s about creating a world together between the real and the virtual; being that lighting bridge. It requires a lot of coordination with everyone; knowing where things are going to be, what it wants to look like, and then having the right tools, in the right places, to give you options from different fixtures and different locations.”
For this shoot, FitzGerald used VL2600s as the main back and key light and filtered around the sides. He also had some GLP X4 washes and a series of ARRI Sky Panels in a more traditional studio setup.
More than a New Brush
“It’s a whole new paint box,” states Halpin of working in XR. “You can create anything. We’ve been restricted for years by the physics of the real world. You want a 300-foot-tall elephant to balance on a pin with beams of light from the tusks and trunk? Well, you can’t do it in the real world, but in XR, you can make that thing look incredible. It’s an incredible new paint brush with a whole new palette from which we can create a myriad of things.” Rooney agrees. “This is just the beginning creatively of what we can do — creating whole worlds to explore.”
Images courtesy PRG and disguise
