Tribeca Film Festival Scores Hit with Premiere of Baobab Studios’ VR Experience, Jack: Part One
Virtual reality experiences are becoming more and more popular, ranging from dinner parties to superhero scenarios. While such entertainment mimics real life actions and events, French film director Mathias Chelebourg wants to take Virtual Reality (VR) another step further by integrating animation with immersive theater. His latest project, Jack: Part One, did just that when it debuted at this year’s Tribeca Film Festival. Created in collaboration with Emmy Award-winning Baobab Studios, it allows participants to become the titular character from the famous fairy tale Jack and the Beanstalk, albeit with some modern Sci-Fi/Fantasy touches.
How It Works
The setup is simple. You play the naive Jack, whose mother asks him to trade a radio for money, but instead he gets a giant magic bean that ultimately sprouts into a huge beanstalk that pushes his house high into the sky. It’s a variation on the original story where the radio is actually a cow and multiple magical beans create the beanstalk that he then climbs up.
Unlike other VR experiences, Jack: Part One has a live actor who plays two different parts and also hands you props. “At some point, it struck me that the best kind of interactivity you’ve ever going to get is human interaction,” says Chelebourg. “So for me, VR is when space comes inside the film. It’s like bringing space back into traditional movies, and that was a challenge.”
Chelebourg believes that their approach is so unique that “it might very well be the first time a VR project has attempted such a complete and uncompromising leap forward in immersive storytelling,” he declares. “In Jack, every sensory input is designed to help the viewer believe he or she is literally inside this animated world. This world is not only virtual, but also fully physical – you can walk in it, touch it, and interact with it with every one of your senses. It is augmented in various ways with touch, smells, wind, vibrations, and more. But most importantly to me and Baobab is that all the interaction and innovation designed in this setup comes purely in the service of telling the story.”
Chelebourg says that this project is different from his previous VR production, Alice: The Virtual Reality Play, because while that one was about stepping into the shoes of Alice in Wonderland, the scale and ambition were somewhat smaller. Jack goes one step further. He specifically made sure that the participants and the actor wore wireless gear to have a full, free range of motion. “We designed this button that allows her to keep in control of the narrative, so she’s deciding all the rhythms,” says Chelebourg. “The actress has all the power over the linear storytelling.”
For Jack’s world premiere at Tribeca, Chelebourg’s team used 21 infrared cameras manufactured by OptiTrack, “one of the leaders in LBE mo-cap (motion capture) hardware,” says the director. “They were carefully picked and placed to cover the entire set, capturing and analyzing in real-time not only the movements of the actress, but also the movements of the props and the viewer’s VR headset.”
The actor’s suit was covered with infrared reflecting dots, according to scenographer Ely Bessis. “They are key to optically triangulating the position of the actor at 250 frames per second with sub-millimeter precision,” he explains. “This way she can interact seamlessly with the environment – lifting and playing with props – and with the viewer, who can be physically touched.”
The Jack experience was run on a Windows and Linux ecosystem carried by multiple HP workstations. Bessis says that everything ran in real time, including the motion capture system, the facial motion capture software, and the sound spatialization. The collected data was streamed to the player’s HP Omen backpack that ran the game engine application of the project. The setup was custom adapted to their needs with additional sound equipment used to record and process the voices inside this virtual world.
“All the data – positional, facial recognition, etc. – is processed and calculated on our powerful remote workstations, which remotely communicate with the viewer’s backpack,” explains Bessis. “The viewer’s backpack then takes care of running the game engine app that renders the stereoscopic imagery for the headset he or she is wearing.”
“We have learned that harvesting the power of technology to build an entirely walkable, interactive, and multisensory world shows that VR is an amazing storytelling medium,” says Chelebourg. “Not unlike planning a good magic trick, it takes the best hardware and design to make you forget that you are wearing anything and to bring out your sense of wonder. In the end, it comes down to the oldest trick in the book: creating an emotionally intense, joyful story with impactful characters and empathic human interactions.”
Low-Tech Effects, Too
Another old trick being used is when Jack’s shack shakes and gets raised into the air. The actual vibrations are created simply by having the actor stomp on the floorboards. (The wooden stage is set a few feet off of the ground, with walls on three sides and a small balcony at the end.) It is a simple yet effective tactic. When immersed in the VR headset, one cannot guess the source of the shaking.
“The story is organically crafted so that every viewer has a unique shared experience with the characters played by the actress,” says Chelebourg. “Everybody who has stepped through the wooden door into the immersive world of Jack so far has lived a unique daydream exclusive to his or herself whose length and dialogue varied depending on how he or she was inspired to participate. The emotional impact one achieves from such a human-centered setup is incredible. Every single feedback we have received so far has been a bit dithyrambic. It’s incredibly uplifting and motivational for us to witness so many people reaching back out to us with smiles on their faces, saying how difficult it was to return to the real word at the end of their journey!”
A second part to Jack is being developed, with Academy Award winner Lupita Nyong’o voicing the Giantess. For more information, visit Baobab Studios’ website at www.baobabstudios.com.
