DALLAS – Technology’s ability to produce an emotional and interactive experience to connect people is being demonstrated at the Dallas Holocaust & Human Rights Museum’s Dimension in Testimony Theater. The stories of survivors of the Holocaust are being preserved and presented with the help of a holographic system that utilizes an artificial intelligence platform, custom show control system and integrated with automated lighting, audio and video.
More details from EW4D (www.ew4d.com):
USC Shoah Foundation, founded in remembrance of the survivors of the Holocaust, by Steven Spielberg in 1994, worked with the museum to create only the second holographic theater experience using Natural Language Processing (the automatic manipulation of natural language by software) to provide visitors with a dynamic narrative of the Holocaust survivor’s experiences during the Nazi Third Reich of World War II.
EventWorks 4D, LLC a Cleveland holographic staging and content developer firm (EW4D) engaged in a yearlong exchange of ideas and proposals with the foundation and museum, overseeing, designing and implementing the theater. Globally recognized for its leadership of many large-format and high-profile holographic projects, EW4D was chosen to provide technological and creative direction for the ground-breaking exhibit.
Combining a 360-degree visual experience and theatrical formats in this spatial installation for a Pepper’s Ghost illusion required the integration of multiple systems for complete continuity. The depth and system design to create an immersive storytelling experience played a major part in the selection of media playback. Content is driven by a 7th Sense NANO-SDI Delta media server and 8 individual BrighSign XT-234 players. CreateLED / LED3 provided 28 panels of their Airmag 1.9mm LED wall as the primary floor bounce to provide a high brightness-to- contrast ratio for the holographic projection. Show control management was formatted with an original GUI template interfaced with a 22” ELO touch screen, remote Surface Pro 6 tablet and back-up fabve for ease of docent operation. Eight LCD 85” 4K monitors and a Panasonic PT- RZ12K projector surrounded the room displaying a realistic appearance of a pre-World War II Jewish home creating a more accurate augmented installation experience. This dual augmentation between the physical home and the holographic survivor enhance the overall emotional audience experience.
Linking AV & Holographic Techniques Creates a “Smart” Hologram
The IT team at the Shoah Foundation is at the forefront of building dialogue systems and developing technologies that allow people to talk to a computer system that looks and acts almost human in the way it converses, understands and exhibits emotion. EW4D took the Natural Language Processing-based application developed by Shoah and evolved it into a long-term and interactive theatrical experience in the museum installation.
Natural Language Processing allowed those in the audience to verbally ask the hologram a relevant question which was then processed via technologies such as tagging different parts of
speech (POS Tagging), stemming and lemmatization of the phrase to determine the most accurate response video.
“With the addition of our customized show control system, developed by our UK technology partner, Interactive Imagination, the utilization of the ASR-NLP was made routine for the moderator operating the system,” says Joel Solloway, President of EW4D. “We could now introduce a new technology and make it user friendly for volunteer docents operating the theater’s daily schedule.”
The latest EW4D innovation was introduced to the Dallas project allowing the audience to be treated to an even more robust and singular experience with the survivors “By utilizing Machine Vision (an automatic audience inspection system developed by Interactive Imagination, to discern a member of the audience standing through the use of a visual feed) along with an array of microphones and speakers in the ceiling and an upgraded show control system, visitors can speak directly to a survivor rather than through an intermediary who rephrases a question, says Interactive Imagination’s Enzo Fiondella. “This will improve the selectivity of the questions from the audience, eliminating the need for a moderator.”
For this installation, EW4D introduced a Velodyne LIDAR Puck Lite sensor with high resolution ideal for applications in the areas of 3D mapping/imaging to target audience members when asking questions. Utilizing a Shure MXA910 ceiling array microphone with 8 separate lobes to capture sound sources from above, Dante connectivity and AES67 audio networking software onboard adjusted toward each participant’s voice, assuring questions where interpreted correctly by the system.
Combined with voice recognition technology, the Pepper’s Ghost holographic illusion technique formed the backbone of the project. High-definition 360-degree footage of each survivor’s testimony was captured using 116 cameras. The footage, comprised of 2,000 questions and responses, was then used by EW4D to create a realistic visual perspective that created the illusion that the audience was talking to a real person on stage.
“The beauty of this technique is that the audience is totally immersed and engaged truly believing that the stage image is quite realistic and believable, “says Solloway. “An enhanced mood and atmosphere were achieved using the pre-show visual display layers created by Vancouver-based NGX Interactive along with carefully-designed lighting effects we created specific to the on-screen visual. The installation is so convincing, audience members sometimes believe they are having a one-on-one conversation with the survivor.”
Show cues were considered when developing the design to ensure a consistent and seamless flow throughout the show. From the introductory high-profile video sequence setting the atmosphere in the theater to the effectiveness of the system’s interface during the Q&A session, a believable two-way dialogue between the audience and survivor was created.
High-Quality Sound Performance
The audio element of the project was two-fold, comprising playback and voice capture. As the interactivity of the system primarily relies on natural language processing, capturing high quality audio was essential.
The acoustics and aesthetics of the theater were carefully fashioned by award-winning multidimensional-scenic design firm, David Korins Design. To fully take advantage of the acoustic environment, EW4D integrated a distinctive 5:1 PA system combining high profile Renkus Heinz IC6 series ceiling speakers, IC7 L/R columns, C112 subwoofer and a Behringer X32 digital mixer with 32 channels of Audinate Dante networking assuring a full range audio perspective as well as the best system EQ and sound pressure levels (SPL) for a dynamic audio experience.
With vocal intelligibility being a focal point along with an emotive soundtrack for the opening video sequence, the system requirements had to be carefully measured for optimum sound exposure without uncomfortable audio levels.
System Maintenance & Support
The system design needed to guarantee long-term reliability, ease of operation for the moderator managing the experience and offer the capability to easily update content and manage all aspects remotely. To implement a system that could withstand the rigors of a 365-day, eight-hour cycle of operation, the team had to research the best-case scenarios for system integration. EW4D’s expertise in building robust interactive exhibits systems and the intuitive approach to show control allowed them to maintain reliable system-operation daily.
The show control system was developed in-house to offer a great level of control over the different components used within the installation as the system had to operate over multiple platforms. The greatest obstacle faced was combining the diverse components and systems while ensuring they worked in unison flawlessly. This was achieved through developing a custom show control system that integrates the components through a range of programming languages such as C++, Python, Node.js and Applescript. Fine tuning the scene lighting to enhance the users’ experience of the hologram was also difficult, especially since the survivor’s content dynamically changes.
What’s Next
The Dimensions in Testimony Theater marks only the second such application utilizing AI and holographic innovations for historical documentation, bringing the survivors’ stories to life with a volumetric hologram capable of conversing with the audience using custom interactive voice technology.
“Future generations will continue to have dialogue with holocaust survivors,” says Solloway. “Our mission in creating this theatrical exhibit was to never let this happen again and to
remember those who experienced the horror of Nazi occupation. This is indeed living history and it wouldn’t be possible without this technology.”
After the first holographic theater was launched at the Illinois Holocaust Museum & Education Center in 2017, EW4D and Interactive Imagination have continued to improve the stability and reliability of the system by further developing the automated processes which manage and maintain the components of the application. The team has also reduced the system’s energy footprint by implementing power-saving methodologies for all the connected components. As part of the maintenance and troubleshooting protocol, the companies maintain constant access to the system, allowing them to remotely monitor the exhibit from the US and UK, ensuring constant stability, functionality as well as gaining usage feedback.
The Way We See the World is Changing
“We’re just at the cusp of something so exciting and I believe AI, holographic technology and ASR-NLP will be the dominant communication tools for museums,” says Solloway. “We hope to continue making an impact by combining our blend of technology with the human element. Museums will gravitate more towards holographic AI in areas such as science, the arts, medicine and politics where those who have made a significant contribution to society can be forever immortalized and their voices heard for generations to come.”
