ILLINOIS – Technology’s ability to produce an emotional and interactive experience to connect people is being demonstrated at the Illinois Holocaust Museum and Education Center’s Take a Stand Center. The stories of survivors of the Holocaust are being preserved and presented with the help of a holographic system that uses artificial intelligence, custom control system, and integrates lighting, audio and video.
More details from EW4D (www.ew4d.com):
USC Shoah Foundation, founded in remembrance of the survivors of the Holocaust, by Steven Spielberg in 1994, worked with the museum to create the first holographic theater experience using Natural Language Processing (the automatic manipulation of natural language by software) to provide visitors with a dynamic narrative of the Holocaust survivor’s experiences of World War 2.
EventWorks 4D (EW4D) engaged in a three-year exchange of ideas and proposals with the foundation, overseeing, designing and implementing the theater. Globally recognized for its leadership of many large-format and high-profile holographic projects, EW4D was chosen to provide technological and creative direction for the ground-breaking exhibit.
Combining film and theater formats in the spatial installation for a Pepper’s Ghost illusion, the design & depth of the stage played a part in the immersive storytelling. EW4D incorporated a fine pixel pitch CreateLED Airmag LED wall of 2.6mm at the rear of the stage to provide a high brightness-to-contrast ratio as well as utilizing it for additional video content.
“The LED wall displayed a realistic appearance of stage drapes when the holographic holocaust survivor was seated on stage, allowing us to create a more accurate augmented installation experience,” says Joel Solloway, President of EW4D. “We call this dual-augmentation since the primary image in the foreground, the Holocaust survivor, is an augmented scene in its own right.”
Linking AV & Holographic Techniques Creates a “Smart” Hologram
The Institute for Creative technology (ICT) at the University of Southern California, working in conjunction with the Shoah Foundation is at the forefront of building dialogue systems and developing technologies that allow people to talk to a computer system that looks and acts almost human in the way it converses, understands and exhibits emotion. EW4D took the Natural Language Processing-based application developed by ICT and evolved it into a long-term and interactive theatrical experience in the museum installation.
Natural Language Processing allowed the moderator to verbally ask the system a relevant question which was then processed via technologies such as tagging different parts of speech (POS Tagging), stemming and lemmatization of the phrase to determine the most accurate response video.
With the addition of our customized show control system, developed by UK technology partner, Interactive Imagination, the utilization of the NLP was made routine for the moderator operating the system,” says Solloway.
Combined with voice recognition technology and NLP, the Pepper’s Ghost holographic illusion technique formed the backbone of the project. High-definition 360-degree footage of each survivor’s testimony was captured using 116 cameras. The footage, comprised of 2,000 questions and responses, was then used by EW4D to create a realistic visual perspective that created the illusion that the audience was talking to a real person on stage.
“The beauty of this technique is that the audience is totally immersed and engaged truly believing that the stage image is quite realistic and believable,”says Solloway. “An enhanced mood and atmosphere was achieved using the visual display layers set on stage, along with carefully-designed lighting effects. The installation is so convincing, audience members sometimes believe they are having a one-on-one conversation with the survivor.”
Show cues were considered when developing the design to ensure a consistent and seamless flow throughout the show. From the introductory high-profile video sequence setting the atmosphere in the theater to the effectiveness of the system’s interface during the Q&A session, a believable two-way dialogue between the audience and survivor was created.
High-Quality Sound Performance
The audio element of the project was two-fold, comprising playback and voice capture. As the interactivity of the system primarily relies on natural language processing, capturing high quality audio was essential.
The acoustics of the theater were challenging due to the physical constraints of the space. To compensate for the live room environment, EW4D integrated a custom PA system that would satisfy a full range audio perspective as well as determining the best system EQ and sound pressure level (SPL) capable of a full range audio experience suitable for the space.
With vocal intelligibility being a focal point along with an emotive soundtrack for the opening videos, the system requirements had to be carefully measured for optimum sound exposure without uncomfortable audio levels. To minimize issues with voice intelligibility, when a question is asked by the audience it is then repeated by the moderator through a wireless interface with a high-quality microphone that provides input to the control PC.
System Maintenance & Support
The system design needed to guarantee long-term reliability, ease of operation for the moderator managing the experience and offer the capability to easily update content and manage all aspects remotely. To implement a system that could withstand the rigors of a 365-day, eight-hour cycle of operation, the team had to research the best-case scenarios for system integration. EW4D’s expertise in building robust interactive exhibits systems and the intuitive approach to show control allowed them to maintain reliable system-operation daily.
The show control system was developed in-house to offer a great level of control over the different components used within the installation as the system had to operate over multiple platforms. The greatest obstacle faced was combining the diverse components and systems while ensuring they worked in unison flawlessly. This was achieved through developing a custom show control system that integrates the components through a range of programming languages such as C++, Python, Node.js and Applescript. Fine tuning the scene lighting to enhance the users’ experience of the hologram was also difficult, especially since the survivor’s content dynamically changes.
What’s Next
The Take a Stand Center marks one of the first applications of AI and holographic innovation for historical documentation, bringing the survivors’ stories to life with a three-dimensional hologram that is capable of having a conversation with the audience using custom interactive voice technology.
“Future generations will continue to have dialogue with holocaust survivors,” says Solloway. “Our mission in creating this theatrical exhibit was to never let this happen again and to remember those who experienced the horror of Nazi occupation. This is indeed living history and it wouldn’t be possible without this technology.”
Since the holographic exhibit was launched, EW4D and Interactive Imagination have continued to improve the stability and reliability of the system by further developing the automated processes which manage and maintain the components of the application. The team has also reduced the system’s energy footprint by implementing power saving methodologies for all the connected components. As part of the maintenance and troubleshooting protocol, the companies maintain constant access to the system, allowing them to remotely monitor the exhibit from the US and UK, ensuring constant stability, functionality as well as gaining usage feedback.
As EW4D innovations are introduced to other museum projects, such as the Dallas Holocaust Museum, the audience will be treated to an even more robust and singular experience with the survivors “ By utilizing Machine Vision (an automatic audience inspection system developed by Interactive Imaginations, to discern a member of the audience standing through the use of a visual feed) along with an array of microphones and speakers in the ceiling and an upgraded show control system, visitors will be able to talk directly to a survivor rather than through an intermediary who rephrases a question, says Interactive Imagination’s Enzo Fiondella. “This will improve the selectivity of the questions from the audience, eliminating the need for a moderator.”
The way we see the world is changing
EW4D also plans to incorporate a pre-show in the seating area for those in the audience whereby a series of LCD monitors will create different visual environments and ancillary educational information prior to the holographic survivor story.
“The project at the Illinois Holocaust Museum and Education Center had a global impact, sparking interest from numerous Holocaust museums,” said Solloway. “On a recent visit to the museum, former British Prime Minister Tony Blair and his wife commented that the experience was unlike anything they had ever seen and that something similar would be valuable in Britain.”
For other mixed reality/AI applications, EW4D is developing a life-size pop-up, interactive hologram platform that uses multiple forms of interactivity and customized content and could be well-suited for promotional purposes at events.
“We’re just at the cusp of something so exciting and I believe AI, holographic technology and NLP will be the dominant communication tools for museums,” says Solloway. “We hope to continue making an impact by combining our blend of technology with the human element. Museums will gravitate more towards holographic AI in areas such as science, the arts, medicine and politics where those who have made a significant contribution to society can be forever immortalized and their voices heard for generations to come.”
