3D AV TECHNOLOGY IS ALL THE RAGE THESE DAYS from single screen theatrical 3D movies and consumer 3DTVs, all the way up to immersive 3D systems, where the viewer is surrounded by integrated 3D full-length screens on all four sides, the ceiling, and floor.

Immersive 3D systems are now in use at leading universities, businesses, and government agencies, including the military. When combined with 3D viewer glasses—and tracking systems that tell the AV system computer(s) where the viewer is looking and what perspective they should be seeing—immersive 3D delivers a true “virtual reality” experience."

It falls to AV technicians and managers to purchase, install, and maintain immersive 3D systems, plus provide training to users. This begs the question: Just how complicated is the care and feeding of immersive 3D AV systems?

Immersive 3D starts with the basic 3D visual experience. It involves displaying separate left and right visual fields in the user’s 3D glasses in real-time, correlated in such a fashion that the user’s brain integrates the two feeds into a single 3D image in their head. To do this, the 3D system requires ongoing software control by one of more computers-both to feed the correct images to the user and to constantly track the user’s physical position, so that the image positions and perspective can be altered as needed to remain realistically 3D. This is accomplished using a tracking system that is attached to the user, so that the controlling software can monitor their positional changes in real time.

This technology then goes one step further by surrounding the user with 3D images around, above, and below them. This technology is typically seen in a room-like immersive 3D system known as the CAVE. Short for ‘CAVE Automatic Virtual Environment’, CAVE systems have three or more screens, each one serving as a wall, floor or ceiling in the CAVE itself. In the ideal CAVE, the user is surrounded on all four sides with full-length screens, with a ceiling and floor also in place. Each screen is covered by a rear-mounted projector. To cover the full floor with imagery, the projector has to be mounted in the room below the CAVE, and shoot upwards underneath the user’s feet.

It is possible to do 3D immersive videoconferencing, which involves shooting the subjects at each end with paired video cameras. However, the need for everyone to wear 3D glasses in order to see the 3D video makes eye contact impossible. It would be like conducting an in-person meeting with everyone wearing sunglasses; it’s not a good way to make eye contact.

Two of the more complex immersive 3D AV campuses in use today are operated by Duke University in Durham, North Carolina, and UC Davis (part of the University of California) in Davis, California. Duke operates a CAVE known as the DiVE (Duke immersive Virtual Environment). UC Davis operates its own CAVE and other immersive 3D systems—collectively known as KeckCAVES (W.M. Keck Center for Active Visualization in the Earth Sciences).

“The DiVE has six 10- by 10-foot screens serving as the walls, ceiling and floor; providing the most realistic virtual reality environment possible for the user,” says Steve Feller, a research engineer with Duke’s Visualization Technology Group, and the person who installed and maintains the DiVE. “We don’t do video, but rather 3D computer simulations that the user interacts with, and which can be shared with other users remotely in real time.”

Oliver Kreylos is assistant research scientist with the UC Davis Institute for Data Analysis and Visualization (IDAV) and the W. M. Keck Center for Active Visualization in the Earth Sciences (KeckCAVES). He develops most visualization software used at KeckCAVES, and maintains its CAVE plus two new 3D video “capture sites” developed by Professor Ruzena Bajcsy and her research group at UC Berkeley. Professor Bajcsy is using these systems to bring together remotely-located dancers in the same 3D virtual space, and allowing geo-scientists in separate locations to work interactively with seismic data.

“A 3D capture site consists of several clusters of four cameras each,” Kreylos says. “Each cluster reconstructs a 3D facade from its point of view, and a complete view of any object inside the site’s capture space is created by merging facades from multiple camera clusters in the same 3D coordinate system.” The capture site at UC Berkeley has 12 clusters (48 cameras total) for a full 360-degree reconstruction. The UC Davis capture site has two clusters, and only captures the front of a person sitting in front of a single-screen 3D display system.

The captured video can be viewed on a 3DTV or full CAVE. “We prefer immersive (i.e., headtracked and stereoscopic) systems, such as CAVEs, for their convincing simulation of colocation, i.e., remote participants appear as if they were really there,” he notes.

The complex aspect of immersive 3D systems are the software package selected to run the system. These vary depending the needs of the user. For instance, UC Davis’ W.M. Keck Center for Active Visualizations in the Earth Sciences (for looking inside the Earth’s structure in 3D) user a suite of visualization applications based on a common infrastructure. They include 3D Visualizer to interactively visualize gridded 3D data, LiDAR Viewer to analyze ultra-high resolution 3D laser scanning data, and Crusta to view sub-meter resolution topography models with global coverage; all developed at UC Davis.

The DiVE has many missions ranging from representing molecules and anatomy to creating realistic 3D environments in which cognitive psychological experiments can be conducted. (In one project, led by Duke’s Dr. Rachael Brady, subjects are exposed to fear stimuli in different settings, to see how well fear can be “unlearned”.) This is why the DiVE uses a range of visualization software such as 3DS Max, Avatar Creation, Avizo, Google Sketch Up, Syzygy, and Virtools.

The software can be expensive, but the AV components are often standard industrial products. A case in point: “The cameras comprising each UC Davis 3D reconstruction cluster are off-the-shelf Firewire cameras, typically ones made by Point Grey Research,” says Oliver Kreylos. “Each cluster is controlled by a Windows computer, responsible to reconstruct a 3D facade from offset camera images, connected directly to each camera in the cluster. An additional (Linux) computer serves as frontend for an entire capture site, offering a service to which 3D video clients can connect to receive 3D facades from all clusters. The front-end additionally synchronizes all cameras in a site using a custom-made trigger cable.”

Meanwhile, “The display side can be any desktop PC (client software is Linux or Mac OS X only), or any immersive display system,” he adds. “The display system integrated with our two-cluster capture site is a low-cost VR environment based on a 72-inch commercial 3D TV and a NaturalPoint OptiTrack 3D tracking system ... It turns out that non-experts (graduate students from non-computer science areas) can assemble such systems independently when following our instructions.”

The complexity and complexity of any immersive 3D system is based on the quality of its software and components. In general, both Feller and Kreylos report their systems to be quite reliable and easy to maintain: “It is the usual kind of maintenance that you do with any AV system,” says Feller. “You replace lamps, watch calibration and ensure that everything is running as it should be.” As for ease of use: Once users become familiar with an immersive 3D system, they tend not to have many problems.

At present, immersive 3D systems tend to run as standalones, with little or no connection to their respective WANs/LANs. This said, “a 12- cluster 3D video stream requires approximately 1MB/s of network bandwidth for each client connection, and works best at low latencies,” Kreylos notes. “We anticipate that deployment and regular use of 3D video systems will stress existing network infrastructure.”

Return on Investment (ROI) is a difficult calculation to perform with respect to immersive 3D. In many cases, these items are funded through special research allocations or grants. “KeckCAVES was funded by the Keck Foundation with matching money from the university,” says UC Davis Geology Professor Dawn Sumner, who uses the KeckCAVES to model and interpret ancient microbial structures. “Although it does have a price tag, our ROI has been substantial in terms of scientific insights we could not have obtained without the KeckCAVES.”

“Our 3D technology allows us to bring people together in a virtual 3D environment from geographically distributed places without traveling that cannot be duplicated in any other way,” says Dr. Bajcsy. “It is hard to quantify its return on investment, because there is nothing to compare it to except for the cost of time and fuel in travel.”

“Immersive 3D allows us to conduct cognitive psych tests that just aren’t practical or affordable in any other way,” Dr. Rachael Brady concludes. “So in a sense, this technology is priceless.”

Although immersive 3D technology is complex by nature, it can be straightforward to install and maintain. As a result, AV technicians and managers need not fear the care and feeding of this new technology, despite its mind-bending capabilities.

AVIZO vsg3d.com
GOOGLE SKETCH UP sketchup.google.com
SYZYGY FX syzygyfx.com
VIRTOOLS virtools.com

James Careless is a frequent contributor to AV Technology magazine.

With seven collaborative environments supporting high-resolution, 3D stereoscopic imaging capability, including a 100-million pixel, six-sided immersive virtual reality room, King Abdullah University of Science and Technology (KAUST) in Saudi Arabia has realized several world firsts for advanced visualization.

“The multi-channel audio system installed at KAUST’s Advanced Visualization Facilities provides students, researchers, and staff with an array of potential applications of audio in virtual 2D and 3D media space,” says Zachary Seldess, audio systems coordinator and developer, Visualization Lab, KAUST. “These systems allow us to heighten the immersive qualities of given virtual space through the use of a variety of spatial audio techniques, ranging from subtle to overt.”

For example, Seldess says, if we happen to be descending into a large, cavernous space in a virtual recreation of an archeological excavation site, we can use the audio system to simulate the ambient acoustic properties of that cave, giving the user subtle, non-visual cues as to the size and physical makeup of the space that he or she is exploring. -Marty Weil