The ZKM PanoramaLabor contains a panoramic screen with a diameter of approx. 6m and a height of approx. 2.16m. There are 6 projectors and 16 speakers installed at the ceiling and walls, eight speakers below and eight above the screen. I could link my own program code into the existing player software of the ZKM PanoramaLabor.The movement of the torch is tracked by the same motion tracking system that I first used for The Sound of Things. The virtual torch itself consists of a plastic pipe that is wrapped in crash and ends in 3D-printed caps. A battery driven infrared LED is incorporated into the tip of the torch.
Eight infrared cameras right above the projection screen are tracking the movement of the torch tip and determine its 3D position in relation to the curved screen. The exact position is calculated by means of pairwise calibrated cameras and stereoscopic algorithms.A light source is simulated in the virtual space at the position of the torch tip. The video of a candle flame, adjusted out of focus, is used as an environment map and constitutes the basis for the color, flickering and dancing of the virtual light.
To shoot the 3D-figures, I attached a Kinect 3D-camera to a small motorized sleigh and let it slowly slide forwards and backwards. I averaged multiple shots to gain more exact depth measurements. Most of the individual sculptures had to be cut out by hand after the 3D capture before they could be assembled anew.
The collage is based on a so called heightmap, an image in which close objects are encoded by bright pixels and more distant objects appear as dark pixels (see image below). 8 bits or 256 greyscale values, which are normally used for textures, are not enough to encode the depth information. This is why I worked on the whole collage using 16 bits resolution.
Figures of the collage which are further away are not only depicted darker but also a little bit smaller to suggest a perspective effect.
During the development process of the work, I studied the structure of stone and the reflection of candlelight from a cave wall in particular. For example, beginning at its center, the light of the candle does not only grow darker. It also appears paler and more blueish than the center area that is bathed in warm light. At a certain distance, there is obviously not enough brightness for color vision anymore.
The technical heart of the work is formed by a so-called shader, a routine that is executed on the graphics cards of the projection computer for every single one of the approx. 8192x1024 effective pixels. In this case, the shader determines light and shadow according to the relative position of the light source and the surface properties.
The cast shadow is calculated by a so-called shadow feeler. The shadow feeler is a ray from a point on the surface to the light source. If there is an object intersecting this ray, masking the light of the source, then the point on the surface is in the shade.
Aside from a varying basic color and glossiness of the stone, there is another special value belonging to the light calculation which changes the perceived depth of the torch in relation to the objects and by which objects that are far away can still be sufficiently lit.As written above, the chroma of the light diminishes with increasing distance from the light source. Something similar happens in the sound domain. Advancing deeper into a cave, more and more of the high frequencies of outside noises get filtered out. Finally all that remains are low frequencies and resonances of the cave chamber – a kind of low humming that eventually served as an inspiration for the sound of the installation.
The picture above shows the spectrogram of an audio recording in a cave on a rainy day. The time axis runs from left to right, the frequencies from bottom to top. The diminished treble frequencies clearly show up as a bright region at the top, the resonance frequencies are well visible as dark bands below that. Sounds of braking and speeding up cars are preserved as lines striving diagonally up and down only in the low frequency band. All the way on the right is the sound of a rattling train that is passing at a certain distance. The vertical lines in the high frequency range are drops of water inside the cave.
The motion tracking and the rendering of images and sounds are running as networked processes and can be distributed to up to three computers depending on the computing capacity at hand.
In the end, the work remains an illusion of light, sound and a little bit of mathematics. I hope that these details on the creation may be illuminative for one or the other as well.