The idea of Sculpture 1 - Sediment developed under the impression of old temple reliefs from Thailand and Cambodia.The depicted divinities, dancers, fabulous creatures and historical figures have strongly occupied and moved me. It was therefore very obvious to me to show a relief of life-sized figures on the drawn-out screen of a panorama.
In the work KÀ I used virtual lighting for the first time. In one scene I project a stone surface onto the stage. The stage is moving and the stone is lit following a virtual light source. Now, almost ten years later, I chose an opposing approach. The observer carries a torch with virtual light. The movements of the torch submerge a virtual relief of stone figures in light and shadow. The physical nature of the figures can be experienced by the movement of the light.
I have tried to work with the things one does not see. The observer decides where the confined torch light shines and what remains in darkness. On the one hand the torch only lights a small portion of the figures and their relationships towards each other. One can never see the whole picture. On the other hand the light of the torch reveals where the observer glances. The observer cannot look privately upon an object anymore. Other people in the room see what the light illuminates and which path it takes. By the things that are omitted, one can see what shall remain hidden.
The poses of the figures came into being during two separate sessions with a male and a female nude model. Both were captured by a 3D-camera in various momentary snapshots. Most postures emerged from cooperative improvisation upon feelings and actions that were sketched in writing beforehand and by the anticipation of a partner or counterpart who wasn't there yet at that point of time. Only the following collage of the sculptural snapshots established the final relationship between and among the figures.
The creation of the collage itself was more of an intuitive investigation into possible interactions between the figures and their surroundings than the realization of a specific concept. Animals and mythical creatures of the projection are owed to magical thinking. I did not revolve around the depiction of religious content but rather the possibility of a metamorphosis of the body. The animal recordings were taken in the State Museum of Natural History Karlsruhe. The stone in the background is created synthetically by fractal noise. Elevations and indentations were incorporated in the stone wall to support the figures or give them more room.
The work received its name very late. The term sediment primarily denotes a certain type of rock that is created by erosion and subsequent compression. Here the term also stands for fragmentary unconscious pictures that accumulate and consolidate in the subconscious over the years, for the composition of an overall picture from fragmented pieces.
setup
The ZKM PanoramaLabor contains a panoramic screen with a diameter of approx. 6m and a height of approx. 2.16m. There are 6 projectors and 16 speakers installed at the ceiling and walls, eight speakers below and eight above the screen. I could link my own program code into the existing player software of the ZKM PanoramaLabor.
The movement of the torch is tracked by the same motion tracking system that I first used for The Sound of Things. The virtual torch itself consists of a plastic pipe that is wrapped in crash and ends in 3D-printed caps. A battery driven infrared LED is incorporated into the tip of the torch.
Eight infrared cameras right above the projection screen are tracking the movement of the torch tip and determine its 3D position in relation to the curved screen. The exact position is calculated by means of pairwise calibrated cameras and stereoscopic algorithms.
A light source is simulated in the virtual space at the position of the torch tip. The video of a candle flame, adjusted out of focus, is used as an environment map and constitutes the basis for the color, flickering and dancing of the virtual light.
the relief
To shoot the 3D-figures, I attached a Kinect 3D-camera to a small motorized sleigh and let it slowly slide forwards and backwards. I averaged multiple shots to gain more exact depth measurements. Most of the individual sculptures had to be cut out by hand after the 3D capture before they could be assembled anew.
The collage is based on a so called heightmap, an image in which close objects are encoded by bright pixels and more distant objects appear as dark pixels (see image below). 8 bits or 256 greyscale values, which are normally used for textures, are not enough to encode the depth information. This is why I worked on the whole collage using 16 bits resolution.
Figures of the collage which are further away are not only depicted darker but also a little bit smaller to suggest a perspective effect.
During the development process of the work, I studied the structure of stone and the reflection of candlelight from a cave wall in particular. For example, beginning at its center, the light of the candle does not only grow darker. It also appears paler and more blueish than the center area that is bathed in warm light. At a certain distance, there is obviously not enough brightness for color vision anymore.
casting light and shadow
The technical heart of the work is formed by a so-called shader, a routine that is executed on the graphics cards of the projection computer for every single one of the approx. 8192x1024 effective pixels. In this case, the shader determines light and shadow according to the relative position of the light source and the surface properties.
The cast shadow is calculated by a so-called shadow feeler. The shadow feeler is a ray from a point on the surface to the light source. If there is an object intersecting this ray, masking the light of the source, then the point on the surface is in the shade.
Aside from a varying basic color and glossiness of the stone, there is another special value belonging to the light calculation which changes the perceived depth of the torch in relation to the objects and by which objects that are far away can still be sufficiently lit.
cave sound
As written above, the chroma of the light diminishes with increasing distance from the light source. Something similar happens in the sound domain. Advancing deeper into a cave, more and more of the high frequencies of outside noises get filtered out. Finally all that remains are low frequencies and resonances of the cave chamber – a kind of low humming that eventually served as an inspiration for the sound of the installation.
The picture above shows the spectrogram of an audio recording in a cave on a rainy day. The time axis runs from left to right, the frequencies from bottom to top. The diminished treble frequencies clearly show up as a bright region at the top, the resonance frequencies are well visible as dark bands below that. Sounds of braking and speeding up cars are preserved as lines striving diagonally up and down only in the low frequency band. All the way on the right is the sound of a rattling train that is passing at a certain distance. The vertical lines in the high frequency range are drops of water inside the cave.
The motion tracking and the rendering of images and sounds are running as networked processes and can be distributed to up to three computers depending on the computing capacity at hand.
In the end, the work remains an illusion of light, sound and a little bit of mathematics. I hope that these details on the creation may be illuminative for one or the other as well.
credits
Sculpture 1 - Sediment developed between 2011 and 2013. The first prototype materialized a few months after the November 2010 release of Microsoft's Kinect 3D camera. It was thoroughly revised and finished in 2013, including the use of the motion-tracker and the development of the multichannel sound for the Beyond exhibition.
I am very grateful for everybody who contributed to this project, knowingly or unknowingly, most of them willingly:
inspiration | Ida Nordpol, Rainer Kuhn |
male model | Axel Jahn |
assistance | Ida Nordpol |
assistance retouche | Tom Hahn |
voices | Ida Nordpol, Holger Förterer |
Naturkundemuseum Karlsruhe | Dr. Eduard Harms, Martin Hörth |
support torch recordings | Katja Löffel |
support torch construction | Felix Kaiser, Napoli-Design, Mirco Fraß |
audio equipment | Frank Halbig |
ZKM PanoramaLabor | Bernd Lintermann, Derek Hauffen, Silke Sutter |
support hardware | Computer Consult Wening |
torch holder | Matthias Gommel |
support optics | Lensation GmbH |
human resources | Kilian Ochs, Frederik Busch, Johann Korndörfer |
Special thanks to the team at Beyond 2013, to Jörg Lemke for the impulse to revisit every single relationship and his insights into the secrets of zen gardens, and to the Wening Family for the possibility to study sound and sight in the cave.