October 2010

Into The Trees is a realtime audio/visual performance with the acclaimed cellist Zoë Keating.

The piece was entitled "Into the Trees". The project ended up being conceptually sound for a few reasons. First of all, Zoë's newest album is called "Into the Trees" and is an exploration of Zoë's love of her home which is situated in the redwoods of northern California. Secondly, I have been fascinated with creating virtual worlds with code for years and was happy to have another opportunity to rewrite a terrain engine. I would create a virtual world with code while Zoë would create an audial world with her music and we would do it at an arts festival whose theme was "Build Your Own World".

Zoë composed a new piece for this collaboration. It is roughly 14 minutes long and builds a wall of sound that is eventually peeled away piece by piece until you emerge back at the beginning. The song begins and ends with the same haunting two notes.

I decided to depart from my usual process of relying on microphone input to create the visuals. I wanted to do all of it by hand. There would be no audio reactive landscape or beat driven content. This decision wasn't entirely without outside influences. Stringed instruments, particularly ones played by bow, are notoriously difficult to analyze. It is very difficult to pull out any beats from a wall of sound. Also, the piece Zoë composed was much more ambient than previous works. It was very dark and somber so beat-detection would have been useless. There were simply no beats to detect. The visuals would have to be controlled by hand.

The rough outline of the piece is as follows:
• Create an algorithmic terrain and meandering path
• Create sky with a dynamic day/night cycle
• Create boulders and grass
• Fill the landscape with towering trees
• Return to barren terrain and meandering path

The Terrain and Path

The canvas for this project is a smoothly winding path through a subtly changing terrain. The path is a combination of sine waves. There is a main wave with a very low frequency and on top of that is added a higher frequency wave with a much smaller amplitude. Even though the wave is very repetitive and predictable, it ends up working fine for my needs. I didn't want to draw too much attention to the path by making it noisier, but I also didn't want a straight line.

The terrain is also very subtle. It is a combination of sine waves but with a little Perlin noise influence. The Perlin is scaled up very high so you never really feel the Perlinness of the algorithm. Like the path, I wanted something between boring and chaotic. Over the course of the 14 minute piece, the altitude of the terrain changes. By the end of the piece, the changes in elevation are about 10x what they are when the piece begins.

The visual of the path is very simple and is done in a shader. I use the distance of every pixel to the path to create a subtle gradient and then add some striping to make it stand out. The camera also follows the sine wave equation for the path, with some variation. It is allowed to swing a bit outside the path, but never wanders too far away. The direction the camera faces is also automated, but can be controlled with mouse input as well.

The Sky

The sky proved to be the most interesting challenge. I wanted to implement a smooth night-to-day-to-night cycle which is something I had not attempted before. I have done some simple sun as light source studies in the past, but never tried to create a proper dynamic sky-box (or dome). It ended up being fairly simple once I came to grips with the fact that dynamic cloud movement was beyond my skill level for now so I should just pretend it wasn't even an option.

I purchased some spherical textures of skies and picked out four which I could cross fade as the sun rose and set. If the sun is below the horizon, I would use the night texture. If the sun was high in the sky, I would use the blue sky texture. For other sun positions, I would linearly interpolate between the 4 textures to create a smooth transition from day to night.

Sadly, this ended up looking rather blah. It all felt very static and felt like a matte painting. It was finally time to implement my first dynamic bloom and God ray effect. Again, this ended up being fairly simple since I had already learned how to deal with FBOs.

From what I read online, bloom effects are created by rendering the lighting to a separate FBO and then overlaying a blurred version of the FBO on top of the fully rendered scene. I decided to simplify things a bit and just render the whole scene to a separate and smaller FBO. After a blur and contrast bump, I then overlay the bloom FBO back onto the scene. This helped me solve (i.e., ignore) the bloom occlusion issue. And it turns out, this is most of what is required to implement the beautiful (i.e., easily abused) God ray effect.

If you render the bloom FBO multiple times with an increase to the image's scale with each iteration, you get a fairly nice atmospheric ray effect. It isn't perfect and unless it is done on the GPU it can be quite slow. Since I was already doing the bloom overlay in a shader, it was easy to throw in a for loop, take the light source location into account, and render the bloom FBO 50 times per frame with the appropriate scale increase. This really drove home how impressive shaders can be. To do a fullscreen post-process render 50 times per frame and still be able to achieve real-time framerates... wow. GLSL, I love you very much.

The God rays allowed me to pay less attention to the realism of the sky and focus more on the fantasy of my world. A failed attempt to create a realistic sky is much less compelling than a successful attempt to create a fantastic, though unrealistic, sky. These crepuscular rays also helped me with the realism of the elements which would eventually populate the terrain. A bright bloom with corresponding rays over mostly silhouetted content can be quite compelling.

I also ended up using a 1D gradient texture which allowed me to fog my foreground assets to a reasonable color as the lighting changes. During the day, the trees and rocks fade over distance to a blueish hue. At sunset, they fade to a pinkish-orange. And at night, they fade to a deep purple. I do these color interpolations using shaders.

The Boulders

The super-ellipsoids I used for the boulders were the only part of the Fuji project that I recycled. It is such a nice solution, I just had to use it again. The boulders first came about when I saw André Michelle use a superellipsoid in an audio visualization. Here is an article on the superellipsoid by the creator himself, Paul Bourke.

For my boulders, I create a superellipsoid and save all the vertex positions to it's own VBO. This way, I am only calculating the positions once for each boulder's lifetime. Since I am storing the positions, I can add some Perlin noise to the vertices in order to randomize the forms a bit without much strain on the processor. Voila! Instant boulders.

The Trees

Along with water, smoke, and fire, trees are some of the more difficult things to attempt with code. Trying to code realistic algorithmic trees is quite the rabbit hole. Once you start, you will never be content with how they look or how quickly they render. Finding the right balance between speed and realism is key. Sure, you can run a few million iterations of some L-system or branching node network code, but good luck getting even one to render 60 frames a second. The flip side is to just stick a billboarded tree graphic into your scene. However, any scrutinization and your laziness will become very obvious.

I decided to only be semi-lazy. I used two styles of trees: large and 3D(ish), or small and flat. I didn't have to worry about full access to all of the terrain from all angles. In other words, this wasn't a free-roaming first-person-shooter. The camera always moves roughly forward and faces roughly forward.  This made the flat trees more realistic because nobody was ever viewing them side-on and I also didn't have to worry about the uncanny effect that occurs when a single image of a tree is made to always face the camera.

The small trees were made with two quads in a right angle cross. I used 16 different images (purchased from a texture site) in one large texture and just chose a random set of UV coordinates for each tree. The large trees needed to have a bit more character so I ended up using a perturbed cylinder to create the massive trunks and stuck a two-quad cross on the top half. I used images of red woods for a couple reasons. They are the type of trees that exist near Zoë's home, and they have a very straight trunk and high canopy which made my job much easier. And let's not forget the God rays... those things shining through the canopy with a bright bloom can really bring some much needed drama to the scene while obscuring just how the trees were made.

In order to bring the trees in without any pop, I create them at a miniature scale in the distance and make them grow quickly to full size (approximately 1 to 2 seconds of growth time). Though not realistic (you can obviously see them grow into place), the effect was quite magical.

The Details

Glowing dust: What forest scene is complete without a bunch of floating glowy things. Whether they are stand-ins for fireflies or dust motes, they push the 3D illusion of the space while adding some interesting visual noise.

Ground mist: Because the project was to be projected, I couldn't rely on the subtle forest-at-night lighting to come through. I didn't know ahead of time what the ambient lighting situation was going to be so I added a ground mist toggle which I could turn on to add some nice smoky effects to the scene. This would help add contrast to the piece by pushing the silhouetted look of the trees. Plus, I had recently played Limbo and I was really impressed with the look. It was definitely an influence on the final piece.

Shooting stars: The night sky was a high resolution photograph of the Milky Way overlayed on a more traditional field of bright stars. To increase the fantasy aspect of the scene, I added a large number of meteorites to help draw attention to the night sky. They were very simple, very short lived, line segments which emerge from a specific radiant and shoot off in a random direction for 5 to 15 frames.

Random trails: The forest scene also had some particle trails which generally hovered over the path and would travel away from the camera. They moved along sine waves and trailed a couple dozen previous positions that were connected with a quad-strip.

Control

Controlling the application was very simple. It was all done manually. I created keyboard commands which would trigger various objects. If I press '1', the skydome would toggle on or off. If I press '2', the terrain would toggle. '3' is for grass. '4' is for trees, etc. Triggering these effects required me to do some camera manipulation to avoid pop-in of the assets. For example, I didn't want the terrain to just appear out of nowhere so when it was time to bring in the terrain, I would aim the camera up at the sky and then press '2'. This way, the terrain would appear while the camera was looking away. I could also do this when it was time to toggle the sky. I would just look down at the terrain and then press '1'.

The trees and boulders were a bit easier to deal with. The key presses would trigger the creation of a bunch of objects rather than just toggle them on or off. The trees are always on, but until I press '4', none exist onscreen. Also, the trees would scale up over time and the boulders would rise up out of the ground so it didn't matter where I was looking when I triggered them. This way, I got to play in the assets.

Finally, there was a minor manual audio effect which I didn't really use much but I wanted to mention it here because I think it is a fantastic effect. It would be better suited to a project with an intense bass beat. Each press of the space bar sends out a shock-wave through the terrain which also effects the boulders and trees.

Final Piece and Performance

"Into the Trees" was performed in the South Hall at the San Jose 2010 Biennial. The venue was brilliant. Inside the South Hall tent, a huge Mad Maxian drive-in theater was created. The audience sat in rusted, dented, and crashed cars, sometimes stacked 2 high. The cars were often missing tires and windshields and some even had empty space where the engine should have been. The drive-in screen was beautifully built, complete with the name EMPIRE written across the top in 3D block letters, but the M and trailing E had long since fallen off. It felt like a post-apocolyptic theater with an audience of survivors trying to rebuild after the bomb, attempting to distract themselves from the destruction by creating impromptu performances about creation. It was really quite moving. Many thanks to the ZER01 team for creating such a beautiful venue and additional thanks to Jaime Austin, Steve Dietz, Craig Hobbs, and of course the extremely talented Zoë Keating.