You can download the screensaver from the 100 Year Starship site. A donation is required to get to the download page but you can suggest whatever donation you feel comfortable with.

Look Up is a project I worked on for the Barbarian Group. They are sponsors of the 100 Year Starship (100YSS) initiative headed up by Astronaut Mae Jemison. The purpose of the 100YSS is “… to make the capability of human travel beyond our solar system a reality within the next 100 years.” For now, this means discussing the many enormous hurdles that stand in our way, but also finding ways to get people interested in the concept and inspired by the possibilities.

I was asked if I wanted to make a screensaver that explored the nearest stars to our sun and I immediately said yes. I am addicted to all things space. I had worked on a series of projects for the 2012 Eyeo Festival and one of those was a very basic star browser. That project was quick and dirty and though I could salvage bits and pieces from it (such as the 100,000 star text file and the code to parse the data), all of the rest of it would need to be designed from scratch. I was further inspired by the great work of Michael Chang and the rest of the team that worked on the 100,000 Stars Chrome experiment.

The Tools
I used the Cinder C++ framework for this project. Cinder makes it incredibly easy to compile a .saver for Mac or a .scr for the PC. I also want to thank Andrew Bell for his assistance in tracking down a bug or two, and for handling the port to VC10 for the PC build. The data used in this screensaver was pulled mostly from Wikipedia with some extra details coming from Sol Station. I also enjoyed reading the post-launch writeup Michael wrote for HTML5Rocks about the 100,000 Stars project. Helped me focus.

Reality vs. Fantasy
One of the goals with this project was to make space more inviting. The reality of space is lonely and dangerous and we wanted this screensaver to minimize the endless black void overtones and try to find a way to show space fantastically without completely deviating from reality.

Challenge: Scale
The scale of the radius of a star compared to the vast distances between the stars is hard to wrap one’s head around and it certainly doesn’t make for easy code. A lot of trickery (and even a bold faced lie or two) was required to make the experience seem reasonable but not too unbelievable. The cheat was to just make the stars bigger. If we left the stars at a radius that was in proportion to the distances between them, the camera would have to get really really really close to a star before it became large enough to view well. This can be quite problematic. So we just made them bigger. A better solution would be to make a camera system that transitions between zones of scale. A challenge for another day.

Challenge: the surroundings
Another issue was the backdrop. I tried a couple different images of the Milky Way Galaxy but I just couldn’t find a look that I was satisfied with. I poked around on Google images for artistic representations of space and the ones that were the most visually rich were the ones that had pronounced nebula clouds in the background. I was not in the position to do real-time generative nebula effects (and honestly have never tried such a thing) so I decided to just use a photo of the Carina Nebula as the backdrop.

Challenge: Binary Stars
Next there is the issue of multi-star systems. Upon beginning the data research portion of this project, I was surprised to read that most bright visible stars are often actually a small group of stars bound to each other’s gravitation influence. Despite being composed of multiple stars, these systems would be hard to visualize because the distance between these bound stars is still pretty vast. As an example, the nearest star to us is Proxima Centauri which is considered to be part of the Alpha Centauri system which is itself a binary star pair. The distance between Alpha Centauri A and B is comparable to the distance between our sun and Neptune. However, the distance of Proxima Centauri to Alpha Centauri is around 15,000 times the distance from the Earth to the Sun. So if I were to try and show all three stars in the system and maintain a realistic relationship of their radii and their positions in space, they would need to be tiny points, the very thing I wanted to avoid having as the centerpiece of this screensaver. So I cheated. I pushed the multi-star systems closer so I could render them larger.

Challenge: Moving forward
Finally, there was the issue of the journey itself. We would need to travel many millions of times faster than the speed of light to travel from star to star in 30 seconds or less. I am aware of all the cliche star-stretching-warp effects of Star Trek, Star Wars and the rest. Since our journeys would be of a much smaller distance, you wouldn’t necessarily pass any other stars on the way. For example, on the 40 trillion kilometer journey from our sun to the nearest star, Proxima Centauri, you would pass by no other (known) stars. This means the star-warp effect couldn’t effectively be used. So I ended up doing something more like a dust-warp effect which is like the better known starry version but less bright. Simply put, they are speed lines. Without it, you just don’t feel like you are moving anywhere at all. I also added some subtle nebula graphics along the way so you feel like you are passing through the occasional sparse dust cloud. They help to imply forward movement.

Full of Stars
The position of the stars was the starting point. Using the HYG database from Astronexus, I was quickly able to plot the stars as 3D points in space. The database provides the star positions as right ascension, declination and distance so I did a bit of preprocessing of the data to convert it into simple xyz coordinates. Since this project was meant to showcase the stars nearest to ours, I thought I would cull the list down by excluding any stars that were many hundreds of light years away. This brought the number of stars down to just over 75,000.

I then divided the list into three different categories: faint, bright, and main.

The faint stars are the ones that are dim and too far out to be larger part of this experience. They would be represented as low alpha GL_POINTS and they are there to just help the background seem less empty. They are rendered with their spectrum information but otherwise, they have no dimension. There are 71,678 of these faint stars in this project.

The bright stars are the stars that are closer and brighter and therefore deserve extra attention when it comes time to render them. They do have dimension and are rendered as a pair of overlapping quads with gradient glow that fades in as they get closer to the camera position. There are about 3600 of these bright stars in this project.

The main stars are the ones you visit. There are 34 of these systems (many of the stars we know are actually double or triple stars so despite there being 34 systems, there are a total of 48 individual stars). These are the ones that require special treatment since they will be viewed from much shorter distances than the rest of the stars. Some of these main stars are large enough to span over half the height of the display. They should be varied enough to not feel repetitive, they should have some subtle texture on their surface, they should feature an intense glow, and most importantly, they should feel energetic. This last feature was certainly the hardest to pull off and I ended up trying out a few different methods before settling on the final effect.

The Journey

One of the things we wanted to add to this screensaver experience is dynamically calculated data about the journey itself. Since this is meant to put as much focus on the journey as the destination, we came up with the idea to compare the trip to things the viewer could more easily wrap their heads around. We calculate things like how long the trip would take by car, 747, Space Shuttle Atlantis and Voyager I. We also mention how many generations would be born during a trip of that duration and even the volume of air each person would require to take the trip if they travelled at the speed of Voyager I.

The Corona
Coming up with an interesting corona effect was an unspoken goal for this project. My first attempt was not unlike what I outlined at Eyeo Festival 2012 when I showed a breakdown of how to make a star. However I quickly became annoyed at how unrealistic the particle emitters looked so I scrapped those right away. For a while, I settled on just using a few different corona images overlapping and rotating in different directions but this too felt too rigid and there was no sense of any energy emanating from the surface of the star.

I then began to tackle the problem with geometry. I built a front-facing disk whose inner diameter was equal to the star’s diameter and I animated a texture away from the star center. This solution was decent. I considered adding more geometry in the form of coronal flares but never followed through with it. So I left the coronal disks in place and moved on to a different problem.

The night before I was to be completed, I stumbled upon a GLSL project that was posted to ShaderToy by Trisomie21. It was a simple fireball effect written entirely in GLSL (WebGL enabled browser required to view). It seemed like something I could definitely use so I started poking at the code and eventually realized that I could add an additional noise variable to adjust the resolution value passed back into the noise function which would create some nice whirling loops which looked not unlike some videos I have seen of the corona of our sun. The following short video shows the corona effect, and eventually the other bits are turned back on to eventually show the final effect.

As a challenge to myself, I started playing around in ShaderToy to see if I could also create the solar surface entirely in the shader without having to introduce a 3D model of a sphere. This is as far as I have gotten with it. The code was getting really messy so I went ahead and saved it but I am happy with the progress. Main Sequence Star shader (WebGL enabled browser required)

So there you have it. You can download the screensaver from the 100 Year Starship site. A donation is required to get to the download page but you can suggest whatever donation you feel comfortable with.

I few weeks ago, I was fortunate enough to get ahold of a developer version of the soon-to-be-released LEAP Motion Controller. If you haven’t read up about it yet, you could probably just think of it as a Kinect-like device that sits on your desktop and points upwards. It has very small footprint as you can see from the photos and runs at a fast clip. It is able to give you fingertip and hand position data at up to 120fps, though for my needs 60fps was fine. The lag is minimal and the makers herald it as having sub-millimeter accuracy.

Since this device is still being developed and isn’t ready for the masses just yet, I will hold off on any formal review of its capabilities. The developers are very good about releasing regular updates and each one of these updates usually ends up addressing a concern or request from the testers.

I have started making a handful of different demo apps, but I am only featuring two here for now. The first is LEAP Flocking.

Those familiar with my work know me to be fond of flocking simulations. The first thing I did when I got the device is to dust off some old fish bait-ball code. I optimized it further by pushing the work load to the GPU which allowed me to go from approximately 500 fish to around 5000 and still hold at a steady 60fps on my laptop. The LEAP version of this code assigns a point of interest to each of the finger-tips that the LEAP device detects. If your movements are slow, the fish will be drawn to these glowing tips. Move your fingers too quickly, and the fish become frightened.

The second demo app is called LEAP Gravitation. My main goal for this app was to showcase the accuracy of the LEAP finger-tip and palm data. The LEAP provides a great deal of information about what it sees that will allow you to reconstruct a hand skeleton with relative ease. By knowing the finger-tip position and direction along with the palm position and orientation, you can easily reconstruct a representation of the hand.

Once I had the hand working, I decided to exploit one of the early extras which was added to the SDK. You can now query the position and radius of a sphere that would fit in your hand just touching the palm and finger-tips. When you put your hand into view of the LEAP and you curl your fingers a bit, you create a small sphere that has a low gravitational influence. If you straighten your fingers, the sphere gets larger and its gravitational influence increases. Floating throughout the space are hundreds of smaller balls. They are drawn to the sphere in your hand. The longer your hand stays in view of the LEAP, the more spherical particles will attach themselves to you.

Here is a short reel which showcases these two demo apps. I hope to add a few more examples in the coming weeks. Special thanks to the friendly people at LEAP for getting me early access to one of the developer devices. Additional thanks to theStark for granting me permission to use a snippet of a track off his album, QUADRATIC (available for download at theStark.com).

UPDATE: The video of the presentation has been uploaded courtesy of Eyeo.

Technical notes
• All of these projects were made with the C++ framework Cinder which I work on with Andrew Bell, Hai Nguyen and Steven Schieberl.
• I have released the source code for all of these projects (except for Collider which needs a solid overhaul). Github.
• With a couple exceptions, these projects run at 60fps on my Macbook Pro.
• I used Screenium to capture videos of these demos running in realtime (except for Forces and Collider, which both are realtime projects but need optimization). Screenium is pretty great but there are still some dropped frames. I went with Screenium instead of rendering out a videos because most of these projects also have an audio component.
• I used FMOD for the audio. This was my first attempt at doing audio playback of sound effects I either created, or downloaded from Freesound.org. Special thanks to freesound users HerbertBoland, pushToBreak, totya, halion, and smokeyvw.

Introduction
On June 5th, 2012, I spoke at Eyeo Festival 2012 in Minneapolis. For the talk, I created 12 new projects. The theme present in all of the projects was the Cornell Box. I wanted to try something different from my usual way of doing things. Usually I focus on the content and neglect the environment so all of my work usually floats in an infinite black or white void. This time, I would create an environment in which to stage my experiments and go from there. This environment, or setting, should be simple. Something I know I could do well as opposed to something more complex like a full terrain engine which I might be able to pull off but it would be sloppy at best.

I created a room. A simple grey box with a full ceiling light panel. In this room, I can control the lighting, gravity, and playback speed of time (allowing me to speed up the simulations or even make time freeze). The camera in the room can be moved around to offer different views of the content which would come later. My goal was to make the room as fully featured as I required without it getting to overloaded with excess code. From there, I would begin building out some content all based on the themes of particle physics, cosmology and collective behavior.

Big Bang
The first project in the series is an artistic interpretation of the Big Bang. I wasn’t sure how best to approach this project and there were a few false starts. I eventually decided to stop taking myself so seriously and to try something more abstracted from what my mind wanted to picture. I ended up going with the ultimate surprise party. You turn on the lights to the room and the center explodes with party balloons, confetti and streamers, to the audio of people yelling “YAY” and applauding the ultimate beginning of everything.

The question I was asked most often after my talk is how I got my balloons so shiny. One of the benefits of putting the content in a box is I can easily make a cube map that looks like the room and apply it to the objects in the room. This way, the balloons have a realistic environmental reflection that looks like ray tracing (though the reflection of other balloons is noticeably absent).

Matter
This was actually the first project I started working on in this series. I had recently been reading about how physicists are still uncertain why there was an imbalance of matter and antimatter when the universe was formed. Scientists believe that for every 1,000,000,000 particles of antimatter, there existed 1,000,000,001 particles of matter. And this extra 1 part per billion of matter is what makes up everything in the universe: every star, every planet, every life-form and every mote of dust (to borrow a favorite phrase from Sagan). Some believe the dark matter or dark energy might be the signs of the presence of antimatter but I think this is considered to be most likely incorrect. With this project, you put a quantity of matter and antimatter in the same room and see what happens. The results is somewhere between cute and violent.

Annihilation
This is a companion piece to Matter. In this version, the reaction is shown in macro-scale with one of each type of particle. Once you turn out the lights, the antimatter begins to reach out towards the matter. Whenever the antimatter ‘tentacles’ touch the dense spherical surface of the matter particle, there is a spark and tiny compression wave. If the core of the antimatter touches the matter, they annihilate each other and the simulation resets. I am drawn to visualizing these incredibly small particles because they cant really be described in terms of what they might look like. They are smaller than the wavelengths of light required to view them so any representation would be subjective. I prefer to think of my antimatter as being overly needy.

Bubble Chamber
My first exposure to the aesthetic possibilites of physics was when I saw my first images from experiments conducted in bubble chambers. I loved those beautifully etched spirals. This project brings that experience into the Room. As particles shower through the space, they have a chance of experiencing a collision with another particle or (imaginary) particles of dust. Because I am impatient, I added the ability to put a bunch of moths into the space in order to facilitate collision events. When a particle hits a moth, it shatters into component particles which go spiraling off until they meet a wall or wink out of existence.

Forces
This is a variation of a project I made a couple years ago called Addition/Subtraction. It is an attempt to show how the presence of forces can warp the space around it. This warping is made visible by the trails of particles as they move through the space.

Ant Mill
I am highly disturbed by the phenomenon of the ant death spiral, or ant mill. I have not witnessed one so I tried to make it happen through simulation. I created a system where ants move around in search for food while leaving behind pheromone trails to help them find their way home. The hope was that eventually a group of ants would get cut off from the main trail and form this death spiral. Due to the relatively small size of the room, and a lack of simulated flash floods or other disasters, no such death spirals appeared. However, it was a great exercise in trying to simulate the behavior of ants which I had not attempted before. Eventually, I got so annoyed with the behavior of these simulated ants, I decided to surprise them with a piece of Anti-Food. And we already know what happens when Food and Anti-food collide…

Shockwaves
I recently decided to tackle how to introduce shockwaves into my work. I was used to thinking of an explosion as an incredibly short event. When the explosion happens, you spawn a bunch of smoke or fire or spark particles and that is it. With a shockwave, the explosion needs to stick around for a while. So I created a shockwave class which has a position, radius and intensity. As the radius increases the intensity decreases. As the shockwave expands, all objects check their distance to the shockwave center. If that distance is close to the radius of the shockwave, you push the object away from the shockwave center and the strength of that push is related to the intensity. Once the intensity becomes negligible, you remove the shockwave.

Flocking
I have been working with flocking simulations for a while. This version is the next iteration in my tutorial on how to do particle flocking in Cinder. This version pushes the flocking logic to the GPU which allows you to get away with 4x as many particles in realtime. I also added some glowing bait spheres which attract the flocking particles forcing them into natural looking vortices. They also have to avoid the predators who are forever hunting.

Repulsion
This piece is a bit of a mystery for me. My experience with GPU-based physics calculations is still very much in its infancy. All I wanted from this project was do basic sphere-sphere collision and response but have all the math done on the graphics card. I thought I knew what I was doing but when I finally ran the project, I realized quickly that I had a bug. The particles would clump together then they would inexplicably form these tendrils that reached out into space and whirled and stretched and though I tried to figure out why it behaved this way, I eventually abandoned trying to find the bug because the behavior was so novel. It almost felt like I had accidentally created a life-form.

Terrain
I created a reaction diffusion piece a couple years ago that used wind as a parameter. The result was something that looked like dune migration. I based this project on that original reaction diffusion calculation and turned it into a 3D terrain. The sphere in the scene helps to facilitate the creation of these dunes and its position also dictates which way the wind blows. The results are quite organic and look very much like dune patterns you would see in large deserts.

Star
I was recently asked how I made the stars in the Planetary app I created with Bloom. This project is a tutorial that shows the steps for creating energetic balls of light. I break it down into 7 steps and use the result to show the scale of some stars in our galaxy.

Catalog
I downloaded a database of 116,166 of the closest and brightest stars to our sun. After plotting these points in space, I created a system where I could look at any star from any other star. The initial inspiration was based on wondering what the night sky would look like from Gliese 581g, the first exoplanet thought to be a candidate for supporting Earth-like life.

Collider
One of the reasons I decided to make all these smaller pieces is because I had been working on a larger project that really started to get away from me. I was so busy adding in visual extras, the whole code base became quite confusing to me which made it much harder to fix the foundational issues regarding the physics and collision detection. These smaller projects were a way for me to better understand the individual pieces that I wanted to work together in the final project. Below is the work in progress video I showed at the conclusion of my talk at Eyeo. I plan on rebuilding it from scratch while paying special attention to the foundation so I can hopefully avoid similar issues as the project continues to grow.

Audio by Hilmar Örn Hilmarsson and Sigur Ros.
Fer<eth> from the album Englar Alheimsins

This project revisits the code originally created for the Written Images project. There were some aspects of the code which always frustrated me that I didn’t know how to address at the time. In reworking this code, I finally figured out how to deal with large numbers of point lights in GLSL shaders as well as getting a better handle on dealing with the magnetic field itself. Previous versions used a rotation matrix to try and whip and spin the magnetic field into interesting forms. This was not an ideal solution. So I gutted the magnetism code and restructured it so that I wouldn’t need to rely on tricks. The results are quite promising.

The project begins with a simple central sphere. Onto the surface of this sphere are placed a large number of charged particles. The charge is random so some are negative and the rest are positive. After giving these forces a few seconds to spread out or coalesce, I freeze the simulation and have each charged particle emit a series of weaker particles whose direction of travel is wholly dictated by the interacting magnetic field surrounding the form.

While the weaker particles trace out strands within the magnetic field, crystalline structures begin to form around the surface-bound charged particles. The stronger the charge, the faster the crystal growth.

These four prints are each a unique 1 of 1. They were created for the GAFFTA benefit gala.

Crystalline was made with the Cinder C++ framework.

I wrote about the process for making the application over on the Bloom Studios blog.

The application, called Planetary, is available at the App store for iPads. In the first few days after its release, it reached #1 in many countries. The feedback has been fantastic. An update is in the works. Below are some screengrabs showing off the new look for the application. Check back at the Planetary website to find out more details about when the update will be released.

The new galaxy view shows you all the artists in your iPad. Here, I am filtering by first letter.

The Flashbulb star and its 6 orbiting planets.

Closeup of the Flashbulb planet "Red Extensions of Me" which features a bright red album cover. The planet and its moons derive their surface patterns and colors from the album art.

Image of the revised eclipse effect.

Looking at the Aphex Twin planet for the album "Selected Ambient Works 85-92". The track Heliosphan is the track I have played the most on that album. Accretion disk shadows have been added in the new version of the app.

Shortly after posting “Getting Stuff Done” Kinect experiment to Vimeo, I got a couple comments from people saying it would make for a nice Aphex Twin video. The project makes use of the Kinect to obtain depth information as well as the infrared video feed. By saving previous frames to an array, I was able to create a time echo effect where your movements are repeated by additional copies of you.

Because the video feed is mapped onto geometry that is created with the depth map information, it was a step better than just doing standard video frame overlays. Now, you could walk backwards and hide in the shadow of an earlier copy of yourself. It made for some amusing evenings of flailing my arms about in my apartment with the lights off.

Surprisingly, I received an email a short while later from WeirdCore who was in the process of working on live visuals for Aphex Twin’s new years eve show in Rome. He had shown some of the videos to Aphex and they were positively received.

I ended up creating a Cinder app that had 10 preset modes and various parameter controls that WeirdCore could interact with in realtime during the show. The modes ranged from simple 3D point clouds to variations of Body Dysmorphia.

WeirdCore did a fantastic job integrating the Kinect content with the other feeds consisting of visuals created with QC, MaxMSP/Jitter, VDMX, and v002. Here is some footage of the last bit of his set.

Nice to see Aphex do the head circle maneuver near the end. I do the same thing!

Many thanks to Richard James (Aphex Twin) and Nicky Smith (WeirdCore) for the opportunity.

Aphex Twin Wikipedia page
WeirdCore.tv
WeirdCore on Vimeo

Body Dysmorphia (or Body Dysmorphic Disorder) is a disorder in which the affected are “excessively concerned about and preoccupied by a perceived defect in his or her physical features”. In this realtime interactive project, I manipulate the data from a Kinect sensor and create a type of mirror which can make the viewer excessively puffy or excessively emaciated.

This effect occurs in realtime and does not require any environmental preparation (such as controlled lighting or uncluttered background as is usually the case with standard webcam-based video mirrors).

I obtain the depth map from the Kinect and create a rough normal map. Using the normal map in conjunction with the RGB webcam data also coming from the Kinect, I am able to get an idea of how the surfaces in the scene are oriented. This is a surprisingly powerful bit of information. From it, I am able to adjust the lighting in the virtual scene creating realistic highlights and shadows in the virtual scene that do not exist in reality.

More interestingly, I am able to push the surfaces out along the corresponding normals in order to make the subject fatter or thinner.

If you do not blur the depth map before calculating the normal, the resulting 3D model will be very bumpy and disturbing.

A couple of blur passes on the depth map will create smoother models.

More information about Cinder: http://libcinder.org
More information about Kinect CinderBlock: http://forum.libcinder.org/#Topic/23286000000453029

In the following images, I have created a dynamic metaball from a group of flocking particles. All of the rendering was done from within Cinder using GLSL shaders. Eventually, I saved out the form as a .obj file and then converted it to a .stl using Blender. I could now have the .stl printed in 3D to create physical sculptures of virtual scenarios.

Zoë composed a new piece for this collaboration. It is roughly 14 minutes long and builds a wall of sound that is eventually peeled away piece by piece until you emerge back at the beginning. The song begins and ends with the same haunting two notes.

I decided to depart from my usual process of relying on microphone input to create the visuals. I wanted to do all of it by hand. There would be no audio reactive landscape or beat driven content. This decision wasn’t entirely without outside influences. Stringed instruments, particularly ones played by bow, are notoriously difficult to analyze. It is very difficult to pull out any beats from a wall of sound. Also, the piece Zoë composed was much more ambient than previous works. It was very dark and somber so beat-detection would have been useless. There were simply no beats to detect. The visuals would have to be controlled by hand.

The rough outline of the piece is as follows:
• Create an algorithmic terrain and meandering path
• Create sky with a dynamic day/night cycle
• Create boulders and grass
• Fill the landscape with towering trees
• Return to barren terrain and meandering path

The Terrain and Path

The canvas for this project is a smoothly winding path through a subtly changing terrain. The path is a combination of sine waves. There is a main wave with a very low frequency and on top of that is added a higher frequency wave with a much smaller amplitude. Even though the wave is very repetitive and predictable, it ends up working fine for my needs. I didn’t want to draw too much attention to the path by making it noisier, but I also didn’t want a straight line.

The terrain is also very subtle. It is a combination of sine waves but with a little Perlin noise influence. The Perlin is scaled up very high so you never really feel the Perlinness of the algorithm. Like the path, I wanted something between boring and chaotic. Over the course of the 14 minute piece, the altitude of the terrain changes. By the end of the piece, the changes in elevation are about 10x what they are when the piece begins.

The visual of the path is very simple and is done in a shader. I use the distance of every pixel to the path to create a subtle gradient and then add some striping to make it stand out. The camera also follows the sine wave equation for the path, with some variation. It is allowed to swing a bit outside the path, but never wanders too far away. The direction the camera faces is also automated, but can be controlled with mouse input as well.

Here is an early study showing a wireframe view of the terrain.

The Sky

The sky proved to be the most interesting challenge. I wanted to implement a smooth night-to-day-to-night cycle which is something I had not attempted before. I have done some simple sun as light source studies in the past, but never tried to create a proper dynamic sky-box (or dome). It ended up being fairly simple once I came to grips with the fact that dynamic cloud movement was beyond my skill level for now so I should just pretend it wasn’t even an option.

I purchased some spherical textures of skies and picked out four which I could cross fade as the sun rose and set. If the sun is below the horizon, I would use the night texture. If the sun was high in the sky, I would use the blue sky texture. For other sun positions, I would linearly interpolate between the 4 textures to create a smooth transition from day to night.

Sadly, this ended up looking rather blah. It all felt very static and felt like a matte painting. It was finally time to implement my first dynamic bloom and God ray effect. Again, this ended up being fairly simple since I had already learned how to deal with FBOs (using Cinder, FBOs are very easy to implement and control).

From what I read online, bloom effects are created by rendering the lighting to a separate FBO and then overlaying a blurred version of the FBO on top of the fully rendered scene. I decided to simplify things a bit and just render the whole scene to a separate and smaller FBO. After a blur and contrast bump, I then overlay the bloom FBO back onto the scene. This helped me solve (i.e., ignore) the bloom occlusion issue. And it turns out, this is most of what is required to implement the beautiful (i.e., easily abused) God ray effect.

If you render the bloom FBO multiple times with an increase to the image’s scale with each iteration, you get a fairly nice atmospheric ray effect. It isn’t perfect and unless it is done on the GPU it can be quite slow. Since I was already doing the bloom overlay in a shader, it was easy to throw in a for loop, take the light source location into account, and render the bloom FBO 50 times per frame with the appropriate scale increase. This really drove home how impressive shaders can be. To do a fullscreen post-process render 50 times per frame and still be able to achieve real-time framerates… wow. GLSL, I love you very much.

The God rays allowed me to pay less attention to the realism of the sky and focus more on the fantasy of my world. A failed attempt to create a realistic sky is much less compelling than a successful attempt to create a fantastic, though unrealistic, sky. These crepuscular rays also helped me with the realism of the elements which would eventually populate the terrain. A bright bloom with corresponding rays over mostly silhouetted content can be quite compelling.

I also ended up using a 1D gradient texture which allowed me to fog my foreground assets to a reasonable color as the lighting changes. During the day, the trees and rocks fade over distance to a blueish hue. At sunset, they fade to a pinkish-orange. And at night, they fade to a deep purple. I do these color interpolations using shaders.

Video of the day-night sequence.

The Boulders

The super-ellipsoids I used for the boulders were the only part of the Fuji project that I recycled. It is such a nice solution, I just had to use it again. The boulders first came about when I saw Andre Michelle use a superellipsoid(Wikipedia) in an audio visualization. And since Wikipedia makes all math look frustratingly difficult (in my opinion, of course), here is a nicer article on the superellipsoid by the creator himself, Paul Bourke.

For my boulders, I create a superellipsoid and save all the vertex positions to it’s own VBO. This way, I am only calculating the positions once for each boulder’s lifetime. Since I am storing the positions, I can add some Perlin noise to the vertices in order to randomize the forms a bit without much strain on the processor. Voila! Instant boulders.

The Trees

Along with water, smoke, and fire, trees are some of the more difficult things to attempt with code. Trying to code realistic algorithmic trees is quite the rabbit hole. Once you start, you will never be content with how they look or how quickly they render. Finding the right balance between speed and realism is key. Sure, you can run a few million iterations of some L-system or branching node network code, but good luck getting even one to render 60 frames a second. The flip side is to just stick a billboarded tree graphic into your scene. However, any scrutinization and your laziness will become very obvious.

I decided to only be semi-lazy. I used two styles of trees: large and 3D(ish), or small and flat. I didn’t have to worry about full access to all of the terrain from all angles. In other words, this wasn’t a free-roaming first-person-shooter. The camera always moves roughly forward and faces roughly forward.  This made the flat trees more realistic because nobody was ever viewing them side-on and I also didn’t have to worry about the uncanny effect that occurs when a single image of a tree is made to always face the camera.

The small trees were made with two quads in a right angle cross. I used 16 different images (purchased from a texture site) in one large texture and just chose a random set of UV coordinates for each tree. The large trees needed to have a bit more character so I ended up using a perturbed cylinder to create the massive trunks and stuck a two-quad cross on the top half. I used images of red woods for a couple reasons. They are the type of trees that exist near Zoë’s home, and they have a very straight trunk and high canopy which made my job much easier. And let’s not forget the God rays… those things shining through the canopy with a bright bloom can really bring some much needed drama to the scene while obscuring just how the trees were made.

In order to bring the trees in without any pop, I create them at a miniature scale in the distance and make them grow quickly to full size (approximately 1 to 2 seconds of growth time). Though not realistic (you can obviously see them grow into place), the effect was quite magical.

The Details

God is in the details, so it is said. I like to look for God in particles. This project uses a few basic particle emitters which really help establish the mood of the piece.

• Glowing dust
What forest scene is complete without a bunch of floating glowy things. Whether they are stand-ins for fireflies or dust motes, they push the 3D illusion of the space while adding some interesting visual noise.

• Ground mist
Because the project was to be projected, I couldn’t rely on the subtle forest-at-night lighting to come through. I didn’t know ahead of time what the ambient lighting situation was going to be so I added a ground mist toggle which I could turn on to add some nice smoky effects to the scene. This would help add contrast to the piece by pushing the silhouetted look of the trees. Plus, I had recently played Limbo and I was really impressed with the look. It was definitely an influence on the final piece.

• Shooting stars
The night sky was a high resolution photograph of the Milky Way overlayed on a more traditional field of bright stars. To increase the fantasy aspect of the scene, I added a large number of meteorites to help draw attention to the night sky. They were very simple, very short lived, line segments which emerge from a specific radiant and shoot off in a random direction for 5 to 15 frames.

• Random trails
The forest scene also had some particle trails which generally hovered over the path and would travel away from the camera. They moved along sine waves and trailed a couple dozen previous positions that were connected with a quad-strip.

Control

Controlling the application was very simple. It was all done manually. I created keyboard commands which would trigger various objects. If I press ‘1′, the skydome would toggle on or off. If I press ‘2′, the terrain would toggle. ‘3′ is for grass. ‘4′ is for trees, etc. Triggering these effects required me to do some camera manipulation to avoid pop-in of the assets. For example, I didn’t want the terrain to just appear out of nowhere so when it was time to bring in the terrain, I would aim the camera up at the sky and then press ‘2′. This way, the terrain would appear while the camera was looking away. I could also do this when it was time to toggle the sky. I would just look down at the terrain and then press ‘1′.

The trees and boulders were a bit easier to deal with. The key presses would trigger the creation of a bunch of objects rather than just toggle them on or off. The trees are always on, but until I press ‘4′, none exist onscreen. Also, the trees would scale up over time and the boulders would rise up out of the ground so it didn’t matter where I was looking when I triggered them. This way, I got to play in the assets.

Finally, there was a minor manual audio effect which I didn’t really use much but I wanted to mention it here because I think it is a fantastic effect. It would be better suited to a project with an intense bass beat. Each press of the space bar sends out a shock-wave through the terrain which also effects the boulders and trees.

Final piece

“Into the Trees” was performed in the South Hall at the San Jose 2010 Biennial. The venue was brilliant. Inside the South Hall tent, a huge Mad Maxian drive-in theater was created. The audience sat in rusted, dented, and crashed cars, sometimes stacked 2 high. The cars were often missing tires and windshields and some even had empty space where the engine should have been. The drive-in screen was beautifully built, complete with the name EMPIRE written across the top in 3D block letters, but the M and trailing E had long since fallen off. It felt like a post-apocolyptic theater with an audience of survivors trying to rebuild after the bomb, attempting to distract themselves from the destruction by creating impromptu performances about creation. It was really quite moving. Many thanks to the ZER01 team for creating such a beautiful venue and additional thanks to Jaime Austin, Steve Dietz, Craig Hobbs, and of course the extremely talented Zoë Keating.

Photo by mangtronix

Photo by creativeartsmag

Photo by creativeartsmag

Photo by rich_lem

Near-final render.

Joe Ford recorded the San Jose Biennial performance and put the video up on Vimeo. You can watch it in it’s entirety below.

One of my original flocking sims featured birds flying around in a 3D environment. I took the easy way out and ignored any collision detection. The birds were pushed about by a combination of gravitational attraction (to keep them fairly centered on the screen) and Perlin noise (to give the illusion that the birds were moving around in a fairly organic manner). The follow-up to that version introduced an extra force in the form of magnetic repulsion. This gave me collision avoidance but it was not very graceful.

The Iain Couzin presentation, featured on the WNYC RadioLab blog, presented the problem of flocking in a beautifully simple 3 rule system.

1) Don’t get too crowded.
2) Don’t get too isolated.
3) Swim with neighbors.

(An optional 4th rule is that you should do what it takes to flee predators)

This is a bit of an oversimplification of what Professor Couzin outlined but it gives a nice insight into the simplicity of these complex-looking systems. Based on these three rules, I pretty much had the first two down. The repulsive force kept the fish/birds from getting too crowded. The attractive force kept the fish/birds from getting too isolated. But I never paid much attention the rule 3 which is why my early studies often looked like chaotic writhing balls instead of what is experienced in nature.

The code is pretty straightforward. Find the distance between me and each one of my neighbors. If this distance is really large, move towards the neighbor. If this distance is really low, move away from the neighbor. If the distance is neither too close nor too far, then allow my velocity to become influenced by the neighbor. The amount of influence each fish has on each other is a curve. If the threshold of attraction is a distance of 50, then a distance of 51 would involve a minimum of influence compared to a distance of 200 which would be a stronger influence. I should probably limit this distance so that a fish 5000 units away would have no influence at all, but I need to do more research to see what is being observed in nature.

Future versions of this project will incorporate the dynamically resizing zones, as mentioned in the Ian Couzin presentation. For now, the zone thresholds are hardcoded. Also, there is no sense of leadership (though I don’t know if such a thing exists in the real world equivalent) and each fish is being influenced by every other fish in the simulation (which is probably not how these things actually work). What I will likely try next is to have each fish only being influenced by its 7 or so nearest neighbors.

Here is a video of the simulation. This version features 2500 fish and 2 predators. It runs in real-time.

This version uses 25000 fish and 2 predators and does not run in real-time.

For my submission, I decided to create a project based on the phenomenon of Coronal Mass Ejection. The CME is a massive burst of solar wind, often resulting in the ejection of solar material. The image below, from NasaImages.org, depicts magnetic field lines emanating from the surface of the sun. I decided this would be my primary inspiration.

For my CME simulation, I decided to start simply. First, I draw a Sphere in the center of the screen. This main Sphere has a far reaching but fairly weak negative charge. On the surface of the Sphere, I add a few dozen to a couple hundred smaller spheres. Each of these smaller spheres contains a much more confined but stronger positive charge. Because all the smaller spheres have the same polarity, they push each other apart. I let them drift on the surface of the larger Sphere for about 2 seconds.

Once the 2 seconds are up, I lock the spheres into place and flood the simuation with a few hundred to a few thousand tiny particles. These particles are at the mercy of the magnetic field. They are immediately pushed away from the smaller spheres. As they move further away from the center, the positive charge from the smaller spheres becomes overwhelmed by the negative charge from the central Sphere and some of them are pulled back into the center.

Early study.

Early study, with spikes which I quickly abandoned.

Nearing the final look.

The final decision was based on each image taking up a full spread in the book. Since the image would be split down the middle with a seam, I decided I would render two spheres per image. However, the 15-seconds-per-image limit was still an issue. I was already at 15 seconds and the thought of bringing it down to 7.5 seconds per Sphere was not acceptable. After 7.5 seconds, the particles have barely had any time to spread out and none of them would be able to make the round trip out and back to the center in that short amount of time. So I decided to cheat.

I would create the Sphere and when it was time to save out the print res version, I would show it from the front on the left half of the image, and then swing the camera around to the back and use that for the right half of the image. Two for the price of one!

On October 21st, the 42 finalists were chosen from the original pool of 70 submissions. I was thrilled to have made the cut. I am very much looking forward to getting my hands on a few copies. Once the book is available, I will add the purchase link here.

The project went through many different looks. My original idea was to try to make the gravitational and orbital forces mold the digits out of a sea of particles. The initial studies were pretty positive but the leading ‘1′ digit was proving to be problematic. The ‘8′ and ‘0′ were fine because they were simple forms made of circles. Spinning an attractive force did a good job of creating the rounder digits but I was struggling with the ‘1′.

I went with an entirely different approach (which was eventually rejected) for a while. I created gravitation depressions where the digits forms were and flooded the scene with force particles. The forces repulsed each other but were drawn into the digits and spread out to create the 18.08. I then flooded the scene with particles and let them do what the forces instructed, much like in the Addition Subtraction project. I was rather fond of the result but it deviated from the original inspiration too much so we ended up going back to the original idea.

I ended up turning the ‘1′ into the particle emitter. It made the most sense to do it this way. Even though I was bothered a bit by segregating the digit from the other three, the end result came out nicely. A glowing series of particle emitters aligns to form the ‘1′ and as the particles spread across the screen, they are pulled and pushed into whirling black holes which end up forming the other three digits (and the decimal).

Final print piece.

Early test render. Audio by Tim Stutts.

Final render. Audio by Tim Stutts.

For this project, I created black and white landscapes that can be explored through keyboard input. Some of the landscapes are simple expansive horizons, others are complex layered caverns. Some of them are static, others morph as the camera comes near.

And somehow, through the alcohol haze, they managed to put together a really nice interview for Juxtapoz Magazine that spans 12 glossy pages. So to them, I say Thank You.

Aside from the near all-cover set, the other curious thing about this tour is he performed without the aid of drums or guitar. It is him, a conductor, and a backing orchestra.

Photograph by Robert Sinclair.

I was assigned the cover of Radiohead’s ‘Street Spirit’. And I had a little over two weeks. Because the short deadline, I decided to do something with reaction diffusion. I love reaction diffusion. RD is my new additive blending. You are probably familiar with RD from the amazing 3D type piece by Karsten Schmidt (Toxi).

Since PG was performing with an orchestra, who rehearses to a click track, I was able to get a recording of the song more or less as it would be played live. This allowed me to code in a few tone changes and transitions manually so the piece would have some relevance and variety throughout the length of the song.

Photograph by York Tillyer.

While PG sings, an ominous and slightly off-putting reaction diffusion pattern spreads and pulses across the background. When he reaches the climax of the song and the key changes to hope, the diffusion dissipates and is engulfed by a bright light.

Photograph by Robert Sinclair.

Many thanks to the Georgina Sinclair, Rob Sinclair, and Peter Gabriel.

I showed five different pieces, ranging from prints to sculpture to an interactive installation. All of the pieces were related to my exploration with magnetism and mathematical phenomena.

Selected Works
Made with Processing and Cinder (C++ framework).

One of the 46″ flat screen displays shows a 22 minute reel of some older video projects. Most are audio reactive. Included are a Magnetosphere rework, Fuji, Cymatic Fluid, Solar with Lyrics, Audio-reactive Landscape, Relentless, and Branching.

Moment of Fission, series of 6 prints
Made with Processing.

For this series, I placed anywhere from a dozen to a few hundred charged particles into a small space. All of the particles want desperately to push away from every other particle. A couple frames after they explode outward, I render out the results. The prints attempt to visualize the act of atomic fission.

Dodecahedral Variations

When I was planning content for the show, fellow GAFFTA alum Ryan Alexander suggested I add some sculptures to the show. In specific, the magnetic solid sculptures I have been making off and on over the last couple years.

These forms are created with cylinder magnets, spherical magnets, and ball bearings. Magnetism is the only thing holding the forms together. They are fairly fragile and picking them up will likely crush them. All of the forms I created were variations of the 12 sided dodecahedron. This particular platonic solid seems to be the form the magnets are happiest with.

The largest structure required a temporary scaffold of sorts. Without the scaffold, I would be unable to complete the form without it succumbing to gravity. This structure took me a few hours to create and on the day of the show, it collapsed on the way to the gallery. I ended up rebuilding it the day of the show.

Quantum Decoherence
Made in Cinder (C++ framework).

This pair of prints was made for 40″x30″ backlit lightboxes. While exploring ideas for the interactive piece, I started playing around with Gray-Scott reaction diffusion. The formulas for reaction diffusion are surprisingly easy but they can create a large variety of beautiful organic patterns which resemble many patterns found in nature.

The aesthetics are based on the results achieved with scanning tunneling microscopes. It is intended to be viewed as a visual interpretation of the wave-particle duality of light and matter. The individual nodes do not exist separately from each other. They are simply different densities of the same ether. They grow and turn white as a wave, emanating from the center, passes by.

Reactive Diffusion, interactive installation
Made in Cinder (C++ framework).

This installation shows off reaction diffusion in all its glory. A 46″ flat screen display shows a view of the webcam which is pointing at the viewer. It functions as a digital mirror, but the reflection is highly distorted. The entire screen is morphed by a reaction diffusion effect. The reaction flows across the screen and undulates in a intensely mesmerizing way.

Music by Helios (“Backlight” from the album Caesura)

When I created this project, it initially managed to eek out about 10 fps when running at full resolution. It ended up feeling really sluggish. I bought a nice tower PC but it still barely reached 15 fps. My mentor and collaborator, Andrew Bell, suggested I move everything to the GPU because the slowdown was likely due to the way I was abusing FBOs. The original version pulled data off the FBO, modified a large quad mesh, then sent the content to a GLSL shader which handled the lighting and coloring.

The rework now has everything running in GLSL shaders. The huge mesh grid is gone, replaced by just one single textured quad being drawn fullscreen. This brought the frame rate up to 60+ on both the PC tower and my MacBook Pro laptop.

For one week in early March, 2010, anyone taking an evening walk down the 400 block of High Road in Wembley, England, will find themselves under the watchful gaze of Overlord.

This project is entirely about mesmerism. It is my intention to cause passersby to stop and stare up at the 50 meter wall on which the Overlord appears. He stares down at them, surrounded in a swirling reaction-diffusion fluid. He appears indifferent and his facial expression does not change.

To create this project, I stared into a camera for 20 minutes. I used this saved video content as input into a reaction-diffusion project created in Cinder. The project runs beautifully in realtime but it made more sense for the end product to be a simple looping video.

The reaction-diffusion responds to the brightness of the pixels in the video as well as the motion from frame to frame. The motion creates ripples which propagate across the screen and influence the formation and destruction of the RD patterns.

Here is the venue.

View Larger Map

Short render from the longer piece. Sadly, Vimeo wasn’t too kind to the compression.

They asked me if I could make a real-time audio reactive version of Solar. I created Solar in 2007. It originally came about because I was working on a demo for a talk I was to give at UCLA. It quickly became one of my more popular works. It features a magnetism and gravity sandbox where particles react to the intensity of the audio. Creating a compelling real-time version is definitely a challenge. The original Solar render probably ran at about 0.5 to 1.0 frames per second. Getting that up to between 30 and 60 fps is not going to happen easily.

Thanks to some great advice from Andrew Bell, I was able to create a robust version that still had a nice visual density but would still run at a good clip. It involved using instances of the parent sphere and child particles instead of calculating a set of particle positions for each sphere. Since I was definitely CPU bound (the magnetic repulsion calculations can get really heavy really quickly), this turned out to be a great way to still show tens of thousands of particles but only need to do repulsion calculations on a set of a few hundred.

I did a render to some audio from an episode of Radiolab to show how it performs.

The reason the black oil appealed to me was that it allows you to visualize the invisible field surrounding magnets. Its like a three dimensional execution of the iron filings test but much more mesmerizing. I instantly wanted to know more about them. Why do they behave the way they do? Why are they spaced out just so? What insane amount of math is going on behind the scenes?

These questions eventually led me to the study of Cymatics. Below are my first studies for replicating cymatic phenomena using code.

The particle checks the pixel array to see what shade of grey it needs to represent. If it needs to show blackness, it grows smaller and its magnetic charge diminished accordingly. If it needs to represent white, it grows larger as does its charge. Once you throw a few tens of thousands of particles into the scene, eventually they spread out in an organically pleasing manner. There seems to be no patterning, but its certainly not a random spread.

I added an additional feature which allowed the particle to scale down small enough that it becomes invisible to the naked eye. I then allowed adjacent particles to stretch a thin black line between them if their distance is short enough. I changed the background color to grey so that the particles and lines could paint in the light and dark parts of the image.

A couple years ago, I asked my mother about the time we spent there. She laughed and said we had never gone to Mount Fuji. The memory I had was totally fabricated. We were actually in a public park just outside Tokyo. Mount Fuji was visible in the distance. It was not foggy. There was no bamboo. I had apparently made the whole thing up.

In this piece, Mount Fuji always remains far away but looms over the landscape. It is always partially obstructed by bamboo and mist and other weather effects. Sometimes you can only see it because it is silhouetted by the passing full moon. It is a moody piece running in black and white.

For this test render, I used some audio by the insanely talented Zoe Keating from the Ghostbird soundtrack.

This project went through several months of changes. Here is a selection of strange bugs and abandoned visions from early in the project’s lifespan.

If I am feeling really anxious or I just want to be distracted for a while, I play with the magnets.

Everything was going as planned. We were making great progress. Then it happened. Something entirely unanticipated. The more time we put into developing the app, the more it became respectable. Lo, our gutter-born brainchild was maturing! It sprouted wings of respectability and soared. The depths whence it came were now but a speck. It defied our intent and expectations and blossomed into something magical.

So, what does it do? What DOESN’T it do!

How many times have you longed to see the famous neo-classicist Jacques Louis David sing a scat rendition of Mouret’s Rondeau? If you are anything like me, the answer to that question is easily in the thousands. Well, my friends, long no more…

Most of the work I have done in the last month has involved creating suggested sample applications in the spirit of learning the ropes. Andrew Bell has been giving me assignments. First up, create a globe and map earthquake data onto it.

I had done something similar a while back in Processing but my data was limited to California and Nevada. Now I would be working with 7 days worth of data from all around the world for any earthquakes with a magnitude of 2.5 or higher. It isn’t a huge amount but I would have to find ways to deal with the clusters that are associated with any earthquake data visualization.

Creating the actual globe was great fun. I was pointed towards NASA’s Blue Marble project. There you can download Earth textures at astronomical sizes. Some are available at 86400×43200 pixels. I grabbed a color map and a height map. Using NormalMappr, I created an additional normal map from the height map.

The one drawback of the NASA data is the river systems aren’t as prominent as I would have liked. I ended up adding in the rivers and smaller lakes using this image as a source.

As I mentioned earlier, earthquakes come in clusters. The Dominican Republic had a few dozen 3.0M to 4.0M quakes in that week. If I just stuck pins exactly over the epicenter, all of the Dominican Republic pins would be reduced to a single blurry pin which would not give an accurate summary of the area.

I decided to go back to my old friend Magnetism. In order to keep the quakes grouped but individually distinct, I anchored the pin to the epicenter but allowed the other end to drift a short distance away. This distance would be determined by making each pin-head magnetic so that it pushes away its neighbors’ pin-heads.

7 Days of Earthquakes from flight404 on Vimeo.

Magnetism to the rescue, yet again. I gave each node along the snakes body its own magnetic force. Each node repels other nodes. It now is possible for a snake to tie itself in a knot, but it generally will not self-intersect.

Eventually, I stuck a couple dozen snakes in the water, attached their heads to each other, added a sine-based undulation force, and voila! Reasonably ugly jellyfish. They may be ugly, but they are audio responsive.

The nodes arrange themselves in space using magnetic repulsion. Any node can repulse other nodes as long as they are of an equal or higher generation. The node at the base of the trunk repels everything whereas the nodes at the branch tips repel only each other. If you code in some decreasing branch lengths and radii as you go from generation to generation, you will create a nice space-filling tree with no branch overlaps. Sadly, these trees are complex enough to elude a reasonable frame-rate but perhaps with some limitations placed on the number of branches that can be created, and killing off the repulsion after the branches settle into place, they will be swaying in the Perlin noise breeze in no time.

I then tried to place some leaves or flowers on the trees and accidentally changed the perceived scale quite a bit. It no longer seemed like a majestic oak. It turned into a bush, and then eventually, much smaller flowering weeds.

The project went through many different looks. The primary art direction we received was that the end result should be dark and should have some baroque sensibilities. We did our best to hit these goals and we are quite proud of the end result.

Learning Experience
For me, the most rewarding part of this job was that I learned a whole lot. Fellow Barbarian, Andrew Bell, showed me many interesting tricks, not to mention working his ass off to optimize this project so it would run at 30+ fps and also handling the port to C++ for the iTunes visualizer version.

The early versions had the primary tendrils rendered as flat strips that always orient themselves towards the camera so you never (well, almost never) see the strips from the side. This was my way of faking the viewer into believing they were seeing a solid mass. However, this method had its problems. It was impossible to texture map it in an interesting way, and occasionally, the strip would twist a full 360° which added unsightly kinks. Andrew suggested we move to actual tubes instead of strips and I thought he was nuts. The piece was already starting to drag down the framerate and he wanted to add more geometry?? Nuts, I tell ya!!

A couple days later, he showed me the new build. I was floored. Not only did he do what he promised, but he improved the frame-rate in the process AND he added support for shaders. Awesome!

BEFORE:

AFTER:

Suddenly, things got a lot more interesting… more lush. He ended up switching the tendril geometry over to VBOs (vertex buffer objects) which sped things up a great deal. He also pre-cached the texture coordinates because they didn’t actually change once the piece was up and running.

The Birds

The idea of flocking crows or ravens is very much in keeping with the Relentless aesthetic but there was some concern that we were simply repurposing a project we did with Nervo for Fox Movies. We decided to do away with everything we did for Nervo and start from scratch. The original Nervo piece relied on using 160 flat images of a 3D bird in flight. For the Relentless version, we switched over to actual 3D birds (admittedly quite rudimentary). Each bird was made up of 12 polygons. The wings actually flapped and they could be viewed from any angle (whereas the Nervo version worked best when the birds were viewed side-on.

The Background

From the beginning we were instructed to do away with the notion of an infinite black or white space. The entire scene should be filled with content and this proved to be quite the challenge. Originally, we just dropped the whole scene inside a large textured sphere but the end result was lacking. Andrew came up with the idea of using a flat plane as the background but by using some FBO trickery, he was able to make these beautiful flourished textures bleed into view. I still have no idea how he pulled it off but the effect is quite stunning. Later he added a dynamic light source to make the scene feel even more dark and slightly disturbing.

The (faked) Volumetric Lighting

After we settled on the look, we wanted to make it seem like the invisible center of the mass of tendrils was a hidden light source. Earlier versions actually had the tendrils glowing near their base but the look of shadow and silhouette was more intriguing so we killed the glow and added faked volumetric lighting. The light beams themselves behave as if they were magnetic particles. Each one is born, lives for a predetermined amount of time, then fades out and dies. While they are alive, they repulse each other if they get too close. This created a really nice organic movement which made it seem like the movement of the tendrils actually caused the light beams to form or disappear. Combined with the background lighting, the effect became quite engaging.

The Content Creation Tool
Erasmus wanted to be able to create renders from pre-recorded pieces of music so we tweaked the live version to accommodate the loading in of external audio as well as an XML file of adjustable variables. By modifying the XML file, they could run the application and customize their experience. Every graphic element could be toggled individually. So if you want a video with no background for compositing, or perhaps you want to leave the birds out, just tweak the XML and run the application. Additionally, since we were no longer confined to creating an experience that ran live in real-time, you could up the amount of objects as high as your memory will allow. This is how we made the following test render. The bird and particle count was turned up really high, as was the detail on the tendril geometry.

The following video is a test render from The REV content creation tool. Please note this is not an official Relentless video. This is simply a test render we did to make sure the audio responsiveness was as we liked it. The audio is by The Flashbulb (“Six Months Without Light” off the album Kirlian Selections) but be aware that The Flashbulb has no affiliation with Relentless. The chosen audio was simply a test track. I chose The Flashbulb because he is my favorite artist to audio visualize and I love this track.

Or you can view the much higher resolution Quicktime here. Note, it is around 300Mb.

The REV iTunes Version
As I mentioned above, there is a free downloadable version for iTunes. You can find out more at the Relentless Energy site. Have at it!

This is the process video…

…which created this image.

The second iteration involved recreating the Magnetosphere study and using those particles as the source of the ink.

Audio by The Flashbulb (“In an Instant” from the album “Red Extensions Of Me”)

In the summer of 2007, Nando Costa approached Barbarian Group and asked if we would help out with a project his new motion graphics company Nervo was working on for Fox Movies Japan. He showed us the boards and instantly we knew we wanted to collaborate because his vision for this project was quite beautiful and surreal.

What he needed from us was videos of flocking behavior. He had seen the previous experiments I have done with perlin noise flocking and thought it would work well for this project. All he wanted was a couple videos of flocking using a 3D crow (or is it a raven) he would provide. Simple enough. But given the tight deadline, the thought of doing a render and posting it and waiting for approval or changes and then implementing the changes then rerendering and reposting, etc… That process didn’t make sense for this project so we decided to deliver them an application instead.

Using Processing, we started playing around with the flocking behavior to make it more customizable. The original version of the flocking experiment had very few controls and they had to be hard-coded. There was no run-time adjustment. This was the first thing addressed. Several new parameters were added. They included population density, gravity, drag, collision avoidance, flight range, camera position and tracking, and a few toggles such as tethering strings, floor plane, and bezier curves. Once the parameters were tweaked to the user’s liking, they need only to hit the spacebar and an image sequence of PNGs would start saving to the harddrive.

Once he had the exported image sequence, it was pretty easy to put it into a post processing application and work his magic. See one of the final spots below. Or you can view them on the Nervo.tv website. The birds where used in the top three spots.

The focus of the talk for UCLA was to show how small steps can combine to form something engaging. I start by showing a single particle, then two particles, then many particles, then add an emitter which can create particles but can also morph into an attractor which pulls particles. Then you add a magnetic repulsive force, then an orbital force, then a gravitational force. Add a few visual tweaks and you can end up with something like this.

I liked the look quite a bit so after I finished the UCLA presentation, I came home and began to make it an audio visualizer. This was a pretty easy step. I just have each particle listen to specific frequencies of incoming audio. The amplitude of the frequency effects the mass and charge of the particle so as the audio gets louder, the particle gets larger and more pushy.

Aubrey, the Barbarian’s ex-CTO, and I thought it would be funny to set a webcam up on the roof to try and make the East Coast offices jealous of our view. We put the camera on the roof and ran the firewire cable down in through my office window. The camera was weather-proofed with a 2-liter soda bottle and was weighed down by a cast-iron skillet.

We decided to use Processing to intercept the webcam feed so that we could add content to each image before letting it go live. I took a still image from the webcam and cut it into several layers in Photoshop in order to create depth masks. I could then apply those masks to the live feed and composite graphics between the layers to give a greater sense of realism and depth. We could have added anything at that point. UFOs, giant kittens, Cher. We ended up going with explosions, smoke plumes, and a Godzilla walking back and forth, breathing fire.

Sadly, there is no documentation of this because we got sidetracked. Then we moved to a larger office and lost the awesome view. However, I ended up recreating the project at home using a camera on the roof of my apartment, sans Godzilla.

An 8×8 grid of powermates was hooked up to a single Mac G5 tower through a daisy-chain of 21 four-port USB hubs and 6 power strips. I now had a low-res (8 pixels by 8 pixels), low-color (256 shades of blue) monitor whose individual pixels were an input device for itself. Now what.

The first project made with the new Powermate grid was a noisy mess. Each powermate can be turned left or right and I used this input information to not only control the brightness, but also to control the frequency of a sine wave tone associated with that particular powermate. What you are left with is 64 overlapping sine wave tones which, combined with a nice set of speakers with a good subwoofer, can create a fairly intense wall of noise. Can range from incredibly annoying to incredibly soothing. Later versions included a series of scales which would allow the sound to take on particular flavors, such as American Jazz or Middle-Eastern. And with some assistance, a beat generator was added to give the sine wave tones a bit of rhythm.

Later, I was sitting around, staring at these devices, and wondering what I should make next. I picked one up and was turning the knob absent-mindedly, and it brought back memories of playing pong with paddles as a child. I made a game of pong. Not a particularly good one. Because of the low resolution of the display, even if you leave your paddle stationary, you still have nearly a 50% chance of returning the shot.

Here is the original image.

I set up an overnight render for a song by Trentemoller. When I awoke the next morning and viewed it for the first time, I was stunned. I watched it a couple more times and sent the link to Andrew Bell. His response, “Yeah, that’ll do well”.

The thing that is most interesting to me about the video is that it was nearly entire created with code. I say nearly because I use a couple premade particle images but every other aspect (aside from the music, of course) was made possible with code. And randomness. If I were to recompile the code over and over, the resulting videos would all be unique.

Here is the first video I made for a song by Trentemøller’s Miss You. Its 92MB so be patient.

Due to the success of this project, The Barbarian Group decided to turn Magnetosphere into an iTunes visualizer. Andrew Bell, the most patient person I know, had the horrible task of porting my bandaided Processing code into proper man-style C++ while remaining faithful to the original look. He did a splendid job and the Magnetosphere iTunes visualizer did quite well. It eventually went on to greater things.

Below are three prints I created for the OFFF 2007 Conference in Barcelona where I gave a talk about my creative process.

Several months after the initial Magnetosphere video, I revisited the Trentemøller piece. It has become my favorite audio visualization to date. It just works so well! No small thanks to the beauty of the song itself. Miss You is a classic for me because it is both FFT friendly (Fast Fourier Transform, the method by which I am analyzing the audio) and extremely soothing to listen to. I will forever associate this song with discovery.

I found pretty quickly that the particles, even when being dynamically modified with audio, really just want to find their place and stay put. There isn’t much activity. I remedied this problem by adding a rotation velocity.

Audio by Ganga (Autumn).

Audio by Goldfrapp (Deer Stop).

And finally, Turbulence with flares. I saved this one for last because it was the most successful of the video renders. Audio by Thievery Corporation (Le Monde).

Over the next few months, I continued to refine this code. I created a couple dozen renders of varying success. The following videos are some more successful from the bunch.

Audio by Ganga (Autumn).

My apologies, I do not remember who created this audio. I will credit it properly as soon as it comes to me.

Audio by Royksopp (“In Space” off Melody A.M.)

I definitely got carried away with the additive blending. It took a while for me to understand that moderation is useful when applying new effects. But for now, make it glow! Like the stars in the heavens above! Everything should glow!

The other reason I relied so heavily on additive blending is that it works as an easy way to get around the depth-sorting issue which plagues semi-transparent image rendering. In OpenGL, if you want to draw a bunch of transparent PNGs, you need to draw them from back to front. When you have a dynamic camera system and a bunch of objects, it can be tricky or time consuming figuring out which objects are behind and which are in front. If you use additive blending, you don’t need to worry about depth sorting.

Audio by Helios (Sons of Light and Darkness). Note, this piece builds really slow so don’t be put off by the very very dark intro.

The thinking was that I could charge a bunch of particles and let them dance about in a scene. Some would be small and fast, others would be massive and slow. They could play off each other, dance with each other, and ultimately create an endless display of randomness.

The magic spark occurred when I decided that the small particles should have a variable charge. I tried random numbers, sine wave curves, even Perlin noise. None of those felt quite right. Then I tried audio analysis.

Each particle’s charge would be dependent on the amplitude of a frequency to which it was assigned. Spark!

Below are some images from the early days of Magnetosphere. At this point, my understanding of OpenGL was extremely limited. I knew how to draw points and lines and not much else. So these early iterations have a simple aesthetic.

Here are a couple fairly low-res videos of Magnetosphere being projected in my living room. I am filming the projection so you can better see how it performs in realtime. The beat detection effects are done programmatically so I apologize in advance for the hyperactivity.

Audio by Cornelius (Tone Twilight Zone).

Audio by Deee-Lite (Groove is in the Heart).

Visit The Barbarian Group NextFest portfolio page to learn more about the process behind this project.

Audio by Bit Shifter (“Flight Risk”). Mastered by Chris Burke at Bong & Dern.
Originally released on Various Artists: Sonic Acts XI • The Anthology Of Computer Art DVD, 2005.

I came up with the concept of the video when I started playing around with feedback loops in Processing. First I would construct a large box that the whole scene exists inside of. Inside that box, I would place some objects. Finally, I take an image of the entire scene and use it as a texture map for the sides of the big box. I would repeat this process every frame and depending on the rotation of the box and the size of the objects, you can get some really nice glimpses of infinity. Below are a couple sample images I created for a presentation I gave at OFFF in Barcelona.