April 2015
Boil-up is a realtime immersive simulation that puts you in the center of a feeding frenzy.
In March 2013, I was contacted by Olivier Jean in New Zealand who was working on an installation for the Auckland War Memorial Museum. This exhibition, called Moana - My Ocean, allows visitors to experience many of the oceanic wonders that surround the beautiful country of New Zealand from the dynamic of the surface waters all the way down into Kermadec Trench which plunges 10,000 meters deep.
One of the installations for this exhibition is all about the phenomenon of the "boil up". If you have watched any of the fantastic BBC, Discover Channel, or National Geographic documentaries about the ocean, you have probably seen video footage of this amazing and energetic display. Many different species of fish, both predator and prey, are drawn to plankton rich waters resulting in an impressive feeding frenzy.
National Geographic - Bluefin tuna eating bait ball
Earthtouch - Ultimate Sardine Run Experience
The bait balls that form in the Hauraki Gulf often contain many of the following species: pilchard, anchovies, mackerels, kahawai, kingfish, dolphins, bronze whaler sharks, diving gannets and the Bryde's whale. So when I was contacted and asked to make a real-time simulation of this extremely complicated display, I was hesitant.
I did a personal project a couple years ago where I was trying to simulate a bait ball using code. I ended up with something that was interesting to look at but I made little effort to try and make my 'fish objects' actually look like fish. I had no experience with using 3D models so I went with shapes that were easy to generate dynamically: tapered ribbons. As you can see below, the results were very un-fishlike. They look more like slugs or leeches.
During the initial negotiations, I explained to the client that I had no experience working with complex 3D models and if I were to take the job, we might want to focus on a data-driven aesthetic so we could avoid the many pitfalls associated with trying to make an experience look 'realistic'. We agreed, the end result would be simple shapes and the overall experience would be about the math and not about immersion. Instead of using a dolphin model, we would, for example, use a colored rectangular solid or some tapered low-polygon basic shape. This plan would save us all a lot of headaches. Or it would have had I stuck to the plan. More on this later.
Step 1: Bait-ball formation
I have been experimenting with simulating group dynamics for nearly as long as I have been coding. I became fasciated with the work of Craig Reynolds who was among the first to show that with very simple rulesets, you could do an accurate simulation of group behaviors that include flocking, predator and obstacle avoidance, and pursuit. Over a decade later, I watched a talk by Professor Iain Couzin who is the head of the Collective Animal Behavior department at Princeton (note: He has since moved on from Princeton and is working with the Max Planck Institute). He explained the rules of flocking behavior in a way that really clicked. That was the impetus to make my first generative bait ball. I was just putting Reynolds' and Couzin's ideas into motion.
Great Wired article about Couzin's research
Craig Reynold's open source steering framework
It was pretty easy to get these fish-objects to do something vaguely flock-like, but getting them to form into a torus proved to be a bit more of a challenge. If you just throw a bunch of flocking objects into a 3D environment, often what happens is they come together into a clumped formation and then they just head off as a group and disappear into the GL fog. I needed a way to keep them corralled but without feeling like I was imposing unnatural restrictions on their behavior. Every time I have to add a new controlling variable, I feel I am moving away from the purity of the behavior. But I didn't know what else to do so I introduced them to the concept of their own centroid. In addition to the flocking rules outlined by Reynolds, I added a general desire to not wander too far away from the average of all of their positions.
This tiny change was all it required to get them to form stable and energetic toroids.
Before: spherical formation but they all swim in the same direction.
After: more natural toroidal formation.
The flocking logic is very processor intensive. Each fish needs to compare it's position to every other fish. So things get pretty out of hand very quickly. I was finding with early tests on my Macbook Pro, I could handle 800 to 1000 fish and still hover around 60 fps. However, if you asked it to simulate 2000, things would drop to 15 fps or lower. Something needed to be done. This phenomenon in nature can occur with hundreds of thousands of fish. Just a couple thousand is going to look pretty paltry. I asked Andrew Bell for help and he suggested a modified binning solution which he coded for me. This improved the performance greatly. 2000 fish would swim around at 45 fps and 5000 at 8 fps. Still not amazing but much much better. And since the final installation version would be run off a very nicely built PC, I had high hopes we could reach 3000 on the PC at 60fps.
At Andrew's suggestion we then implemented multithreading. You throw 15 threads at the problem and suddenly things get much faster. On the PC with 15 threads working on the flocking logic, we were able to get 7000 objects flocking at 60fps and this was more than enough to make for a dense bait ball.
Technical note: I considered briefly moving all the flocking logic to the GPU which would probably have gotten us numbers in the tens of thousands, but I was concerned. I have done GPU-based flocking before but it is my understanding that once you move one aspect of this system to the GPU, you have to move them all. And though I had confidence I could easily make a GPU-based flocking system, the thought of making all the predators and whales and birds also GPU-based... I worried it was asking too much in the time we had. Not to mention, I find GPU-based coding much harder to debug. It was not a challenge I was willing to take on in the few weeks we had remaining. A battle for another day.
Step 2: The (smaller) Predators
The bait ball was coming along. However I still needed to introduce predators into the simulation and have them behave reasonably like their real-life counterparts. With the exception of the Gannets (the diving and swimming birds), the level of concern regarding the simulation of the other species was proportional to their size. Kahawai and Kingfish were pretty easy. They are small to medium sized predator fish that exhibit some loose schooling but wouldn't behave all that differently from the predators I used in my earlier bait ball and flocking projects.
The original placeholder visuals for the predator fish (Kahawai and Kingfish) were simple rectangular blocks. Eventually, I got tired of seeing these blocks so I went and grabbed a nice looking fish model I could use as a place holder.
This was a bad idea.
Pretty quickly, I got a good looking swimming tuna but then right down the rabbit hole I went.
Step 3: The Ocean
Doing a reasonable ocean environment was awkward, but I had the luxury of having a pretty solid plan B. By just using a gradient sky dome and a simple textured mesh for the surface, I could fake a passable ocean environment with little extra work. But you know how these things go: once I started, it became difficult to stop. I just wanted to make it more and more believable. As you can see from the picture with the tunas above, the original version of the ocean was very bright and had little falloff as you got further away from the camera. I wanted to try a pass at a darker cloudier environment.
I created this much more somber ocean look and though I thought it was a nice aesthetic, I eventually brought back the lighter background because we didn't want the fish to be bathed in so much shadow. I tried a couple different ocean surface meshes but ultimately dialed it back to make it less processor intensive. There was a version that used a tiled reaction diffusion effect (around the 00:20 mark) which had promise but ended up just being odd. It made the surface look more like a viscous liquid than ocean water. So I went back to just doing a simple mesh that undulated based on big animated sine waves. Eventually, I added an ocean surface normals texture so I could get some nice details without needing to complicate the mesh itself.
I could spend years trying to make a great real-time ocean simulation. I decided this was good enough and I should move on because those damned diving birds and huge whale were still waiting patiently for my attention.
Technical note: The light beams were done simply. I stuck a hundred tall skewed quads in the scene and had them additively render a soft-edged gradient texture. They rotated in place along their vertical axis so their width was constantly fluctuating. I would have preferred a more realistic approach using motion blur and scaling and actual refraction, but because of the logistics of the installation space, I already had to render the scene 3 times per frame (explanation below). To attempt too much post processing frag shader work at 5160x800 seemed unrealistic to me, though the video card might've been able to handle it like a pro. I'm just not sure.
Step 4: The Installation
A bit about the setup. As I mentioned earlier, this project is for the Auckland Museum conceived by Rawstorne Studio and PleasureKraft. They have designed and installed a 4-projector setup with a cylindrical housing. Here is a render of how the final installation will look.
Nice concept and design. But right away, I had my first concern. this is a 270° panorama. I can't just make a regular Cinder camera and set the field of view to 270°. That just wouldn't work. The fish on the sides would be incredibly distorted and I think maybe the max field of view is 180°. This was my first problem to tackle. Doesn't matter how awesome the fish look if I can properly create an image to project on this curved surface. So I asked around and did some research and found that what I wanted was a way to render an accurate cylindrical projection.
Turns out, the concept is pretty simple to grasp even if the shader logic is a bit confusing. You can read about it on this OpenGL forum post that explains the process.
- Create a camera with a 90° field of view and a 1:1 aspect ratio.
- Render the scene to a square FBO.
- Rotate the camera 90° to the right. (Repeat for as many degrees as you require. I need 270° so that is 3 renders, 90° rotation between renders.)
- Tile the 3 square FBOs into one wide FBO.
- Render it at the final resolution with a frag shader that deforms the wide FBO using tangent (shader math is in the forum post linked above).
In the following example images, the scene being rendered is a long square tube that is textured with a checkerboard pattern. There are yellow flocking objects swimming about inside this tube. Also in the tube are 4 long red poles that extend from one opening to the other.
Before and After images of the GLSL shader.
Projection from outside and inside. Note: when you are inside the cylindrical projection, the curved red lines seem to become perfectly straight again.
The good news is the problem was easily solved with help from the OpenGL forum and some Googling. The bad news, seems I have to render the scene 3 times and then do a huge frag shader over the whole thing. I didn't know how much of a problem the Nvidia would have with that many frags being processed per frame but it handled it well. The bottleneck remains with the flocking logic.
Step 5: The (larger) Predators
The sharks and in particular, the dolphins, were reasonably easy to make look realistic, but making their behavior realistic was a challenge I definitely feel I could have used a couple more months on. For the visuals, I grabbed a couple models off Turbosquid.com (note, a 3D artist was eventually hired and they were responsible for making all the species present in the final installation) and started making the dolphins and sharks swim around. I had them employ the same general flocking rules the bait fish abided by. This way the dolphins could hunt with other dolphins and the sharks could hunt with other sharks. Turns out, the sharks are a bit more solitary so I just turned down the intensity of their reaction to some of the flocking forces.
To make the dolphins and sharks swim, I did the same thing I did with the fish. I made the adjustments in the vertex shader and that ended up being enough... mostly. I was indeed able to make the larger predators look like they were using their bodies to propel themselves forwards, but I wasn't able to get their bodies to bend believably when they had to do a sudden change of direction. I think this was my biggest failing with this project. When a dolphin swims at a shark, right before they would collide, they avoid each other which makes the sudden change of movement look extremely rigid because in effect they are simply pivoting around a point below their eyes. As I continue to work on this code, I am going to tackle the bending bodies problem. I have an idea of how to do it but it would likely involve a larger overhaul of their fundamental behavior.
The dolphins exhibit some interesting behavior when attacking these bait balls. They occasionally blow bubbles to try and stress the bait fish. They also tend to attack from below in attempt to push the bait ball to the ocean surface. If the bait ball is pinned to the surface, that is one less direction they can escape. However, once the bait ball starts to reach the surface, the gannets begin attacking from above. This pushes the bait ball further down until the dolphins scare it back to the surface again.
Happily I was able to recreate these particular behaviors within the simulation. The dolphins sometimes blow bubbles which do bother the bait fish. I also told the dolphins to start their attack by aiming for the bottom of the mass of fish but as they get closer, they start to aim for the center of the mass. This caused their attacks to push the bait fish from below, exposing them to the gannets above.
Technical note: I attempted a creature cam. This was a camera that followed a specific creature in an over-the-shoulder style 3rd person point of view. It was compelling, but it was also a bit disorienting and nauseating. It is one thing to view something in front of you, but this installation would put the experience around you. Swaying movements were a big no no.
Step 6: The Birds and the Whale
I will start with the Whale. I was originally more concerned about how to deal with the whale but he ended up being a reasonable challenge. There is still much to do to make it's movements more true to life but he is doing just fine. Big thanks to the person who made the 3D models whose name i actually don't know. Once I find out, I will credit them appropriately. Whoever you are, you did great work.
The whale model has a mouth with a rudimentary skeleton that I could manipulate programmatically. When the whale was ready to swim through the bait ball center, I could tell his mouth to open and then eventually shut.
I should add that I owe Éric Renaud-Houde a great deal of gratitude for writing a Cinderblock that allowed me to use Cinder and the Assimp library to control the 3D animations. He even went out of his way to help me get started with the programmatic bird animation. Thank you Éric. You helped me more than you know.
But lets talk about the bird. The diving Cape Gannet. What an asshole. It isn't enough to be a bird that can fly and swim on the water surface, but you also feel the need to be able to dive many meters below the surface of the ocean and swim, if you can even call that spastic jerking 'swimming', to depths of 22 meters. You made my life miserable. There is no grace to your movement. Every limb just flails around. And you leave a thick trail of tiny bubbles as you burst into the ocean, your wings folded back, your form like a missile. Asshole.
I didn't know what to do about him which is why I saved him for last. I am not a creature animator. I had never done any work with animated models and I have a sneaky feeling this stupid diving bird is at the top of the difficulty list for creature animators. I bet there is some awkward gannet trophy somewhere waiting to be claimed by the first person to do a good animation of a swimming gannet but nobody would dare attempt such an impossible task.
I did what I could. And lacking a 3D bird model, I started with a model of the Airbus 350. This first pass at the behavior worked out pretty well, though there was certainly room for improvement in how the Airbusses oriented themselves on the ocean surface. Very twitchy.
I decided to make these birds individual state machines. They would have 3 main states. First is hunting. When in hunt mode, their logic is to ascend to 200 units and fly around in a loose flocking pattern until they are above the bait ball. Then they enter diving mode. When diving, they succumb to gravity and buoyancy. Once they hit the water, the water tries to push them back to the surface but their swimming allows them to dive deeper. When they get to near the center of the bait ball, or if they swim too deep, the resting state is triggered. While resting, they swim back to the surface and stay there until the are hungry again. Repeat.
After some finagling with wing bend angles and sine-based swim animations, I ended with something passable. And awkward. I am happy with the behavior of the gannets but the look and animation could use the gentle caress of a proper 3D animator. I have a new respect for anyone that does character animation by hand.
Final Installation
Sadly, I didn't have the opportunity to make it to New Zealand for the launch so I was not able to get any documentation of the final installation. Here is a render from the final version of the application.