Format journey

A few months ago, I had the pleasure to work with Luciano Foglia and Anrick Bregman on a visual experiment called Format Journey. The installation uses data – a series of images and a sound file – to modify the texture and the shape of a 3D object. Combined with a slight randomization it gives very nice and unpredictable results.

When we first talked about the project, the idea was to use either Flash or Processing, but I though that it's a good occasion to try Unity3D (of course… :) To achieve the desired visual effect I needed a shader system that's fast, flexible that and works with 3D. Unity's ShaderLab seemed like a good option.

The material on the shape is transparent and is composed of a blurred version of an image applied as regular texture and the image itself applied as reflection. The images change over time, and are animated by scrolling UVs.

All the data for the installation is loaded dynamically, the application itself is just composed of some scripts that make it run, so the file is only 55KB! It's controlled by an external configuration file that allows to define the source images and sounds. There are also quite a few other settings to tweak they way it works, ex. how transparent or how reflective the material should be.

One particular challenge was to make the shape react to the sound. To make this happen I needed to read the spectrum and Unity3D doesn't offer access to sound data on the code level. One possible solution was to write a plugin that does that, the downside however is that it wouldn't run in the web player.

I solved the problem by pre-processing the sound in a small Flash app. It dumps the spectrum data to a text file which is then loaded to Unity3D. When the application plays the sound file, it uses this data to check the values of the spectrum at any position in time. Simple, but does the job.

The installation is part of a project called Tango and Hawaii created by Anrick and Luciano. Be sure to check the live demo. It was originally presented at the Magdalena Festival in Maribor, Slovenia.

It was a new experience for me to work on an art piece was. It was very inspiring to see what Anrick and Luciano did with a tool I helped them to create. Thank you guys!