Loading and manipulating images in Unity3D
As I promised here's the first article exploring Unity as a 3D web application tool.
Every web application nowadays is data driven, therefore the ability to dynamically load assets is crucial. Unity3D can load plain text, images, video and sound. Since loading plain text is not so interesting, I thought I'd see how it works with an image.
Here's a little experiment I called "Image Decomposer". It loads an image passed as parameter in the URL and creates a wall of little cubes which colors are based on the pixels of the image. When you rollover the cubes they fall down slowly decomposing it, hence the name.
The code is pretty simple, you can get the source here. Below are a few things I found out while making it.
URL parameters
A very useful way to pass parameters to your swf files is to attach them as a query string in the embedding HTML, like this: flash.swf?paramA=valueA¶mB=valueB. It's a Flash classic!
The good news is that is also works in Unity. Take a look a the HTML source of the demo to see how the image URL is passed to the unity3d file. Inside Unity3D it's very easy to get access to it through a static property called Application.srcValue. It returns the whole path, including the file name and you need to parse the query string yourself. Here's a sample code in C#:
string su = Application.srcValue;
string qs = su.Substring(su.IndexOf("?") + 1);
char[] deli = "=".ToCharArray();
string[] ps = qs.Split(deli);
The 'ps' array now contains pairs of key/values from the query string.
Loading freedom
Unity3D supports JPG and PNG format. Sadly GIFs are not supported, even animated… ;) The really nice thing is that there aren't any security restrictions when loading files from other domains. This is something that drove me crazy so many times with Flash!
In Unity3D pass a valid URL and it will load the asset, it's so simple. If you don't believe me – just paste any link pointing to an image in the demo ("image" parameter in the URL) and see it for yourself.
I wouldn't expect things to be like that forever though, so use it while you can people!
Coroutines and C#
To deal with asynchronous events, like loading an external asset, Unity3D uses coroutines. They are a bit different from events in AS3, but not very much. If you are a fan of C# and want to use coroutines you may find the official documentation a bit unclear. So, first, read the official documentation here and here, and continue reading this post afterward.
What caught my attention is that to use the yield statement, the enclosing method must return IEnumerator. But all of the inherited methods in MonoBehavior, like Start(), Update() and the event handling ones return void.
Surprisingly, it turns out that you can just change the signature of any of those methods so that they return IEnumerator! If this isn't breaking OOP rules, I don't know what is. But, since it works, it's ok I guess. It's certainly not obvious. Otherwise it is possible to create a custom function that returns IEnumerator and use the StartCoroutine() method of MonoBehavior to run it.
Another thing is that with C# you need to write yield return... not just yield... The official docs being written in Javascript they tend to omit the new operator as well, and C# does not like that. So, if the docs say:
yield WaitForSeconds(2);
and you copy/paste this to a C# script, you need to edit this line and make it look like this:
yield return new WaitForSeconds(2);
If you keep this in mind you will spare yourself some annoying compile errors.
Scaling images (or why you don't need to do this)
Once the image is loaded, it can be assigned to a Texture2D object, which is the type to store bitmaps in Unity3D. From now on it can be used as a texture for any material. For example, you can create a plane and assign it as the main texture of the plane's material or use GUI.Label to display the picture in 2D. But let's do something more interesting.
Texture2D has a few methods to extract information from the image, the most basic being GetPixel() and SetPixel(). They work exactly the same way as their equivalents in AS3 BitmapData class. The first allows to get the color of a pixel at a given coordinate, the other does the opposite.
When you start to play with these and you use images embedded in the project, make sure to select the "Is Readable" checkbox in the inspector. Images loaded dynamically are readable by default.
In the demo, no matter how big the loaded image is, the amount of cubes generated remains the same (it will adapt for different aspect ratios though). My first idea was to scale the image to something like 50×50 pixels, and then using getPixel() to fetch the color for each cube. But I found out that there isn't any function to resize an image in the API. Sure, I could write my own function, but this can get quite complex as you can see here. Fortunately there is a better way.
Texture2D has a method called GetPixelBilinear(). It uses bilinear filtering to evaluate the color of the image at a specific point defined by normalized coordinates (i.e. in 0-1 range). If the word "bilinear" sounds familiar to you it's because Photoshop also uses this algorithm to scale images. With this method the same code can be used to get pixel colors from images regardless of their size and aspect ratio:
Color c1 = image.GetPixelBilinear(0,0);
// always top-left color
Color c2 = image.GetPixelBilinear(1,1);
// always bottom-right
It's much more elegant (and faster) than scaling images!
As a side note: it would be also very easy to implement an actual resizing function using this method, in case you really need that. I'm sure that by now you can figure out how to do this :)
What's next?
The above example is simple but I hope it shows you how to get started with dynamic content. As far as images are concerned, much more is possible. By using custom shaders and render textures you can apply filter effects to images (and video). However, these features are available only in Unity3D Pro and besides, explaining it would make this post way too long. I'll do that in a separate article instead.