Sprite multi-recoloring shaders and Game Maker: Studio implementation

Today we will get a little bit technical. Using a simple set of fragment shaders and color maps, we will entirely recolor a sprite on the fly.

Although the idea is implemented in Game Maker Studio, its principles should be applicable in any other shader-enabled application. Note that in Game Maker: Studio you may need to turn off "Interpolate colors between pixels" in the Global Game Settings for this to work!

This tutorial is aimed mainly at people with a basic or intermediate (like me) understanding of shaders. The method presented is not intended to be the fastest or more compact and efficient, and is broken down in steps that would be avoided by more experienced programmers, but I chose this structure in order to a) Make the method easier to understand and b) Highlight some specific things like texture coordinates, shader values and parameters[*]If there is one thing I hate in tutorials, it's this: "How to draw a kickass elf warrior: Step 1. Draw three circles. Step 2. Draw the rest of the kickass elf warrior.".

First of all we need a character sprite. It can contain many sub-images but it's not mandatory. Here is our guy:

Since the sprite will be recolored, we don't necessarily have to use good-looking colors: In fact, it helps if we use distinguishable hues to separate visually the various body parts and/or potential body accessories. For the same body part, we can use different brightness values to visualize that variance in the final sprite. But really, one can just paint a sprite with "normal" skin colors and everything: As long as all possible different colors are set up correctly, we are ok.

This is our "base sprite": It serves as the precursor to our "color mapped" sprite. But how exactly does this method work?

At first, we set up a "color mapping" shader and, using this shader, we draw all of our sprite's subimages on an empty surface. The only "additional" info for this first shader is a color map. The color map is simply a one-pixel-height sprite containing every color value in the base sprite that we created, in no particular order (but some hue grouping helps). If you want a value of the base sprite to be unchanged, like the sprite's outline, you can omit this color from the color map.

For our example, I am using a 32x1 sprite as the color map. Remember, if you are using Game Maker Studio, check the "use for 3d" box in the sprite properties of the color map! This will create a separate texture page for it and thus getting the texture coordinates of every pixel in the shader will be much easier[*]Other sprite dimensions will work, but for the "use for 3d" option to be available, they must be integer powers of two, like 1, 2, 4, 8, 16, 32, 64 etc.. Also, not all pixels of the color map have to be used: If you have a 20 different colors sprite, you can use a 32x1 color map where the first 20 pixels will be the various sprite colors and the rest can be a color not used anywhere in the base sprite.

The "color mapping" shader does the following: For each pixel sampled from the base sprite texture, check if there is a same-colored pixel in the color map texture provided. If there is a match, the output pixel will represent the coordinates of the same-colored pixel in the color map.

1:  varying vec2 v_vTexcoord;  
2:  varying vec4 v_vColour;  
3:  uniform sampler2D texColmap;  
4:  void main()  
5:  {  
6:    vec4 spr;  
7:    vec4 outp;  
8:    spr=texture2D(gm_BaseTexture,v_vTexcoord);  
9:    for (float i=0.0;i<=17.0;i++) { //we only have 18 colors (0-17)  
10:      if (vec3(spr)==vec3(texture2D(texColmap,vec2(i/32.0,0.0)))) outp=vec4(i/32.0,i/32.0,i/32.0,spr.a); //32: The color map width  
11:    }  
12:    gl_FragColor=outp;  
13:  }  

So let's say for example that the shader is currently sampling the purple pixel that is the character's left eye. In the color map, this is the 10th pixel. Remember that shaders read texture coordinates and red, green, blue and alpha values from 0 to 1! So the "shader-friendly" coordinate of the 10th pixel is 10/32 (32 is the color map width in our example)=0.3125. Our output pixel of the fragment shader can have its red, green, blue, or alpha value set to the number above. In this example I am setting the r, g and b component of gl_FragColor to this number. So effectively we render a black-and-white sprite (I have avoided using the term "grayscale" here because the output is not a grayscale version of the original. The brightness or darkness of the output pixel has nothing to do with the brightness or darkness of the original pixel):

The darker the shade of gray of the sprite, the more to the left is the corresponding pixel in the color map. Lighter shades correspond to the right-most pixels of the color map. Also, the shadow opacity has been altered due to how surface alpha works, but as long as we know the shadow color, we can restore the shadow alpha in our next shader. Oh, and since this is drawn to a surface, don't forget to create a sprite from that surface, with the same properties (image number, offsets etc) as the base sprite.

In Game Maker: Studio we use the texture_set_stage() function to provide our shader with another texture to work with (like the color map), besides gm_BaseTexture which is the thing we are currently drawing (sprite, surface, whatever). So the texColmap in line 3 of the code above can be provided in GM:S after "activating" our shader with shader_set() in this way:

 texture_set_stage(shader_get_sampler_index(shader_basecolor,"texColmap"),sprite_get_texture(spr_basemap,0));  

Where spr_basemap is the "base color map" sprite.

Now it's time to introduce another color map with the "proper" colors. The "proper color map" is a 32x1 surface where every pixel of the "base color map" has now the "correct" color value:

So the bluish skin pixels of the base color map are now portrayed as a proper skin tone, the orange-brown shoe pixels are now brown etc. Why a surface and not a sprite? Because it's easier to alter the colors of a surface in real-time. So if your character, who wears brown shoes, finds a pair of red shoes, you can set this surface as the draw target and replace the brown shoe pixels with some red ones!

The second shader (let's call it the "recolor shader")  samples the black-and-white pixels of the sprite we created, and based on their red (or green, or blue) value (remember, they range from 0 to 1), interprets it as a texture coordinate in the "proper color map" and outputs the final pixel with the color values of the corresponding pixel of the "proper color map".

1:  varying vec2 v_vTexcoord;  
2:  varying vec4 v_vColour;  
3:  uniform sampler2D texColmap;  
4:  void main()  
5:  {    
6:    float alpha;  
7:    vec4 spr;  
8:    spr=texture2D(gm_BaseTexture,v_vTexcoord);  
9:    vec4 col;    
10:    col=texture2D(texColmap,vec2(spr.r+0.01,0.0)); //or spr.g or spr.b since we set all 3 components to the same value. +0.01 is to make absolutely sure we are "inside" the correct pixel  
11:    if (spr.r==0.0 && spr.a!=0.0) { //hack for restoring shadow alpha - in our example the shadow color is the first in the color map  
12:      alpha=0.5;  
13:    } else {  
14:      alpha=spr.a;  
15:    }  
16:    gl_FragColor=vec4(col.rgb,alpha);    
17:  }  

The texColmap sampler in line 3 of the code above can be set up in the same way as for our first shader.

So, setting the "recolor shader" and drawing our black-and-white sprite gives us this result:

Not bad, eh? Now, we can keep separate color maps for the hair, shirt and pants and every other body group we want like this:

A different shirt color

A different shoe color

And, during runtime, in every game step or whenever a change in our colors happens, we draw those color maps to our "proper color map" surface:

You can download the related .gmx project from here.

I hope this tutorial helped you learn something new! Shaders can be tricky business, but the payoff from learning how to use them is pretty big. Happy coding!