August 28, 2011

Modelling in Maya 101 – Part 2

by Mathias Tangen Leganger

Welcome back! We created a character in part 1, using various 3D-modelling techniques inside of Maya (or your preferred modelling package). We now have our 3D-mesh and UV-layout ready to go, though our character looks kind of dull just with that default-gray Lambert-material in Maya.

Colors anyone?

In order to unleash your creativity and bring your character to life, you need one (or several) texture maps. We will start out with the diffuse map. In a nutshell, the diffuse map is a 2D picture that gets projected onto your 3D mesh, telling your application (or game) what color should be assigned to which polygon. Similarly, you have tons of other possible maps for things such as specularity, transparency, bumps and what not. The picture above depicts the notorious Sir Nigel Rochester (known from “Evil Junior” of course) as a 3D mesh with a diffuse-map containing all of his color information.

Let’s see how you go about making one of those diffuse maps!

Enter Photoshop

Photoshop is perhaps the application for painting 2D texture-maps (but feel free to give me a shelling in the comments field if you disagree). You may absolutely use other packages (or even hand paint your textures and scan them) but in this tutorial series I’m going to stick to Photoshop. Many 3D packages also provide more or less useful texture-painting capabilities, most notably ZBrush and BodyPaint. I will actually showcase some techniques in BodyPaint later on, so stay tuned for that.

Back to the action – fire up Photoshop and create a new file. I recommend the following settings for your texture maps:

  • Width x height: 2048 x 2048 pixels
  • Resolution: 300 pixels/inch
  • RGB Color with 8 bits

This should result in a PSD file of ~12 MB. 2048 x 2048 pixels = “2K” in pixel-pusher lingo. 1024 = 1K. You get the idea. Now one piece of advice (which I learned for myself the hard way): don’t be cheap on your initial texture resolution — ever! By heeding this advice you will hopefully avoid the painful mistake of starting out with a texture resolution that is too low, only to find yourself starting from scratch because the texture you spent several hours working on looks blurry on your mesh. If your initial texture-file is 2K, you can always export your texture as 1K if need be. But a 1K master-file upscaled to a 2K texture map is surely an abomination.

Ok, so 2K should do for this project. Save the newly-created file as a PSD in the sourceimages directory of your Maya project. Name it something like: yourcharacter_diffuse.PSD.

Texturing preparations

Now you should have a clean 2K-canvas facing you in Photoshop. Let’s go through a few hints before you continue on your texturing business.

First, make sure you have exported a “UV Snapshot” from the UV Texture Editor inside Maya, as depicted in the picture below. I will use this snapshot as reference in the painting process.


Next, you can either save your texture map during the process as an image file and insert it into your map in Maya (as we discussed under “Assigning Maps” in part 1) — or — you could utilize this nifty trick: create a new material in your Hypershade Editor in Maya by selecting “PSD File”. Browse to your PSD file in “Image Name”. Select your mesh in Maya, then RIGHT-CLICK the “place2dTexture” node in Hypershade and click “Assign texture’s material to selection” (see image below).

You should be able to save some time by hitting “Save” in Photoshop and see the PSD material instantly update inside Maya. With that out of the way, let’s set up some layers inside of Photoshop.

Layers are among the texture artist’s best friends: they help creative (and often disorganized) people be effective (and in the process: more organized). They eat RAM, so don’t go crazy on your layers. But a few will help you experiment and save you a lot of time. Remember that you may also be part of a team and that other artists may need to work on your files. How to win friends in the art department: keep your files clean, properly named and organized.

The picture above shows my layers for Nigel’s PSD file.

  • Put your UV snapshot at the top, with “normal” or “multiply” as the layer blending-mode.
  • Create a folder for ambient-occlusion effects. This is a very nice trick I will dive into next.
  • One folder for your “diffuse”, i.e. where you put all your paint layers.

Alright, now let’s bake that ambient-occlusion map…

Ambient occlusion

This step is optional, but with the process being a snap and the results being good — why not give it a try?

Ambient occlusion (AO) is an effect in which you let the renderer determine shadows for you. A white material is applied to everything in the scene, and then the scene is lit. Lighting and shadow information is calculated and the renderer “bakes” an AO-map containing only shades of black and white. By doing this, you basically let the renderer figure out shadows on your mesh. It can really add that little extra and also save you a lot if time in the texturing process. Lazy artists are good artists. I will briefly walk you through the process of baking AO-maps in Maya right now..

Ok, so this is what we will do: open up your 3D mesh in Maya and apply the ambient-occlusion material in the Hypershade Editor:

  • Create the surface shader: Maya -> Surface -> Surface Shader.
  • Create the AO shader: mental ray -> Textures -> mib_amb_occlusion.
  • Connect the AO to the surface shader in the Work Area by MMB-dragging AO output to surface input (see picture). Choose default (outValue -> outColor).
  • Select the AO shader and open the Attribute Editor (CTRL+A).
  • Play with different settings under “Samples” (quality of output) and “Distance” (effect). Try setting “Samples” to 128 or 256.
  • Apply this material to your mesh.

Now we need to “bake” (or in other words: render) this texture:

  • Select your perspective camera in the Outliner. In the Attribute Editor set Environment -> Background Color to white.
  • Select Rendering in the menu selection.
  • Select Render -> Render using -> mental ray (you may need to enable mental ray in the plug-in manager) in the main menu.
  • Select your mesh.
  • Hit Lighting/Shading -> Batch Bake (mental ray) Settings. Set the resolution to your texture map resolution, e.g. 2K.
  • Hit Convert when done.

The resulting map will be baked to renderData -> mental ray -> lightMap in your project directory. The result should look something like this:

Awesome. Now paste your freshly-baked AO-map onto its own layer in Photoshop, set the layer to “Multiply” and make sure it’s placed on top of any and all diffuse-layers you might already have. Adjust the blend-percentage to taste. Voilá!

3D texture-painting in BodyPaint

This step is optional but highly recommended if you have access to an application supporting 3D-painting. Painting a texture on a 2D canvas can be both limiting and frustrating because of distorted and slow feedback. 3D texture-painting enables a much more intuitive and responsive process because you paint directly onto the surface of the mesh. There are two methods for 3D-painting used by different solutions. ZBrush offers polygon-painting where you paint individual polygons directly. BodyPaint stores color information as an intermediary 2D-image. Maya also offers 3D-painting capabilities. I will use BodyPaint in this example. The principles are the same, regardless of application used. Here is what you need to do:

  • Export the mesh you want to 3D-paint to an .OBJ file in Maya.
  • Open BodyPaint. Create a new project. Import your .OBJ file.
  • Select all mesh objects in the “Objects” tab.
  • Create a new material in the “Materials” tab. RIGHT-CLICK the new material and hit “Apply”.
  • Click on the material icon. In the “Attributes” tab click Color -> Texture -> Create New Texture. Alternatively, load your existing texture.
  • RIGHT-CLICK the material icon and hit “Enable 3D Paint”.
  • Create a new layer under the “Layers” tab.
  • Start painting!

Now you can either make a texture from scratch or enhance your existing textures. 3D-painting is also great for painting in reference for 2D-painting. This is what I have done in the picture below. This way I can easily figure out where I want to paint Nigel’s shadows and skin in Photoshop.

When you are finished simply RIGHT-CLICK the texture icon next to the material under the “Material” tab and press Texture -> Save Texture as. Save as .PSD if you want to keep your layers.

Painting in Photoshop

Contrary to popular belief, you don’t need a PhD in Photoshop in order to paint awesome textures. All you really need is the paint brush. Here are a few additional tricks you may find useful:

Layer masks: Click in the background (outside the grid markings) in your UV-snapshot layer with the Magic Wand Tool (W). Click CTRL+SHIFT+I to invert the selection. Create a new layer. Then hit “Add layer mask” in the layers panel. You can see the result in the picture below. Next to the mini-version of your paint layer, you have a black-and-white representation of your layer mask. 100% black in the mask = 100% invisible, 100% white = 100% visible. In other words, layer masks are an extremely flexible and powerful tool in the painting process because you can “paint” in or out what should be visible in this particular layer. This is very nice when you want to blend your layers manually.

External textures: You can achieve insanely nice and lifelike effects by adding external textures and images onto a separate layer and experiment with the blending mode of the layer, e.g. “Multiply” or “Overlay”. Does your character wear armor? Try adding a metal texture with scratches and wear-and-tear.

Warping and liquify: Sometimes you need to push things around, like if you import a 2D texture that needs to be warped according to the grid lines of your UV snapshot. You can use Edit – > Transform -> Warp or Filter -> Liquify.

Pressure-sensitive tablet: This is a no-brainer. Texture-painting solely with a mouse is possible but extremely limiting. A pressure-sensitive tablet is a good investment (if you do not already have one) and will definitely make your day easier as an artist.

Pressure-sensitive brushes:  If you have a tablet, try experimenting with the brush settings under “Brushes”. Especially “Shape dynamics” (size) and “Other dynamics” (opacity).

Basic brushes: I tend to use a basic hard round brush, a soft one, and combinations of these with sensitivity for either size or opacity. Try making other brushes by painting a black-and-white pattern on a square canvas and hit Edit -> Define Brush Preset (or simply download free brushes on the net).

With those few tricks freshly memorized, it’s time to paint!

  • Start with a color filling the entire background (left image).
  • Import reference from 3D-painting if available.
  • Using your UV snapshot, and popping in and out of Maya to check the effects of your actions, start painting in the basic colors with a hard round brush for each mesh piece or body part (middle image). You can also use your AO layer for reference.
  • Once you have painted in all your desired colors, try enabling your AO layer to see how it looks (right image).
  • Want more detail? Experiment with more advanced brush settings or add external textures with blending.

When satisfied, save out your work. I like saving as .PNG (which also stores transparency), though you may find a better choice depending on the requirements of your project (consult your art director or lead programmer if you have one). If you have an established pipeline, it will most likely do batch conversions of textures and assets at some point, so keep your original textures in high quality is always a good thing. Save the file in the same resolution as your PSD. Name it something like yourcharacter_diffuse.

Specularity and other maps

If you want additional control over other properties than just color, you can create other maps as well. For example, you may want Sir Nigel’s top hat to be extra shiny. Shininess, or rather specularity, is determined by a specular map. Create a new layer in your Photoshop texture-file and paint with black-and-white, black representing matte, gray semi-glossy and white very glossy. Paint as in the picture below:

You can use the same technique for controlling other surface-properties such as bumps (e.g. for hairs and fur) and transparency (the glass of Nigel’s monocle). A much-used map in video games is the normal map. Normal maps are color images providing rendering information for how light should be reflected on the surface of the mesh. They are often used for giving the illusion of indentations and protrusions on flat surfaces, such as muscles, bones and wrinkled clothing and so on. Normal maps can be created from high-polygon versions of your mesh.

Wrapping it up

OK, let’s conclude this part of the tutorial by adding our diffuse and specular maps in Maya:

  • Create a Blinn material and assign it to your mesh.
  • Open the Attribute Editor (CTRL+A) to edit the material properties.
  • Enter your diffuse map in Common Material Properties -> Color.
  • Enter your specular map in Specular Shading -> Specular Color.

If you want to render an image of the result, do the following:

  • Add a primitive plane to your scene. Stretch it out so that it acts as the ground. Select two sides of the plane (the ones visible in your perspective camera), extrude those edges and “lift” them to create walls. When done, hit “2” on your keyboard to smooth it. Apply a white Lambert-material to it.
  • Then add an ambient light to your scene and position it.
  • Set your renderer to mental ray.
  • Position your perspective camera, open the IPR renderer and select a small portion of the scene for live preview-rendering.
  • Now, play with the settings of your mesh’s material and the lights/background until you are satisfied with the result in the IPR renderer. Also try experimenting with the render settings such as quality.
  • When happy, render the current frame with mental ray.

That concludes this tutorial part. We now have a 3D mesh with maps for diffuse and specularity. I hope you have enjoyed this tutorial, and please feel free to ask questions or add your own work in the forum!

About the author, Mathias Tangen Leganger

Mathias is a student of technology and business in the day, and game artist in the night. He has been a mentor in 3D modelling at The Gathering, 3D designer at the Norwegian Defence Research Establishment, and modeller and texturer on the game Evil Junior for Illuminatics Gameware. Currently stuck in the godforsaken city of Trondheim, he plays and coaches tennis when he is not pushing vertices or crunching literature.

Leave a Comment