Troubleshooting: Creating a Toon Shader

Troubleshooting this Tutorial Sometimes, even though you try hard to understand the information in a tutorial, things don't work out quite like you want it to. This page is here to help you resolve any problems you might be having with the tutorial on creating a toon shader. The Common Mistakes section describes common problems that people have when doing the things in this tutorial, and how to resolve them. The Frequently Asked Questions section describes questions that people have that aren't related to mistakes, but rather, trying to understand the stuff better or exploring how it can be used. If your problem or concern isn't addressed here, feel free to add a comment below, so that I know where you're having trouble. I like to keep these pages fairly clean, so I may remove comments that I felt like have been addressed. If I remove your comment and you don't feel like the problem has been fixed, repost the question and we'll take another look at it. If a tutorial has a mistake in it, I will fix the mistake and reply to the comment with a brief explanation. However, after a couple of weeks I'll likely go back and remove the original comment as well as my reply, because, hopefully, the problem will have been fixed, and it won't be a concern any more.

---

Common Mistakes None listed yet…

---

Frequently Asked Questions None listed yet…

---

Chris (guest) 06 Feb 2011 16:11 Hi, first of all I would like to thank you for making these tutorials. They are really nice for a beginner like me. I would like to use your toon shader to render a test model of mine. But I can't figure out how to use it in my XNA code… In my draw-method I've set the effect parameters "World", "View" and "Projection" as shown in an earlier tutorial. When it comes to mesh.Draw() I get an InvalidOperationException saying that "The current vertext declaration does not include alle elements required b the current vertext shader. TextureCoordinate0 is missing". Is it because I did not set the effect parameter "Texture"? Please help me out, I've no clue what to do. :-/ (I'm using XNA 4.0) Thanks! Zac (guest) 03 Apr 2011 21:10 Did you ever solve this? I'm having the same problem :( Is your model using a texture, or just colors? rbwhitaker 04 Sep 2011 16:18 Is your model using a texture, or just colors? If your 3D model does not have any textures associated with it, then when XNA loads the model, it will use colors instead of texture coordinates. (It is probably using something like VertexPositionColor, instead of VertexPositionTexture, if you know what I mean by that. If not, I discuss it a bit in the Primitives tutorials.) Since our shader uses texture coordinates, I can easily imagine getting an error when you try to supply geometry to draw that does not have texture coordinates (but rather, vertex colors instead). If this is the problem, there are a couple of ways that you might be able to go about fixing it. For one, just use a texture instead. The texture would probably be rediculously simple—something along the line of several colored regions, and you just map the right parts to the right colors in your 3D modeling/texturing tool. That's what I've done with my Helicopter model, and the frog model in the picture for the tutorial, and I'm about to upload a beach ball model for people to use as well. If that's not an option, then you'd need to modify the shader to work with vertex colors instead. My version of that is below, though I'll admit I haven't tested it. (If you get it to work, please post that on here. It compiles, but I ran it with my original helicopter model, and it crashes in a very similar way to what you described, because now it is looking for colors, while my model supplies texture coordinates—so the opposite of the problem you're having, meaning that this one will probably work for you.) //--------------------------- BASIC PROPERTIES ------------------------------ // The world transformation float4x4 World; // The view transformation float4x4 View; // The projection transformation float4x4 Projection; // The transpose of the inverse of the world transformation, // used for transforming the vertex's normal float4x4 WorldInverseTranspose; //--------------------------- DIFFUSE LIGHT PROPERTIES ------------------------------ // The direction of the diffuse light float3 DiffuseLightDirection = float3(1, 0, 0); // The color of the diffuse light float4 DiffuseColor = float4(1, 1, 1, 1); // The intensity of the diffuse light float DiffuseIntensity = 1.0; //--------------------------- TOON SHADER PROPERTIES ------------------------------ // The color to draw the lines in. Black is a good default. float4 LineColor = float4(0, 0, 0, 1); // The thickness of the lines. This may need to change, depending on the scale of // the objects you are drawing. float4 LineThickness = .03; //--------------------------- DATA STRUCTURES ------------------------------ // The structure used to store information between the application and the // vertex shader struct AppToVertex { float4 Position : POSITION0; // The position of the vertex float3 Normal : NORMAL0; // The vertex's normal float4 Color : COLOR0; // The vertex's color }; // The structure used to store information between the vertex shader and the // pixel shader struct VertexToPixel { float4 Position : POSITION0; float4 Color : COLOR0; float3 Normal : TEXCOORD1; }; //--------------------------- SHADERS ------------------------------ // The vertex shader that does cel shading. // It really only does the basic transformation of the vertex location, // and normal, and copies the texture coordinate over. VertexToPixel CelVertexShader(AppToVertex input) { VertexToPixel output; // Transform the position float4 worldPosition = mul(input.Position, World); float4 viewPosition = mul(worldPosition, View); output.Position = mul(viewPosition, Projection); // Transform the normal output.Normal = normalize(mul(input.Normal, WorldInverseTranspose)); // Copy over the texture coordinate output.Color = input.Color; return output; } // The pixel shader that does cel shading. Basically, it calculates // the color like is should, and then it discretizes the color into // one of four colors. float4 CelPixelShader(VertexToPixel input) : COLOR0 { // Calculate diffuse light amount float intensity = dot(normalize(DiffuseLightDirection), input.Normal); if(intensity < 0) intensity = 0; // Calculate what would normally be the final color, including texturing and diffuse lighting float4 color = input.Color * DiffuseColor * DiffuseIntensity; color.a = 1; // Discretize the intensity, based on a few cutoff points if (intensity > 0.95) color = float4(1.0,1,1,1.0) * color; else if (intensity > 0.5) color = float4(0.7,0.7,0.7,1.0) * color; else if (intensity > 0.05) color = float4(0.35,0.35,0.35,1.0) * color; else color = float4(0.1,0.1,0.1,1.0) * color; return color; } // The vertex shader that does the outlines VertexToPixel OutlineVertexShader(AppToVertex input) { VertexToPixel output = (VertexToPixel)0; // Calculate where the vertex ought to be. This line is equivalent // to the transformations in the CelVertexShader. float4 original = mul(mul(mul(input.Position, World), View), Projection); // Calculates the normal of the vertex like it ought to be. float4 normal = mul(mul(mul(input.Normal, World), View), Projection); // Take the correct "original" location and translate the vertex a little // bit in the direction of the normal to draw a slightly expanded object. // Later, we will draw over most of this with the right color, except the expanded // part, which will leave the outline that we want. output.Position = original + (mul(LineThickness, normal)); return output; } // The pixel shader for the outline. It is pretty simple: draw everything with the // correct line color. float4 OutlinePixelShader(VertexToPixel input) : COLOR0 { return LineColor; } // The entire technique for doing toon shading technique Toon { // The first pass will go through and draw the back-facing triangles with the outline shader, // which will draw a slightly larger version of the model with the outline color. Later, the // model will get drawn normally, and draw over the top most of this, leaving only an outline. pass Pass1 { VertexShader = compile vs_2_0 OutlineVertexShader(); PixelShader = compile ps_2_0 OutlinePixelShader(); CullMode = CW; } // The second pass will draw the model like normal, but with the cel pixel shader, which will // color the model with certain colors, giving us the cel/toon effect that we are looking for. pass Pass2 { VertexShader = compile vs_2_0 CelVertexShader(); PixelShader = compile ps_2_0 CelPixelShader(); CullMode = CCW; } } nic (guest) 02 May 2011 02:47 Hi, could you share the model as well as the texture used in this tutorial please? Thanks rbwhitaker 04 Sep 2011 16:28 I'd love to, but I'm not sure I can. I don't know where I stuck it. I just looked around for a while on my computer and I can't seem to find it. The Helicopter model, which is already in the model library should have a very similar effect, as well as the beach ball model that I'm about to put in there too. If I find the frog model, I'll upload that as well. Incorrect Normals? EmpiresBane 23 Sep 2011 23:25 After playing around with this shader a bit, I realized something looked wrong. I increased the LineThickness and set the render to wireframe and the vertices were transformed in odd directions. Shouldn't a transformation along the normal result in a scaling? I have a picture of it here: i<dot>imgur<dot>com/Yrj6V<dot>png (in wouldn't let me post a link, so you'll have to copy it) To clarify, I have FillMode set to WireFrame, and CullMode set to None in order to better show what I'm talking about. Re: Incorrect Normals? rbwhitaker 27 Sep 2011 02:26 Hmm… after playing around with this for a little bit, I think you're right. I'm going to see if I can fix this sometime over the next little bit. I've put it on my To Do list. If you haven't heard anything back from me by, let's say, a week from now (Monday, October 3rd 2011) then send me and email to remind me I said I'd look into this. Re: Incorrect Normals? EmpiresBane 28 Sep 2011 19:21 Ok. Thanks. Now that I know that it's not just me I can rule out a ton of possibilities. That should make finding the solution a lot easier. Re: Incorrect Normals? EmpiresBane 29 Sep 2011 21:30 I just solved it. You need to change LineThickness to a float instead of a float4. I'm double checking all my code to make sure that's the only change I made, but that seems to be the issue. Re: Incorrect Normals? rbwhitaker 05 Oct 2011 03:53 You're right. It's that simple. I don't know why I was putting float4 in there instead. I guess just out of habit. Too many vectors to pass over. At any rate, EmpiresBane is correct, it should be float, not float4. I've updated the tutorial to reflect this. BogacG (guest) 24 Jun 2012 12:41 Hi; I've tried to mix this tutorial with Skybox. However after first draw, graphics.GraphicsDevice.RasterizerState becomes null hence I get an error. (First draw gives an expected render with model in toon shader and skybox in background) I expect the reason is CullMode is set in Draw method code for skybox, then is set in HLSL technique for Toon Shader. Which nulls graphics.GraphicsDevice.RasterizerState. So I took originalRasterizerState as a property, not a local instance in Draw method, and then added a bool, firstRun to check if it's the first Draw or not. if (firstRun == true) { originalRasterizerState = graphics.GraphicsDevice.RasterizerState; firstRun = false; } else It works. However I'd like to know if there would be another way to interoperate CullMode in HLSL and .NET code without doing this trick? Thanks rbwhitaker 14 Feb 2013 16:20 Yeah, there should be a way of doing this without the special code for the first time. Would you mind posting the relevant code here, or perhaps emailing it to me or something? I'd be happy to take a look and see. After all, it sounds like what you're describing is that I'm setting something one way in one of my tutorials and not setting it back correctly, so it doesn't behave well when you combine the ideas in the various tutorials. I'm just having a hard time pinpointing the actual problem without seeing how you're putting it all together. Yaser Saqib 14 Feb 2013 01:51 Thanks for doing these tutorials! I need some help implementing this shader in my game. Got it all up and running and everything works great. I would however like to make a change where there is a toon Outline but with Diffuse surface‎ so no celpixelshader? i guess? How do i go around this and edit the shader to just outline my object and just show diffuse texture? Heres what it looks like atm. Regards Last edited on 14 Feb 2013 16:22 by rbwhitaker Show more rbwhitaker 14 Feb 2013 16:27 This should be do-able by turning the pixel shader code into something more like what you find back in the diffuse lighting shader tutorial. The shader I presented in this tutorial is actually doing a lookup of the model's texture and then just "discretizing" the intensity. That's what gives it the cartoon-like feel. If would seem like if you just remove the code below from the pixel shader, you'd get what you're looking for. // REMOVE THIS CODE // Discretize the intensity, based on a few cutoff points if (intensity > 0.95) color = float4(1.0,1,1,1.0) * color; else if (intensity > 0.5) color = float4(0.7,0.7,0.7,1.0) * color; else if (intensity > 0.05) color = float4(0.35,0.35,0.35,1.0) * color; else color = float4(0.1,0.1,0.1,1.0) * color;

---